Tag: data

Email has been a part of my life for well over nearly thirty years. It’s kind of amazing to think of a time back in high school when there weren’t electronic communications. That seems foreign now, as we don’t have to send physical objects or converse with voice to communicate. The world of communications have changed dramatically.

These days email is still a preferred method of communication for many people. Even if we use sites like Facebook, NextDoor, or SQLServerCentral to communicate with others, we often get notifications of changes through email. If you’re like me, many of you might keep far too many emails in your mailbox, rarely removing unnecessary ones. In the era where we measure storage in dozens or hundreds of GB, or even in TB, do we bother to even manage text communications?

Some firms do require this, often for legal reasons. With the GDPR, I wouldn’t be surprised to see more organizations starting to set retention policies that ensure that communications don’t live forever. There are some systems that do this now, but the practice isn’t ubiquitous, but maybe it will be soon. Google is redesigning Gmail, which will include Confidential Mode. Not only will there be limits on these emails, but one of the more interesting is the ability to expire an email and have it automatically deleted.

I don’t love the idea of having communications disappearing, but that might be because of the way I’ve grown up. I don’t like using Snapchat with my kids, because I don’t want pictures I take to disappear. However, younger generations don’t feel this way. As I wonder why I try to hold onto old communications and records, I start to wonder if the idea of expiring data is something that we should be embracing as data professionals. Do we really need sales records from a decade ago? Are recordings of web traffic valuable from the early days of SQLServerCentral? Is there really any point to holding onto much of the data we generate?

I know that there are corporations that hold onto decades of paper records. I worked at one that had nearly a 100 years of old records, most of which might never be examined again. Likely plenty of them aren’t even legible or useful at this point. They’re being stored for, well, I’m not sure why. I’m sure there are plenty of writers that might come up with a detective story that requires old paper records, but I’m not sure there’s practical use for this data.

I expect that we’ll start to see organizations changing their record retention policies as we look to avoid more liability and risk from data breaches. Every old record, every piece of PII data that we no longer use probably needs to go. Even records for existing customers might need to be removed. I don’t necessarily need to ever access the record of my first Amazon order from 1998. I’m really sure that Amazon having liability for holding my old address, which potentially could be used to validate identity, is a bad idea for both them and me.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 4.2MB) podcast or subscribe to the feed at iTunes and Libsyn.

This is interesting to me. Let’s leave aside the moral, ethical, or political sides of this. Let’s just look at the data. A cop gave a ticket to a self driving car. Apparently the officer thought the car was within 10 feet of a pedestrian. The logs for the car show that it was 10.4 feet away. A negligible difference to me, but that might matter in a court of law.

As we use digital data more and more to control vehicles, devices, even behavior, we are going to have problems, complaints, and unforeseen issues amongst humans and computers. Certainly our laws need reforming to deal with the digital world, but we also will have some cultural transformations to undertake.

Much of our arbitration in the world has dealt with how well humans argue or debate topics. We are inconsistent, easily confused creatures, yet we often use our judgment and impressions to make decisions about whether we believe one individual over another. This has often been the case in law, even as more science with cameras, DNA, and other forensic techniques have tried to provide definitive proof one way or the other.

As we use more digital devices, there will be, or should be, more logs and audit records of how systems behave. This worries me a bit in Machine Learning systems, but experts seem to believe we can unpack the rationale for decision making if necessary. In any case, we need to learn to trust and believe in the data, even if our eyes and instincts lead us to different conclusions. This doesn’t mean that the data makes the decision, but we can’t discard the data because we don’t like it. Certainly context matters, and in the original case above, we’d want to consider velocity and acceleration, and possibly other factors to determine if the car was too close.

Ultimately, I believe digital records will start to prevail in more circumstances than we might be comfortable with. We’ll slowly change, and likely subsequent generations will trust data more than we do, perhaps even more than their senses. To me, this means that those of us that deal with data systems need to ensure there is extremely strong security and integrity of data records, and that we disclose algorithms and data processing techniques in a transparent way. Trust requires knowledge, which requires transparency. That’s something we certainly need to improve on as an industry.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 3.1MB) podcast or subscribe to the feed at iTunes and Libsyn.

We are great about sharing knowledge in the technical community. I think the SQL Server community is even better than most technologies as we do a good job of doing so without the rude RTFM that I find in other communities. However, overall we share lots of information with others. I do know that many other communities are catching up and I really appreciate the answers I’ve gotten from others when trying to repair my tractor or auto.

As the world looks to bring innovative solutions to various problems, it seems that machine learning will play some role. There are some amazing advances taking place in all sorts of areas, and certainly no shortage of organizations that are trying to see if ML enhancements will improve their capabilities. We hear about ML all the time as data professionals, and some of you are starting to try and use these algorithms to build new applications.

I ran across a great post about innovation that notes that the fuel for ML and related technologies is data. It’s important for training and evaluating systems, and there are companies making a lot of money providing training data. However, there is also lots of data inside companies, especially some of the big social media and communication companies that gives them an advantage. I think that’s OK, after all, these companies have innovated to build large scale enterprises and devoted resources to collecting data.

However, I do think that more and more data is needed, and companies will work to collect it if they can. That means more and more of us will manage additional data, all of which needs to be cleansed, stored, managed, and protected. More work for us, but also lots of opportunity. Especially if you learn how to process data in new ways.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 2.9MB) podcast or subscribe to the feed at iTunes and Libsyn.

Is unstructured data a bad term? I saw some data professionals complaining about this, saying all data is structured. That’s usually true. A CSV, even a ragged one has structure. XML and JSON have structure, even if it might vary node to node. Certainly our relational tables are structured and some formats can be rigidly mandated between organizations (like EDI). Even data in PDF, Word, MP3, MP4 or other audio/video mediums is structured in that we know the format.

Given that, is it a misnomer to use the term, unstructured, when describing flexible formats, such as XML? Is it OK for a PDF? I have had a presentation called Unstructured Data in SQL Server. This is primarily about FileStream, FileTable, and searching those objects. In the talk, I classify data in known formats as structured. These would be SQL Server tables and similar objects. At any point in time, we know what all data in the table looks like, even though we can have NULLs or missing data in rows.

I call XML and JSON semi-structured formats. We can certainly determine the format for any node or section, but we wouldn’t know without querying or examining the data. It’s semi-structured in that there is a hierarchy, but the structured from section to section (essentially row to row) can vary. There can even be depths to hierarchies that vary. In many ways, that makes these great formats for flexibility in data exchange.

I tend to view data in Word, PDF, MP4s, as unstructured. We don’t necessarily know where the data is, or how to separate it. We can get pages in Word or PDF, but those can vary and don’t necessarily help us extract information. They are XML, but the XML tags don’t relate to the content, unlike many other XML documents. Scenes or tracks in audio/video files might be separators, but those aren’t necessarily helpful in gathering information. Instead, we need other tools that can help deal with that data, finding words, concepts, or more inside of the binary stream.

I like the term unstructured data because it helps me understand where the information is. While the tables in a database might be full of nonsensical information in some rows, or be poorly designed with data combined into text fields, at least I know where the fields are. Actually, in that case, I’d argue the data in varchar(max) text fields is really unstructured. You might disagree, but give me a better term to describe there the information is stored in a data format.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 3.7MB) podcast or subscribe to the feed at iTunes and Libsyn.