Voice of the DBA

Tag: data management

  • Data Preservation

    This editorial was originally published on Dec 3, 2013. It is being re-run as Steve is on holiday.

    Maintaining data across time isn’t something many of us think about. We work with data in the here and now, and in the database world, we typically only need to recover or restore data from a short window. Like most of you, I would usually plan on recovering data that’s only a few days old. Being forced to restore a database from two weeks on any of my systems would make me cringe. It certainly would be embarrassing for me personally if it were my fault I couldn’t restore to a point in time that was more recent than that.

    In planning to recover our systems, we typically know the versions of software we have to recover from, and we can easily re-download copies of SQL Server or the patches we need. Most of us are dealing with SQL Server 2000 or later, which is good since those are the only versions still documented on MSDN. If you need SQL Server 7.0 or SQL Server v6.5 documentation, I hope you have copies.  The same goes for the media. You can still download SQL Server v6.5, and SP5, but if you needed SP3, it isn’t easily available. I ran into that situation about 10 years ago, and we had to make a special request through our TAP manager to get someone in Redmond to dig up a copy.

    In some ways it might not be important to worry about long term storage. Most of us will end up transferring our data to newer systems (and formats) over time. As we upgrade SQL Server, our databases move along to newer formats, or we abandon them because they are no longer needed. That’s fine for some data, but not all.

    Long term archival and storage is a challenge, as you can see in this short look at how old films are maintained. It just touches the edges of what’s being done, and doesn’t address costs. Plenty of old films have been lost forever, and perhaps that doesn’t matter, but it does concern me. I have thousands, maybe tens of thousands of digital images. While I love the ease with which I can share them with family, and make extra copies, I am worried that perhaps the lack of a physical copy means my great-grandchildren will struggle to find evidence of my generation if there is a catastrophe or storage formats change.

    This is one area of our industry in which we have a lot of maturing to do, and I hope that we can come up with some new ideas for maintaining our data for the long term, across not months or years, but decades or centuries.

    Steve Jones

  • Separation of Duty

     In the Vietnam war we found people gathering data that had to support the analysis or decisions of the leaders. That’s a fundamental problem. Arguably this led to the extension of the war, and more lives lost, without more decisive actions being taken.

    We should not have the people or systems designed to collect data that supports a particular position. We want data that is accurate and complete, which can then be used for analysis. We ought to have people gathering data separate, both in terms of hierarchy and process, from the people that analyze and make decisions. We know that keeping them in the same area leads to data that supports what the boss wants, or what the bonus warrants, rather than what the data can show.

    Have you heard of the Franklin Gambit? I was reminded of that recently while reading Obliquity, a book that examines how we pursue goals and achieve success. Franklin’s Gambit says that we often use data and explanations to justify a decision we’ve already reached, rather than actually prove that some choice is the best one. I’m sure most of you feel you don’t live your live this way, but as I’ve thought about some of my decisions, I think this isn’t necessarily true.
    It’s a natural tendency of humans to do justify their actions, regardless of the data, and I suspect many of us suffer from it. I think it also can extend to the data analysis and interpretation that many of us perform at work. We’d like to think that our reports and even data gathering are objective ways to examine a problem, but it can be easy to influence the data, especially in the way that data is gathered if we are not careful.
    We want to separate the data gathering from the analysis. It might not ensure that the data analyzed is as complete and accurate as possible, but it helps prevent one side from influencing the other.

  • This is Big Data

    I’m sure some people have bigger data, but Facebook has some serious daily growth. The highlight? 500TB a day.

    A day.

    That’s serious data. I’ve never had a database that was a TB, must less growth by over a TB a day.

    To be fair, lots of the Facebook data is unstructured, stored in Hadoop or NoSQL structures for images/video, but there’s a ton of database-like status updates, links, comments, etc. that have to be managed.

    Rather impressive, but not something I want to administer.

  • Intertia

    Inertia can be hard to overcome

    This editorial was originally published on August 22, 2006.

    My son asked me about physics the other day and what it meant. He’s been on a bit of a science kick, looking to understand and explain his world. I can understand that because most of us are curious about how things work in our world, not necessarily in a scientific way, but just in general.

    I somewhat butchered the definition because I was caught up slightly in the actual concepts, but one of the things we talked about was inertia. Maybe because I’d seen this article on overcoming institutional inertia the same day.

    We all have things at work that we do a certain way. Maybe we learned it that way, maybe we’re more comfortable, or maybe someone yelled at us daily until we did it that way. Regardless, fighting the way people work is a difficult thing to undertake. The article gives a few examples of process changes and how they were effectively moved forward and how the issues were resolved at a high level. It’s common sense if you’re wondering, use diplomacy and take your time.

    I’ve seen changes occur from both sides of the fence, from someone trying to implement change and from someone who was told changes were occurring. I was never completely comfortable in either situation mostly because with change there’s usually conflict. Someone doesn’t want to change.

    As DBAs we often try to control our world and ensure a high degree of stability, often at the cost of slowing or severely impeding changes in our systems. While I think it’s good to aim for a stable, reliable environment, I don’t think this should necessarily impede change.

    Understanding the business needs and goals is critical both for DBAs in protecting their systems and developers in building new applications. In either case, it also means that the needs of your organization need to dictate how quickly things change.

    Not an employee’s personal feelings.