Voice of the DBA

Author: way0utwest

  • A Really Bad Day

    I was out of town for work one day, literally on the other side of the country, in a car bound for the remote office when my cell phone rang. I had flown into LaGuardia Airport in New York City and needed to drive to Connecticut, to an office just outside of Hartford. The company was paying, and it was more convenient for me to fly into NYC and then take a chartered car to CT.

    I glance at caller ID and it’s a client of mine back in Boulder, CO who I do some occasional SQL Server work for. I answer, thinking that he needs help with some SQL command and I can score some brownie points here without charging him.

    “Steve, I have a problem”

    The words that anyone in technology is loath to hear. Even if you’re being paid by the hour, clients that cause issues are often going to be more trouble than the payment is sometimes worth. I much prefer to hear “Steve, we’d like to do x. How can we do that?”

    I ask him to tell me what’s wrong and he said that RAID card was throwing errors on the development server, so he performed a SQL Server backup, copied the backup to a file server and then replaced the RAID card, rebuilding the array. When he went to restore the server, he got an error.

    I am the backup and restore champ, at least of my little world. “Read me the error, “ I say, confident I can solve this for him before we arrive at the office..

    “The media set has 2 media families but only one are provided”

    Uh, oh, I’m thinking, and not because of the poor grammar. This is not good and I’m not going to be able to fix this. Breaking bad news to a client is never fun, even if it’s not your fault. I’ve lost a couple because they were so upset they let me go at some bad news. That’s actually OK with me because if they can’t tell the difference, they are likely to leave me when I least expect it anyway and I’d rather know sooner than later.

    I explain that he must have used two backup files and he needs to add the second one in there. The guy is insistent that he didn’t pick two files when he made the backup and he copied the right one off. He even included the date in the filename (like I showed him) and reads it back to me.

    I know what’s happened. He’s made an inadvertent striped backup, and since the RAID array was blown away, he’s likely lost the other file. I try to keep from laughing as I explain patiently what happened. He’s upset, then he’s confused, then he’s unhappy. I can picture the sad panda face on him from 2,000 miles away as he realizes that he’s not getting this restore to work. The driver must hear it in my voice as well since he’s smirking in the rear view mirror.

    To add misery here, he wasn’t copying backups off this development server to save space on the file servers. He’s lost a few weeks of work now, and he has to break the news to the developers, who likely don’t have backups of their own.

    I sign off, not sure what to do as I get to my destination. It’s a good example of why SQL Server still needs a DBA, or someone with decent DBA skills. Despite the ease with which you can get many things done, not understanding why you do them cause big problems.

  • Jobs for Data Scientists

    This is the trend line we like to see for database jobs

    I wrote about data scientists while back, which seems to be a field that involves the analysis of large amounts of data and the tools involved in this work. The definition of this area is a little nebulous, but that hasn’t stopped companies from looking for people to work in this area. There are jobs for data scientists, and perhaps for interesting, there seems to be more and more jobs for people to work with “big data”.

    This article shows a huge trend up in the “big data” keywords in job listings. That makes sense as I think the data growth has far outpaced the ability of tools to work with large datasets. Excel had a problem with large amounts of data until recently, and most of the other tools used are either very expensive or home grown.

    Many of the people subscribing to this newsletter are technical people, those that work to form the computer into a tool that someone else can use to do their job. It’s an interesting job, and one I hope you enjoy, but there are some of you out there that would like to do more with data than just find ways to help others query that data. Perhaps a data scientist is a job that you would like to move into at some point.

    This is an area that I suspect many companies will want to focus more efforts on in the future, and I think that being a DBA who knows how to query data and find patterns gives you a head start on moving into a position that can be challenging, fun, and perhaps most importantly, without much after hours work.

    If you like statistics, patterns, and helping your company actually make use of the information in data, perhaps a data scientist might be the position for you.

    Steve Jones

    The Voice of the DBA Podcasts

  • For your CIO: Building Technical Leaders

    I might send this to a VP or CIO: Global CIO: A Framework For Developing Technical Leaders. It’s an article that talks about building senior technical leaders in your company, which I think is important if you view your Information Technology department as strategic.

    That also means thinking about retention, and building an atmosphere where people want to work. That doesn’t mean you can’t have layoffs or need to keep every person, but management does have to show competent and hard working technical people that they are valuable. It should also involve some way to balance the disparate needs of ambitious people that you want to retain and those average employees that want a secure job and are willing to do the average jobs.

    It also involves keeping a career path for technical people that doesn’t lead to management. Plenty of smart technical people want to keep growing, so give them a way to do so.

  • The Impact of Outages

    This is not what any of your clients want to see. So prepare for issues

    Are you going to start seeing more pressure for outages in applications? I suspect many outages are caused more by application issues than database ones, but those two are becoming very tightly linked as we look to more rapidly deploy features and enhancements in our applications, which often include database changes

    This outage from United shows that there can be a huge impact, not only financially, but also an inconvenience to clients and potential lost future business. A company might struggle to with future business after a large outage, especially when there are so many other choices easily available to consumers across the Internet.

    That brings to mind a very interesting problem as companies grow and look to build scalable systems. Large groups of servers require some level of standardization, mostly for the ease of management by IT workers as well as the ability to train future workers to understand the systems. However that standardization becomes a point of failure when there is a problem during an upgrade, or even a hack from some type of malware.

    I saw an interesting piece on how Netflix has tried to expect, and handle failures in the cloud, and a comment from Jeff Atwood that you ought have your own chaos monkey to regularly test your systems. Interesting advice, and in many cases, it’s probably good advice to ensure that both your systems and your people know how to deal with outages.

    You probably cannot eliminate outages, as Netflix and many other companies have learned. However you can work to ensure your people and systems know how to respond. I also wonder if having (at least) two versions of your systems out there at all times that work in a similar way might be a way to provide some tolerance against a single point of failure. I don’t know how you might implement this, but it might provide some protection against a failure in a completely homogeneous environment.

    Steve Jones

    The Voice of the DBA Podcasts