Voice of the DBA

Category: Editorial

  • A Double Dose of Trouble

    It’s Friday the 13th, during the month of October. Halloween, a strange holiday associated with horrible events, and Friday the 13th, a bad luck day, are happening this month. Both are associated with horror movies, implying that this isn’t necessarily a good month for the world. It’s a double dose of bad luck, which is what Equifax would have you believe. I wrote about this yesterday, where the company would have you believe a single employee makes a mistake when a scanner fails, and hackers were ready to attack.

    I’ve, in general, had a lucky career. I’ve worked hard, and things have gone my way. That doesn’t mean I haven’t made mistakes or systems haven’t gone down, but overall things have gone well. As I think about Friday the 13th of October, I’m reminded about Murphy’s Law, where things that can go wrong, may just do so. I’ve tended to expect that some things will go wrong and prepare accordingly. In fact, in keeping with a corrollary that I’ve sometimes heard to Murphy’s law that if one thing goes wrong, I’d expect that something else will fail.

    On this double downer of a day, I’m wondering if you’ve experienced that. Have you had something bad happen at work, only to find that overshadowed by something worse? Hopefully you didn’t make a security mistake and then get hacked, but perhaps something else has broken?

    In my career, I’ve had this happen, though not directly to me. I consulted with a small company to make a little extra money and because I liked their business. I was rooting for them to succeed and wrote some database code to help speed up their application. One day they call me to say that a drive array failed with the database mdf on it. I wasn’t worried, and while in another state, I was ready to walk them through a restore process on the new array. Unfortunately, this was a default install, and the default backup path was on the old array. I hadn’t set this up, nor was I aware of their backup process.

    The lead developer was proud of himself. He’d made a backup the day before, using a file share on a network drive. He had the .bak file. I thought this would be a quick and easy call until he read the error message that not all media sets were provided. That’s when I realized that the developer, while making the backup in SSMS, hadn’t cleared the previous backup file from the dialog. He’d inadvertently created a striped backup.

    Eventually they managed to locate a backup that was many days old. I’m not sure how far back, but they angered a number of clients, who I’m sure all started looking for a new vendor. Eventually the company failed, and I moved on. There wasn’t much to be done in that situation, but it was a certainly double dose of things going wrong at the same time.

    I hope few of you have had double the bad luck in your career, but if you can share a story, we’re happy to commiserate with you.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.8MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • The Blame Game

    Make no mistake, this is going to be something that happens again. The former CEO of Equifax blames their massive data breach on a bad scanner and a person. I’m not talking about a data breach, of course those are going to happen, and when they do, assume that every piece of data in the system is compromised. I know some digital forensic scientists are really talented, but is a company that didn’t necessarily pay attention to security in the first place going to ensure the analysis is done right? Not likely. Assume every record is compromised.

    In this case, the former CEO calls out a person that made a mistake, and then says technology failed. I don’t think that’s true, and I’d agree with Patrick McKenzie, who has a good thread on Twitter. A bad engineering decision, or even a process, is the result of multiple people making mistakes. Certainly there are people that must back up the Apache Struts patch person when they’re on vacation. Or there should be. If there isn’t, then that’s a management failure at multiple levels.

    The thing that concerns me is that we, as tech workers, are going to be blamed going forward. The individual isn’t named here, but I bet at some point they will be. And some, or many, tech workers will get sacrificed for a company that wants to show contrition and action for security mistakes. It’s common for someone to take the blame, but I haven’t seen a specific person be identified (or their inaction be called out) in the past. I’m sure some tech people were probably fired after previous incidents at large companies, but not publicly.

    While the person wasn’t named, there was a report that this individual was no longer employed. Fired? Quit? Who knows. Certainly it’s likely that once this breach became public, anyone who might have been responsible for watching CERT lists, applying patches, or anything to do with Apache Struts might be blamed. In fact, I don’t know I’d want to continue working at a company that might publicly blame my role for a massive breach. My career might be dead with that management, so I might as well move on. Much easier for everyone to blame me than accept responsibility.

    This is the first time I’ve seen an IT employee blamed. BA said an IT systems failure with their major issues. Yahoo and Target were hacked, but no one in IT was blamed. Sony didn’t blame their IT staff after their emails and films were released. Yet Equifax did. I hope this isn’t a sign of things to come.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.5MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Time is Precious

    I’ll hear people constantly say they don’t have time to work on their career. They can’t attend a UG meeting to network. They can’t spare a minute to go through a Stairway Series. They have family commitments, kids, hobbies, volunteer activities, spiritual needs, and more. That’s not even counting all the work they need to get done as a part of their job. When can they spend time on R or Machine Learning or CosmosDB or anything else?

    I get it. My life is chaotic as well, with deadlines and a pile of work that never goes away. I sometimes dread travel and vacation because that means my work piles up on either side of those events. This is on top of commitments to keep up on chores at home (I have cooking and laundry), fix things at the ranch, spend time with kids, get date nights with my wife, and still get to the gym. Most weeks feel like a crazy treadmill for me.

    In my life and career, I’ve learned a few things. One of which is that I need to prioritize what’s important to me. That often helps me decide when to do X instead of Y. I regularly make choices between opportunities, but I also try to evaluate the relative importance of events. I rarely miss family events or kid activities, but sometimes I do. I know that there are times that I feel my work is more important in the moment. However, I try to balance those out to be sure I remember that family is more important overall.

    I’ve also learned that I can find time for things that are important. I have to adjust priorities and scheduling. It is important to not just overload myself with more work, which is easy to do. I find that if I try to get more done in a week, I can’t do that for long. Like a big project at work, I’ll burn out without balance. Instead, I’ve learned to let something go a bit, or cut back on some part of my life to make a little time.

    Everyone can spare some time. Especially those of us in the technology industry. We need to learn and grow, even if it’s just for ourselves. We need to keep our brain’s active, and on a regular basis, we should try something new. Other professions can do this, and we should as well.

    I know you can’t dedicate a half a day a week to a class, but maybe you can spend 30 minutes or so working through a book. I know you can’t make the user group every month, but you can make one or two meetings a year if that’s something you enjoy. You can’t go through all our Stairways, or maybe not even all of the Indexing levels in that series, but you can start learning a bit more with one. Perhaps you can’t make SQL Saturday this year in your area, but is it worth a bit of your day to watch a presentation from the SQL Bits archives?

    I bet it’s possible, and I bet you can find a little time every month or two to grow a bit as a technology craftsman. Make sure you take care of yourself, and actively work on your career, in balance with the rest of your life.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.4MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Citizen Scientists

    The last few years have seen quite a bit of demand for people that can analyze data. Certainly the Microsoft platform has embraced more ways to import and visualize data with tools like Power BI, more intelligent analysis with SQL Server Machine Learning and Azure Machine Learning. I think this is because there’s been quite a demand for data scientists, and with all the media attention, plenty of IT executives are searching for these people.

    It’s a good time to change to that type of career, if it’s what you want. You certainly should investigate what’s involved if you have any interest. Buck Woody has a Data Science series that can help you. Microsoft has a Professional Program in Data Science, and there are not shortage of R and Python resources that can help you experiment with data analysis in a new way. SQL queries work well, but you might find that having other tools in your toolbox is helpful.

    As much as there is a demand for professionals, Microsoft, Amazon, Google, and others are trying to find ways to reduce the cumbersome nature of the tools so that anyone that understands the science part can do the work. Will this mean the average business analyst be able to leverage tools and platforms to perform complex data analysis? Maybe, but I don’t believe these citizen scientists will remove the need for dedicated professionals. There’s an argument in this piece that they might, so you’ll have to decide what you think.

    The danger is that it becomes so easy to perform some analysis and create a visualization that we will likely have lots of people building reports and drawing conclusions without really understanding how they’ve aggregated or filtered data, perhaps without even understanding the implication of making these changes. I could see all sorts of poor decisions being made because a manager thinks anyone can use a tool to extract information from data, so they let just anyone do so.

    Maybe this is where the data scientist steps in. Help users to refine their analysis, understand the problems when data is put together or taken apart with these tools. Certainly they should ensure that users have good, clean data sets. The last thing we want is another IT bottleneck, but perhaps using highly technical people to review other analysis and ensure no fundamental mistakes are being made by the analysts is a good use of IT skills.

    This is another place where we need to ensure that different groups inside of a company can work together to be more effective for the organization. That’s the DevOps mentality. We get things done, regardless of who does the work or whose responsibility it is on paper.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.4MB) podcast or subscribe to the feed at iTunes and Libsyn.