Voice of the DBA

Category: Editorial

  • Dig Out the Root Cause

    Early in my career in technology, I worked as a system administrator on a Novell network. Periodically we’d have crashes of servers on our network, or application failures, and while we understood that sometimes these were caused by transient errors, we often invoked a root cause analysis process when an issue repeated itself. I’m sure that the environment in which we worked, a nuclear power plant, contributed to the idea that we should always understand why some system failed.

    I was reminded of this while reading Grant Fritchey’s “Understand the True Source of Problems” in which he relates a few “best practices” that are likely folklore stemming from ignorance rather than actual good ideas. I’ve seen a few of these rules at various places in the past, often implemented because they appeared to work. However, we need to remember this:

    Correlation != causation

    Just because you perform some action and observe an effect doesn’t mean that one caused the other. If that were the case, I’d never install new hardware or applications on my computer systems as I’ve had crashes during various installations. However I often back out a change, or retry it and realize that I had a coincidental result, rather than a related one. The same thing has occurred in more complex systems where an action appears to cause an issue, but in reality, the two items are unrelated, or loosely related.

    We often don’t take the time to determine the root cause of many issues, which is disappointing. While it often doesn’t seem to be a worthwhile use of resources, I bet that often we’d learn there are actions we are taking (or code we’ve written) that actually is the cause. If we learned from our mistakes and could avoid making the same ones again, we’d greatly improve the quality of our technology systems, with many fewer issues over years.

    Unfortunately, “good enough” is often good enough, even if it does result in a bit of downtime.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.2MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Microsoft Exercise

    Have we got a deal for you! Microsoft is concerned about the health of your company and is doing something about it.

    There’s a trend in modern society of obesity, complacency, and general laziness. Not only does is this bad for society in general, but it also affects your health. So we have a new solution for you, designed specifically to combat this degenerative lifestyle in an easy way.

    The Upgrade Treadmill!

    Designed by the engineers up in Redmond, this highly engineered device ensures that you won’t sit around getting stagnant on those junky-technologies and deep-fried solutions. Instead we’ve built a system of carefully selected products released on a regular cycle that ensures you’ll be running for the rest of your career.

    And it’s easy to get started. Just call 1-800-SHARE-PRICE and enroll now in our assurance program. With a short lifetime and quick expiration, our automated reminders to your boss will ensure that you never slack off and slow down your pace of upgrades.

    It’s all in fun, but there’s some seriousness here. I originally wrote this awhile back when it seemed Microsoft was pressing for upgrades to SQL Server 2008 R2 after a relatively short development cycle post SQL Server 2008. However the piece got lost in the shuffle. I found it recently and was reminded of the sentiment when I saw a post to upgrade away from SQL Server 2005 with support ending.

    There are good reasons to upgrade; just be sure the reasons are valid for your instance and environment.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 1.8MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Code Reviews

    When I first started to be paid money to write software, I worked on my own. I had to test my own code, decide when it could be deployed, and make the decision to deploy it. I had to notify users, and over time, I had to delay deployments because my mistakes would stop a business from getting work done. Fortunately I didn’t cause any serious loss of revenue for a small construction company due to my coding errors.

    Later I worked at a large company and was surprised when I finished a piece of work and had to submit to a code review. My boss told me to get two other programmers, at least one a senior level developer, and have them review my code. In those days we reviewed code on paper, with a red pen reminiscent of my school days. The amount of ink expended on my work was a bit overwhelming, but fortunately I recovered and was able to fix most of the issues quickly.

    I’ve talked to a number of developers over the years and it seems that most of them make their own decisions about how ready their work is for deployment. Code reviews seem to be the exception rather than the rule, but I wasn’t sure if that was the case for SQL Server professionals. Are we more conservative and formal in our approaches? Here’s the question this week.

    Do you perform code reviews?

    Do you review just application code (C#, Java, etc)? Are stored procedures and functions included? Do you include DDL and DML changes in reviews? Are DBAs able to apply changes to production without a second employee checking them?

    In many ways, despite the stereotype that DBAs want to be careful with their systems, I’ve seen many DBAs rashly apply a change to a production server without any oversight or second opinons. It has seemed to me that having the control to decide what changes are made to a SQL Server instance has led to the execution of that power without any oversight for many individuals.

    Let us know this week if you review code, and what things matter. I certainly thing naming and structure matter, though I’m willing to let formatting go as many tools these days will reformat code in whatever structure an individual developer prefers.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.7MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Searching for Plans

    Today’s editorial was originally published on Sept 13, 2010. It is being re-run as Steve is away at training.

    I was re-reading Conor Cunningham’s “Does Join Order Matter?” recently and found this quote in the post that stood out. As an aside, read this post. It’s fantastic.

    “…SQL Server’s QP is actually not set up to give you the best plan.  Instead, it is set up to give you a good enough plan quickly (where good enough is very close to “best”).”

    I get that, and it makes sense. Why spend 4 hours optimizing a query that runs in 2 minutes to get it down to 1:50 (Conor’s example) when the optimizer can spend 1 sec and have it execute in those two minutes. If the optimizer looked for the best plan, I’d be very angry as a customer. Especially if this happened with any frequency.

    Or would I?

    I’m not suggesting that we ought to optimize every query, or that I’d prefer the optimizer didn’t try to normalize queries, or that I want to see the internal tree and try to figure out if I can rewrite the query. I might, however, want to do that for some queries. There are queries that I might want to find the “best” plan for, either because they take a long time to run, or they are run often. I could then use that in something like a USE PLAN.

    Computers are cheap these days, as is disk. If I were to somehow able to copy my database onto an instance of SQL Server on a cheap PC, wouldn’t it be cool if I could set that optimizer loose on my query and let it try ALL possible plans? I’d be happy if it spent that 4 hours and gave me back an optimal plan that I could carry back over to my expensive, busy server, and shave that 10sec off a query that I run 10,000 times a day.

    I could see some value in a “test” instance of SQL Server that just looked for optimal plans. Now if I could just get Microsoft to build it.

    Steve Jones