Voice of the DBA

Author: way0utwest

  • Etiquette

    There is no book for modern etiquette

    The world has changed a lot in my lifetime in the way we communicate with others. I am sure that when the telephone was first invented, or live news broadcasts were available, people felt a similar shift, but the way that technology has allowed us to keep in touch with each other, instantaneously, almost anywhere on the globe, is amazing. I grew up with one phone in the house, with an extra long cord, and now have a Smartphone that gives me not only voice communication, but incredible amounts of data on a regular basis. I find myself communicating with text (Twitter), voice (phone), and even with myself (Evernote) almost every day.

    The shifts have also occurred very rapidly, which has brought with it challenges, and occasionally problems. This Intel survey highlights a few issues, and this PSA shows one that many of us are probably affected by at different times.

    In technology we have to deal with interruptions, and sometimes urgent situations that require immediate attention. We have to respond like a doctor on call for emergency room situations, but the reality is that most of our “alerts” aren’t that critical and no one, or no business, will die if we are a few minutes late. There are exceptions but for in most cases these events aren’t extremely time sensitive.

    As you build monitoring and alerting systems, include a little etiquette . Don’t unnecessarily page people, or alert them for situations that don’t need immediate attention. Bring visibility to problems, but don’t overwhelm someone with a large amount of data, and please, don’t alert people that things are working correctly. At this point in computing, automated systems ought to triage out most of the minor issues without involving humans.

    Our world has changed rapidly, with the speed and accessibility of communications growing quicker than our ability to deal with it. In some sense, I think that the deluge of data that we now have will bring new problems to our world, especially as more and more businesses look to analyze data in real time. We will face challenges that would have seemed like science fiction a decade ago.

    Steve Jones

    The Voice of the DBA Podcasts

  • T-SQL Tuesday #19 – Disaster Recovery

    TSQL2sDay150x150This month Allen Kinsel hosts T-SQL Tuesday #19 with a disaster recovery theme. This is the blog party idea started by Adam Machanic where everyone that participates writes on a single theme.

    If you want to host, contact Adam. If you want to participate, read Allen’s post and write your own post and link it in.

    What’s a Disaster

    It’s interesting that this is the theme since I already had two posts on DR scheduled this week and am working on modifying my “Preparation for Disaster” presentation for SQL in the City.

    It’s hurricane season for Allen on the Gulf, and while hurricanes give you advance notice when they are coming, you have to be prepared. I’ve lived on the East coast, and we always were worried about getting hit in Virginia Beach. We never did get hit, but that didn’t stop us from being prepared and sometimes even initiating some of our disaster protocols.

    As a side note, we had an office in High Point, NC, and it actually got hit by a hurricane, or the remnants of one and lost power. I was in a downtown hotel at the time and woke up without power.

    The thing is large disasters are rare. You have very little chance of your data center getting hit by a hurricane, a tsunami as Japan recently had, a tornado such as the Northeast have experienced this year,  They happen regularly, but the chance of them hitting your data center is low. The chance of you “losing” a data center is low. It just doesn’t happen that often, and the complete loss of a data center might mean you have bigger things to worry about.

    However there are plenty of disasters that you are likely to experience. It’s possible that you could have a fire in your data center that affects the database server. Not likely, but it could happen. It’s more likely that you could have one of these:

    • an operator spill coffee on your server
    • someone trips over a power cord or fiber cable and brings down your system
    • a memory chip go bad and get in memory corruption.
    • it’s possible power goes out for some reason, even with a UPS.
    • It’s highly likely that a drive will fail that contains data you need.

    These are the types of disasters that you really have to prepare for, since they are possible, and even likely.

    What’s more likely are the “Whoops” disasters as I like to call them. It’s very likely that someone will import the wrong file, update all prices instead of one, or even more likely, a developer or DBA will run the wrong code on the wrong server. The most likely disaster is probably this one:

    drwhoops

     

    If the DBA hits “Execute” here, it’s a disaster. Not a big one, but if this is a critical table in a critical system, you might have an even bigger reaction from manangement than if a hurricane hit.

    You have to prepare for disasters, but don’t get caught up in worrying about the data center being destroyed. Those disasters are rare. Most of your preparation, your practice, your checks, have to be focused on the more likely disasters, which are often smaller in scale and focused on your database.

  • What’s a Disaster?

    In working on my Preparation for Disaster presentation and doing research, one of the questions that came up early for me was “what is a disaster?” After all, if you can’t answer that question, how do you prepare for it.

    I’ve got a list of stuff here, which ones of these qualify as disasters?

    • Hard Drive crash
    • Hurricane wrecks data center
    • Fire in the server room
    • Corruption in a clustered index
    • DBA deletes a table by running DELETE without a WHERE clause
    • User clicks submit on a web page and the database crashed before they get the acknowledgement
    • Admin uses Access to update and lock a table, and forgets to save their work before leaving for lunch
    • DBA deploys code to production instead of development by accident.

    The answer is all of these are disasters. Most people seem to plan for the first 3, and forget to plan for the rest.

    Anything that interrupts your system’s ability to serve clients and get work done is a disaster. It’s worth keeping that in mind and planning for all sorts of potential problems. It doesn’t mean you have to be ultra paranoid, but you ought to have thought about the various potential problems and have some ideas about how to deal with them.

  • Cloud Outages

    Cound vendors need to make an effort to make their service "smarter"

    It seems that almost every few weeks there’s some announcement of an outage for a large service provider on the Internet. We’ve had Amazon Web Services go down, numerous Salesforce.com issues, and Microsoft’s Office 365 recently had problems. Most of us that work in IT are probably painfully aware of our own outages and how difficult it is to keep servers up across a long period of time.

    There are plenty of companies that have had outages, and I expect we’ll regularly see more announcements as companies look to engage cloud computing services. This isn’t to say that cloud services are unreliable, or even less reliable than in-house services. As is the case with many choices, your mileage may vary (YMMV) no matter which way you choose to provision your IT infrastructure.

    In my mind, we need to ensure that cloud computing vendors take these outages seriously, inform customers what went wrong, what was fixed, and what is being done to reduce the chances of outages in the future. Google has a dashboard, which is actually nice, albeit a little simple. It ignores some large Google sites, like Blogger, which had an issue a few weeks ago that was very annoying for me as I tried to publish a few entries.

    One would hope that cloud services would take their responsibility to offer a service seriously, and have less downtime than any random IT department. However the people running many of the daily operations at cloud service providers aren’t the best of the best IT people. They’re the IT people just like you and me that have a job to do, and do it with varying degrees of professionalism, and success. I would hope that they have better procedures, more consistent monitoring, and quicker response times, and some will, but not all of them.

    Moving to the cloud solves some problems in an IT infrastructure, but at the expense of less oversight, potentially less security, and certainly less control over how the daily operations are handled. It should result in better reliability for the systems over time, but as I mentioned, YMMV.

    Steve Jones

    The Voice of the DBA Podcasts