Voice of the DBA

Category: Editorial

  • Multiple Failures

    Imagine this right after a failed RAID array.

    When we have a large scale disaster from a man made system, usually we have had multiple things go wrong. Chernobyl had multiple failures, so did the recent issues in Japan at their nuclear facility. It seems to me that in many cases when Mr. Murphy has struck in my career, he has done so multiple times.

    Recently I was studying some high availability (HA) scenarios. In a mirroring situation the implication of a long mirroring queue was shown dramatically as a the primary server failed and the company experienced data loss in a failover to the secondary server. The data was eventually recovered by bringing up the primary server, extracting the missing data, and then manually merging it back into the system.

    Incidents like that are never fun to deal with, but are they common? It seems that the old adage about “when it rains it pours” has happened more than a few times in my career, but for this Friday I wanted to see if that has been the case for the rest of you.

    How many of you have had multiple failures at the same time?

    A RAID drive fails and shortly thereafter a network card dies. You find corruption and while working on that, the server hardware crashes. Have any of you had two or more failures occur on the same system at almost the same time?

    Let us know this Friday, along with any lessons learned. Have you changed the way that you prepare for disasters or plan for issues in your career? I’m sure there are many people that would love to learn those lessons from a post rather than experiencing them first hand.

    Steve Jones

    The Voice of the DBA Podcasts

  • Understanding Security

    We have to learn to use better encryption than this.

    SQL Server has become a very complicated product, with so many subsystems and features that I don’t know anyone that is an expert in all of them. There are lots of people that become somewhat familiar with many features, and learn to understand enough to use them competently. However there is one area that seems to confuse many people, but is one area that is also quite important to a secure SQL Server: encryption.

    It seems that the idea of encryption is easy, but once we get into the actual practice of managing keys, indexing encrypted columns, and dealing with disaster recovery techniques, encryption quickly becomes complex. If the technical people managing servers struggle to deal with encryption, what hope does the average user have to implement encryption? Likely little to no hope of doing it well, which is a problem as many end users will have data on their machines. TDE is supposed to make this easy, but it solves only certain problems and isn’t available in all editions.

    I ran across a very interesting article in the Economist on what a general understanding of what encryption means in a practical sense. The article is somewhat based on the Dropbox issues I wrote about recently, but also speaks to the general misunderstanding many people have about what encryption actually means.

    I’ve always been hesitant to implement encryption widely, mostly because of the problems of managing keys. Keeping track of them, ensuring they are safe, in multiple places, and easily deployed in a DR situation, is a complex task, and making a mistake can have permanent consequences.

    I don’t know how to both maintain security, and also implement enough safety to ensure access to encrypted data is available, but I do know that this is a task data professionals need to learn to accomplish.

    Steve Jones

    The Voice of the DBA Podcasts

  • Etiquette

    There is no book for modern etiquette

    The world has changed a lot in my lifetime in the way we communicate with others. I am sure that when the telephone was first invented, or live news broadcasts were available, people felt a similar shift, but the way that technology has allowed us to keep in touch with each other, instantaneously, almost anywhere on the globe, is amazing. I grew up with one phone in the house, with an extra long cord, and now have a Smartphone that gives me not only voice communication, but incredible amounts of data on a regular basis. I find myself communicating with text (Twitter), voice (phone), and even with myself (Evernote) almost every day.

    The shifts have also occurred very rapidly, which has brought with it challenges, and occasionally problems. This Intel survey highlights a few issues, and this PSA shows one that many of us are probably affected by at different times.

    In technology we have to deal with interruptions, and sometimes urgent situations that require immediate attention. We have to respond like a doctor on call for emergency room situations, but the reality is that most of our “alerts” aren’t that critical and no one, or no business, will die if we are a few minutes late. There are exceptions but for in most cases these events aren’t extremely time sensitive.

    As you build monitoring and alerting systems, include a little etiquette . Don’t unnecessarily page people, or alert them for situations that don’t need immediate attention. Bring visibility to problems, but don’t overwhelm someone with a large amount of data, and please, don’t alert people that things are working correctly. At this point in computing, automated systems ought to triage out most of the minor issues without involving humans.

    Our world has changed rapidly, with the speed and accessibility of communications growing quicker than our ability to deal with it. In some sense, I think that the deluge of data that we now have will bring new problems to our world, especially as more and more businesses look to analyze data in real time. We will face challenges that would have seemed like science fiction a decade ago.

    Steve Jones

    The Voice of the DBA Podcasts

  • Cloud Outages

    Cound vendors need to make an effort to make their service "smarter"

    It seems that almost every few weeks there’s some announcement of an outage for a large service provider on the Internet. We’ve had Amazon Web Services go down, numerous Salesforce.com issues, and Microsoft’s Office 365 recently had problems. Most of us that work in IT are probably painfully aware of our own outages and how difficult it is to keep servers up across a long period of time.

    There are plenty of companies that have had outages, and I expect we’ll regularly see more announcements as companies look to engage cloud computing services. This isn’t to say that cloud services are unreliable, or even less reliable than in-house services. As is the case with many choices, your mileage may vary (YMMV) no matter which way you choose to provision your IT infrastructure.

    In my mind, we need to ensure that cloud computing vendors take these outages seriously, inform customers what went wrong, what was fixed, and what is being done to reduce the chances of outages in the future. Google has a dashboard, which is actually nice, albeit a little simple. It ignores some large Google sites, like Blogger, which had an issue a few weeks ago that was very annoying for me as I tried to publish a few entries.

    One would hope that cloud services would take their responsibility to offer a service seriously, and have less downtime than any random IT department. However the people running many of the daily operations at cloud service providers aren’t the best of the best IT people. They’re the IT people just like you and me that have a job to do, and do it with varying degrees of professionalism, and success. I would hope that they have better procedures, more consistent monitoring, and quicker response times, and some will, but not all of them.

    Moving to the cloud solves some problems in an IT infrastructure, but at the expense of less oversight, potentially less security, and certainly less control over how the daily operations are handled. It should result in better reliability for the systems over time, but as I mentioned, YMMV.

    Steve Jones

    The Voice of the DBA Podcasts