Voice of the DBA

Category: Editorial

  • Two Days Off

    I almost couldn’t believe this when I saw the article. The Verizon Cloud is shutting down for 48 hours. Apparently they have maintenance scheduled for this weekend and notified their customers that their virtual machines will be shut down early Saturday morning. There are some legacy Verizon cloud-type services that will be available, but the platform they’ve been pushing to customers will be down.

    This isn’t good news for Verizon or their customers, but it also doesn’t help the cloud overall as a service. This outage reinforces the idea that reliability isn’t necessarily better for vendors than individuals. If costs for the cloud are anywhere near that of on-premise hosting, this event would certainly make me think twice about moving anything really important in my organization to a single cloud vendor.

    I suspect that most cloud vendors have outages like this, but they don’t shut down their entire clouds. When a large amount of maintenance is needed at a data center, most vendors would migrate customers to a separate data center or another part of their cloud while they perform their work. Either this mainenance is a major change to Verizon’s entire infrastructure that can’t be staged on just a part of their system, or they’ve poorly planned their architecture and maintenance.

    Either way, this weekend will certainly be a good DR test for enterprises that might have important applications hosted with Verizon. It will also be a chance to test how these clients notify their own customers of potential issues or how they respond to problems. I wouldn’t want to have an application hosted with Verizon as it would be a lot of work for me and likely a weekend away from family, but I know I’d learn a lot about how well I’ve prepared my own systems for fault tolerance.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.2MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Test Your Situation

    I gave a talk on Transparent Data Encryption (TDE) recently and a number of people in the audience were using the feature. However when I asked how many of them had restored a TDE database, not all hands remained up. When I asked how many people had restored their TDE encrypted backup  to a different server, one that didn’t have TDE enabled, very few hands remained in the air.

    That’s not good, and I certainly hope those people don’t experience a disaster from which they cannot recover. I’m sure they are not alone. I suspect that many of the people managing a TDE database have restored a database this year, and are confident they can do so. However what they don’t know is if they can restore those TDE databases on any other instance, including a newly installed one.

    They’re not alone. I see many, many people implement features they don’t really understand. Microsoft has made it easy to set up replication, clustering, and more in your environment, but without providing some of the robustness and reliability that many people need. The ease of setting up a feature is one thing. The ease of ongoing management and recovery when issues occur is something else entirely.

    I really wish that Microsoft would go further than making implementation easier and include direction for ongoing tasks. When databases are created, ask the user to set up backups and help them create the jobs. When encryption is implemented, do more than display a warning message. Help adminstrators prepare for recovery with templates or jobs that automatically build certificate or key backups. When replication is set up, include a script to rebuild the environment for when it breaks.

    I doubt we’ll get this, and many companies and employees will continue to implement features they don’t understand. You can only help your own situation, and you should be ensuring you understand and can rebuild all the extra features you’ve installed in the event of a disaster.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.3MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Recovering from Bad Deployments

    Someone asked me recently if I stored backups of my database in version control. This person wanted to recover from a bad deployment and use a backup to do so. They felt that keeping a copy of the backup in a VCS, alongside the code being deployed, would be important. It might be, but I said that recovering from a bad deployment isn’t something I want to do with a restore if I can avoid it. Then I was asked how to recover from a bad deployment if you have a busy, 24×7 environment.

    I, of course, answered, “it depends.”

    It does depend on the deployment, but it also depends on your preparation. There are ways in which you can work to minimize the problems that might occur during a deployment. Obviously testing your scripts and deployment process is important, but it’s also good for you to understand how your scripts work and what techniques you can use to rollback problematic deployments.

    There’s a switch in SQL Compare that lets you build a deployment script, and then immediately generate a rollback script. It’s handy, but it’s also not going to always work. If you’ve added a column during deployment, you might not want to just remove it on rollback. 

    However you can prepare for issues, like having a script that might save data in the new column before you remove it. You might choose to copy the table as part of a pre-deployment process (or during deployment), having this copy of the table used in a rollback scenario. You might even bring up a warm copy of your database and prepare to swap entire databases if problems arise. This would allow you time to save and move data that was changed after your deployment, but before your rollback.

    There are lots of possibilities in how you might recover from a failed deployment, but as with many of the solutions that we build in technology, a well thought out plan makes everything run smoother.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.1MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Staying Successful

    This is an interesting piece on data predictions. There’s a college that’s using data to help students, but not in the way you might think. Rather than customizing learning plans or finding more efficient ways to teach, the college is analyzing data to find students that might be struggling with college life overall and intervening early to help them cope with college and continue to move forward in their studies before they drop out.

    This is similar to something HP has done, trying to predict if employees might leave and intervene to see if something can be done. I heard an interview with the HP managers, who praised the program, and found they were among the most likely candidates to leave the company. However HP also realized that identifying people likely to leave allowed their HR people to try and find creative ways to retain their employees.

    I’m not entirely sure this is a process I’d like to be a part of, but it does seem like HP is doing well in their experiment. They realize they can’t solve every issue and keep every employee, but they can work proactively to try and reduce the losses. However privacy is a big part of this data analysis and I hope HP keeps that in mind as they move forward.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 1.8MB) podcast or subscribe to the feed at iTunes and LibSyn.