Voice of the DBA

Tag: administration

  • Selling Automation to Ops

    The DevOps movement isn’t new in some companies. It’s the same coordination and teamwork that has existed for a long time between the development and operations staffs. Developers take advantage of the skills in Operations to get standardized environments for their work, and let the Ops people manage (and track) changes. Operations people talk to developers about the challenges and issues faced in production, and the let the developers build applications that can easily be deployed. The sharing of information ensures each group knows what the other faces, and the regular contact builds bonds and respect between employees. Neither wants to let the other down or make someone else’s job any harder than it needs to be.

    However that’s not the case in many companies where developers view Operational staff as complainers that slow the process down. Operations staff see developers as wild and irresponsible, tossing code into production that they don’t need to support and haven’t tested. Both of these views are correct in that each side sees a reality in the process that makes their job more difficult.

    Ultimately I believe it’s up to developers to change things. Those of us that build the software need to respect the problems that instability causes and learn to help ensure that our changes can be deployed smoothly. The development side of an organization has more skill in tracking changes in version control, in managing the movement of those changes among environments, and in programming systems. We should be working to help push that knowledge through to Operations personnel that become responsible for our applications.

    That means we need to build scripts and tools to migrate our changes and give them to Operations. I’d recommend that we learn how to automate the configuration of our development systems, as well as script our changes. Most modern platforms allow us to programmatically make changes, so let’s do that. Then let’s take a few hours and show Operations people how to use these scripts, and let them setup and change our development environments. It will be slow at first, but they’ll learn to make changes faster, but also bring stability to every environment from development to QA to production, and can ensure we have the same configurations everywhere. We’ll have one less thing to manage, and our changes will get deployed faster, but also more consistently.

    Ultimately we all want the same thing. Better software delivered to customers faster. We want to Ship Safe, and Ship Often.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.5MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Multiple Backup Files

    I’ve been writing a little PowerShell lately that will back up databases, move files, and restore them. I’m often testing things, and having scripts to quickly and easily move files is very handy. After one of my posts recently, a reader asked if I’d considered multiple files and how to handle them in scripts. I confessed I hadn’t, mostly because I haven’t had to deal with them.

    In my career, I’ve tended to work with small to medium sized databases. I’ve had young children for a large portion of my SQL Server career and had no desire to babysit multi-hour (or multi-day) restores when things break. I know some people have been through those situations, and good for them. I just know I’ve had enough issues with the low-GB sized database, and haven’t been interested in supporting TB sized systems.

    However that likely wouldn’t be the case in the future. More and more companies are collecting and storing data that reaches into the TB, even in small companies. The rapid advances in sensors, development tools, and cheap storage means that many people are dealing with hundreds of GB in at least one of their databases. That means for a reasonable RTO, making quick backups, and maintaining good performance, multiple backup files are becoming a necessity.

    Is that really the case? Data volumes are exploding, but you many of you using this feature? I wanted to see how many people have implemented, or at least thought of striped backups. The poll this week is:

    Do you have any databases that benefit from backup to multiple files?

    I’ve consulted with clients that accidentally produced striped backups and then lost one of the files. That’s never a good situation, and it’s bad news to have to give as a consultant. However, I’m sure many of you have consciously implemented striped backups because they can perform better than single file backups and make for quicker restores. Others of you may suspect (or have tested) striped backups will help your systems but haven’t gotten around to setting things up.

    Let us know if this is a common feature you use, or is it still something esoteric that you have no need for. And if you have put striped backups in place, have you tested a striped restore? I certainly hope so. Let us know either way.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.7MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Test Your Situation

    I gave a talk on Transparent Data Encryption (TDE) recently and a number of people in the audience were using the feature. However when I asked how many of them had restored a TDE database, not all hands remained up. When I asked how many people had restored their TDE encrypted backup  to a different server, one that didn’t have TDE enabled, very few hands remained in the air.

    That’s not good, and I certainly hope those people don’t experience a disaster from which they cannot recover. I’m sure they are not alone. I suspect that many of the people managing a TDE database have restored a database this year, and are confident they can do so. However what they don’t know is if they can restore those TDE databases on any other instance, including a newly installed one.

    They’re not alone. I see many, many people implement features they don’t really understand. Microsoft has made it easy to set up replication, clustering, and more in your environment, but without providing some of the robustness and reliability that many people need. The ease of setting up a feature is one thing. The ease of ongoing management and recovery when issues occur is something else entirely.

    I really wish that Microsoft would go further than making implementation easier and include direction for ongoing tasks. When databases are created, ask the user to set up backups and help them create the jobs. When encryption is implemented, do more than display a warning message. Help adminstrators prepare for recovery with templates or jobs that automatically build certificate or key backups. When replication is set up, include a script to rebuild the environment for when it breaks.

    I doubt we’ll get this, and many companies and employees will continue to implement features they don’t understand. You can only help your own situation, and you should be ensuring you understand and can rebuild all the extra features you’ve installed in the event of a disaster.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.3MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Recovering from Bad Deployments

    Someone asked me recently if I stored backups of my database in version control. This person wanted to recover from a bad deployment and use a backup to do so. They felt that keeping a copy of the backup in a VCS, alongside the code being deployed, would be important. It might be, but I said that recovering from a bad deployment isn’t something I want to do with a restore if I can avoid it. Then I was asked how to recover from a bad deployment if you have a busy, 24×7 environment.

    I, of course, answered, “it depends.”

    It does depend on the deployment, but it also depends on your preparation. There are ways in which you can work to minimize the problems that might occur during a deployment. Obviously testing your scripts and deployment process is important, but it’s also good for you to understand how your scripts work and what techniques you can use to rollback problematic deployments.

    There’s a switch in SQL Compare that lets you build a deployment script, and then immediately generate a rollback script. It’s handy, but it’s also not going to always work. If you’ve added a column during deployment, you might not want to just remove it on rollback. 

    However you can prepare for issues, like having a script that might save data in the new column before you remove it. You might choose to copy the table as part of a pre-deployment process (or during deployment), having this copy of the table used in a rollback scenario. You might even bring up a warm copy of your database and prepare to swap entire databases if problems arise. This would allow you time to save and move data that was changed after your deployment, but before your rollback.

    There are lots of possibilities in how you might recover from a failed deployment, but as with many of the solutions that we build in technology, a well thought out plan makes everything run smoother.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.1MB) podcast or subscribe to the feed at iTunes and LibSyn.