Voice of the DBA

Tag: software development

  • Recovering from Bad Deployments

    Someone asked me recently if I stored backups of my database in version control. This person wanted to recover from a bad deployment and use a backup to do so. They felt that keeping a copy of the backup in a VCS, alongside the code being deployed, would be important. It might be, but I said that recovering from a bad deployment isn’t something I want to do with a restore if I can avoid it. Then I was asked how to recover from a bad deployment if you have a busy, 24×7 environment.

    I, of course, answered, “it depends.”

    It does depend on the deployment, but it also depends on your preparation. There are ways in which you can work to minimize the problems that might occur during a deployment. Obviously testing your scripts and deployment process is important, but it’s also good for you to understand how your scripts work and what techniques you can use to rollback problematic deployments.

    There’s a switch in SQL Compare that lets you build a deployment script, and then immediately generate a rollback script. It’s handy, but it’s also not going to always work. If you’ve added a column during deployment, you might not want to just remove it on rollback. 

    However you can prepare for issues, like having a script that might save data in the new column before you remove it. You might choose to copy the table as part of a pre-deployment process (or during deployment), having this copy of the table used in a rollback scenario. You might even bring up a warm copy of your database and prepare to swap entire databases if problems arise. This would allow you time to save and move data that was changed after your deployment, but before your rollback.

    There are lots of possibilities in how you might recover from a failed deployment, but as with many of the solutions that we build in technology, a well thought out plan makes everything run smoother.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.1MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Citizen Programmers

    The Citizen Programmer, a piece at Simple Talk, really made me stop and think a bit. On one hand, the idea of building a platform that enables any end-user to perform rudimentary programming is a noble goal. If you can do that, then you can dramatically reduce the costs associated with waiting for developers to build applications.

    On the other hand, the article praises the Visual Basic of the 90s, which allowed almost any one to build an application.

    And that was a problem. Despite the tremendous number of applications built, far, far too many of them were poorly written, prone to crashing computers, unable to scale to more than a single user in many cases, and were almost un-maintainable over time. These applications removed people from their knowledge work, having them spend time programming instead of their regular job. That might be good in some cases, where people had talent and desire to build software. It certainly forced the person coding the system to better understand the idea behind the work that they were doing.

    However in many cases, I think we might have ended up wasting lots of time. Certainly the people spending more and more time maintaining an application weren’t necessarily getting more work done. People dealing with buggy software might have been doing their job more slowly overall. There were also the problems with turning a VB application over to professional programmers who were loathe to work on it, and perhaps did little work to keep the application running. A lack of responsiveness from technology departments might restart the whole process with another poorly written VB application, each one a custom work of art that stumbled along inside of a business.

    I do think that giving tools to enable end users to perform some of their own analysis and review of data is important. I like the idea of PowerPivot and other tools that let users query data and build their own reports. I don’t know if we need professional developers for every piece of software, though I am sure we need software to be written faster and in a more agile fashion. In all cases, however, I know we do need professional DBAs to manage data and ensure it’s protected and intact as the speed and scale of our systems continue to grow.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.7MB) podcast or subscribe to the feed at iTunes and LibSyn. 

  • Always Canary

    Deploying software is hard. Despite decades of experience working to update software smoothly, it seems that many of us tasked with that responsibility find ourselves nervous about making changes. This is especially true for database changes, where we must not only update the schema and code, but we must preserve the data as well. If only we could drop and recreate the database during every deployment.

    One of the ways in which you can make changes in a live environment is to have a second system already in place and running. With a second system available, you can stop the secondary system from handling user requests, make changes, test them, and if things are working well, you can have users connect to the updated system while you take the one one offline and perform your updates there.

    This is the same process that many people use to patch clustered database servers, though the “rolling upgrades” are usually done for patches from Microsoft. It would be great if we could easily apply this same system for our schema upgrades. Deploy our changes to the passive node, have the application able to connect to this node through some switch, test the changes, and if things are working, switch users over. The problem is that our databases on the passive node aren’t live.

    Really the ability to deploy database changes to a live system and perform checks requires a strong architecture in both the database and application to support this. Your code must be able to handle additions to the database objects without breaking. The use of feature flags, turning functionality on and off with switches, allows certain users to test new features without most clients being aware of the changes.

    It’s a pattern that more and more software companies are using when they want to deploy changes rapidly to systems for limited numbers of users. However it’s not a design pattern that is widely used. I hope we get more sample applications and reference architectures in the future that will help developers and DBAs code more robust and resilient applications, allowing changes to be deployed live without affecting every user.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.5MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

  • The Value of Code

    It seems that many businesses can operate with a large margin for error. Managers can make some poor decisions, processes can be a bit inefficient, and employees can be disengaged, yet companies can still turn a profit. Non-profits, and certainly government organizations, can run without optimizing their work, much to the chagrin of many. However I continue to find companies that look for ways to not only become more efficient, but also to fundamentally alter their business models by better understanding their particular market. They can turn profits (or operate efficiently) far in excess of the averages for their industry with one tool: software.

    There’s a belief in many of these highly optimized companies that data contains lots of value, but in order to understand and harness the information contained within data, an enterprise must be willing to invest in better software. That requires better software developers. In a forward looking enterprise, that means lots of opportunity for those of us that work with data and build software.

    Those of us that create technical solutions need to learn to better interpret data and find ways to help our clients find and understand the patterns revealed by our software. Those of us that primarily manage data, need to better understand how software renders and showcases information compiled in our databases. We truly have the ability to reshape the way organizations run with our technological skills.

    Computing power continues to grow, accessibility to information improves, and our enterprises become more dependent on our systems. Now is the beginning of an era where we will have tremendous influence in helping change the way business works. I’d urge all of you to continue to look for new ways to manipulate data and find ways that showcase the value that software can bring to your organization. Even if your current company doesn’t value your skills, I bet another one will sometime in the future.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.3MB) podcast or subscribe to the feed at iTunes and LibSyn.