Voice of the DBA

Tag: continuous delivery

  • The Bad Data Shutdown

    I’m a car guy. I like cars, I like driving, and I’ve spent a lot of time and money over the years on vehicles. I’ve swapped and enjoyed a few dozen automobiles as part of my life. If you are on Twitter, you might occasionally see @BrentO and myself go back and forth on some car topic, usually Porsche related. This usually results in an hour or so of life wasted on dreaming of a new car (including getting distracted while writing this piece and pasting that last link in, where I spent quite some time drooling over the Macan).

    Recently there was an issue with the navigation system in Lexus vehicles. Apparently bad data was sent during a software update, which is not exactly what you want to happen in a car. I’ve had a few modern vehicles, some of which would be quite handicapped if they onboard computer were frozen or rebooting. In my current vehicle, this would cause issues with climate, navigation, entertainment, and potentially other systems. After all, I suspect many things from door locks to speed control are all integrated together. Certainly the drivetrain is as an open door will automatically shift my car from drive to park, at least at low speeds.

    As we move to more drive by wire, bad data or bad software that disrupts the computer systems could be very dangerous. It’s not just updates, but this could even be some internal Denial of Service issue from a USB device or bluetooth connection. In this case,  Lexus acknowledged the problem, which I’m glad to see. The Internet ensures that problems can be reported from many users quickly and very publicly. That makes it hard to deny a widespread problem.

    Delivering updates across wireless links is great. It’s cheaper for everyone, saves time, and owners appreciate convenience. However, moving to this model often requires some sort of continuous delivery (CD) process, which should also allow for rolling forward, and releasing fixes for problems. However, if the updates you deliver cause the system to cease functioning, then this doesn’t help. At the last, your QA system needs work and you don’t have a well designed software delivery process.

    Various companies are getting better at delivering updates to our systems without downtime, but there’s still work to be done. The smaller your domain of clients, the easier this is. For many of us that work on small systems, an application server or two and a database, we can certainly get much better at ensuring our updates are tested, and more importantly, that we can quickly deploy a second patch if we find an issue. That requires engineering a process that is known and stable, with the ability to respond quickly. For larger systems, with many clients, you need a really solid engineering and deployment process.

    Above all, however, no matter what your deployment mechanism for updates, you need to be sure that any data you include is at the quality level you’d expect would be delivered to you.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.7MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Software Maturity with Release Management

    For most of my career, releasing new versions of software that I or others had written consisted of manually running scripts or copying files, maybe using an installer once in awhile, but that was rare. In many cases, the process of actually deploying the software involved a developer sitting with an operational person, especially if the operational person had been burned by previous deployments that were poorly executed. Once burned, you usually require the person building the scripts to be available when they are run.

    That’s a poor way of managing the release of software. We write code to allow computers to repeat the instructions we’ve given them over and over. Computers are good at that, and they don’t make mistakes the third or fifth or hundredth time they execute the instructions. We might not give them the correct instructions, but computers don’t make mistakes and do execute what we code. If we find mistakes, we fix them and let the computer then execute them again.

    However it seems that for many individuals that release software, they want to do it manually. Especially when the release is to just one machine (or database) inside their organization. And time and time again, I find that these people spend time building scripts, correcting them, and still finding mistakes it production deployments that they need to fix. Often these fixes are made on the fly, with no testing.

    There’s a good mantra in software that if something is painful, do it more often. Make it easier with automation and execute it multiple times. Learn to code it better. There’s a whole industry that has grown up to help you release software more often, multiple times, and in a controlled manner. These are release management (RM) tools like Octopus Deploy and TFS Release Management. I’ve been using both of these internally, and I couldn’t imagine not using an RM tool at any point in the future. It should be easy to deploy software from one machine to another, and the process should be repeatable. That’s a sign of mature software development.

    If you’re not using a RM tool, I’d urge you to check one out. Most let you get started for free, and a small Proof of Concept (POC) project is the way to get started. Once you learn the ins and outs of how you really deploy software, you will never want to do it manually again.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.7MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • The Next DLM Demo Webinar–Feb 23

    A quick note that I will be presenting a webinar on Feb 23rd at 10:00am MST. That’s noon EST and 5:00pm GMT. You can sign up here if you’d like to watch. This is similar to other DLM webinars I’ve done that look at the Redgate stack of products that help you develop database software in a smoother fashion.

    I am changing things up a bit. This time the stack is:

    • Git – version control
    • Team City – Continuous Integration or Build server
    • Octopus Deploy – Release Management tool.

    As with the other webinars, I’ll show how I track the version of the database in each environment, then show how to make changes in SSMS, commit them to my VCS with SQL Source Control, and then deploy them in an automated fashion across various environments. In this case, I’ll have this flow of environments

    • development
    • integration
    • test
    • acceptance
    • production

    It’s an impressive demonstration. As I’ve watched this grow at Redgate, I am impressed with how well things work. I wish I’d have had these tools when I built database software earlier in my career.

  • Is Your Deployment Process Perfect?

    I’m sure your deployments aren’t perfect. I’ve had many successful ones, but I have always found some flaws in my systems over time. I’ve also found that the less flexibility I allow, the more likely things are to succeed.

    However less flexibility isn’t what most companies or clients want. In fact, they want the opposite. I know most developers want this as well, without constraints on what they can do.

    I think Redgate has been working to make things easier and more reliable for most companies. Certainly we have work to do and there are places to improve our products, but our DLM work on various products is really designed to reduce, or eliminate, silly mistakes that cause issues. We’ve tried to help companies set up a process that limits regression issues.

    I saw this short video from Redgate, where the lead Data Anlyst at Yorkshire Water talks about some of the improvements in their development process come about by implementing a DLM process. Over a couple of years, they’ve gotten many of their projects into a Continuous Delivery stack and that’s just the way they do things.

    Yorkshire Water

    The time savings are the big issues. There is less copying of production data around, more confidence in development, and as Shaun notes, time is money.

    If you think you want to build a more repeatable, reliable database development process, I think we give you a few options. The traditional, work in SSMS approach with SQL Source Control, or our Ready Roll work in Visual Studio. I’m particularly excited about Ready Roll as that’s how I’ve done a lot of development in the past.

    And, of course, I think our DLM Automation suite, allowing you to deploy changes to a CI environment and build deployment packages works well.