Tag: DevOps

  • What’s Your Smoke Test?

    Many DBAs and operational staff regularly stress over software deployments to production systems. Even when the administrator has built and tested the deployment scripts, there is still a nagging fear that something will be missed, incurring downtime for systems. I think automation, building an ALM And DLM process, as well practicing deployments in other environments is the way to alleviate concerns, but that’s a discussion for another day.

    However, I had a question this week that is related to the deployment process. I’m curious, do you smoke test your production deployments?

    Perhaps the first question is do you know what a smoke test is? If you don’t, maybe you want to read a bit about smoke tests. If you know about smoke tests, do you have anything more than testing if a server or service is running?

    If you don’t, that’s fine. In fact, throughout most of my career our smoke tests consisted of checking that we could connect to a server or database. In fact, in most cases, we did this manually. We’d deploy some change or patch, then see if we could connect to the server. Sometimes a person would connect with an application, but not much more than logging in. Since many apps were fairly simple client-server applications, this was usually sufficient, but there were times we had problems with not all changes being deployed. A few times we didn’t realize this for days.

    If you deploy changes, there should be some small things that you check as an method of validating that your systems are working correctly. There may be initial checks that machines are actually running. If those checks fail, there’s no point in going further. However, if the basic connections work, perhaps you should have a few critical functions that can be called and ensure that basic functionality of the application is there. Or that your new changes actually are deployed. If you have complex, interconnected systems, maybe you want a few smoke tests that you can automate and quickly ensure everything is running as expected and communicating.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.0MB) podcast or subscribe to the feed at iTunes and Mevio

  • A Buggy Release

    I definitely believe in a DevOps process, though a thoughtful, incremental one. I think this is the best way to develop software, whether you release every day or every year. Yes, you can implement DevOps and release once a year. You just end up tracking, testing, communicating, and being ready for that once a year release. Of course, I bet you don’t release once a year since I’m sure you’ll patch the system at least once.

    One of the core principles of DevOps is to use automation where you can. Remove humans and ensure that repeatability is possible for moving software from one machine to the other. Communicate, test, and then alter your process to work better. This requires the monitoring and input of humans to examine the process, but they shouldn’t be involved in deployments other than approving them. It’s too easy for an individual to make a mistake.

    However, DevOps isn’t a panacea for building better software. Witness the issues at Knight Capital, where they went from having $364mm in assets to losing $460mm in 45 minutes. Mostly because of a problem deployment, where an engineer didn’t deploy code to all the servers in their farm. Certainly a clean deployment to every system might have prevented this, but the reuse of old flags in code is problematic, as is leaving old code around that could be executed.

    In addition to moving to a DevOps mindset, I’d also say that you should be sure that you follow good software development practices as well. Clean out old code (including database code) and be very, very careful about reusing any part of your software, including flags, for a new purpose. It’s far, far too easy to make mistakes here.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.5MB) podcast or subscribe to the feed at iTunes and Mevio .

  • Building locally from VSTS

    One of the things that you need with a Continuous Integration server is that ability to build your software on some computer system and verify things work. With Visual Studio Team Services, using Visual Studio online, this seems to be a challenge.

    The VSTS team has thought of this and includes the ability to target your builds on a hosted system in the VS cloud, where you don’t need anything installed. However the hosted build agents and servers have a limited amount of software that’s supported. SQL Server isn’t on that list.

    However, there is another option. If you go to the control panel for your account, and click the Agent Pools tab, you’ll see something like this.

    2016-06-28 18_03_29-Agents for pool Default - Microsoft Team Foundation Server

    Notice the “Download agent” link. That’s what you want. As you can see, I’ve been testing and I have agents actually setup and registered on 4 machines. Here I’m going to add a fifth.

    Once I download the file, I’ll extract it to see a list of files and folders.

    2016-06-28 18_01_52-agent

    What I first want to do is configure the agent, so I’ll enter ConfigureAgent from a command prompt. Note, I need to do this as an administrator level command prompt.

    2016-06-28 18_08_35-cmd - ConfigureAgent.cmd (Admin)

    In my case there are some settings, but I’m overwriting them as I rebuilt my machine. Once I hit Enter, I then get the chance to Authenticate.

    2016-06-29 09_18_49-

    After this, the old Agent appears. However, since I’ve rebuilt and rename this machine, I’ll change it. I answer a few more questions about configuring the agent properties. At the end I’ll also authenticate to the Azure cloud once again.

    2016-06-29 09_21_09-cmd (Admin)

    Now that things are configured, I can run the agent. I could set this as a service, but I prefer to know the agent is (or isn’t running) and see the output. I have set this as a service before, and it works fine.

    All I have left to do is RunAgent.cmd and I have an agent running locally that takes instructions from VSTS.

    2016-06-29 09_26_45-cmd - RunAgent (Admin)

    If I go back to my control panel, I see a new agent.

    2016-06-29 09_32_41-Agents for pool Default - Microsoft Team Foundation Server

    I can also trigger a build. I happen to have one setup for a project that points to local instances. Here’s the build definition, which uses the VSTS Extension from Redgate to build a database.

    2016-06-29 09_34_05-Microsoft Visual Studio Team Services

    I can click “Queue Build” and a build will start.

    2016-06-29 09_34_18-Microsoft Visual Studio Team Services

    I see the build running in the console:

    2016-06-29 09_34_31-cmd - RunAgent (Admin)

    And online in VSTS if I want. The Build Agent sends logs back to the VSTS service as it’s working.

    2016-06-29 09_36_19-Microsoft Visual Studio Team Services

    This is a basic way to get a VSTS build working on your local machine with an agent. There is a lot more to configure if you need to, and if you need multiple agents, you can certainly pay for them with a different VSTS plan.

  • Baby Steps to DevOps

    I’ve been re-reading the book, Continuous Delivery, as part of the current San Diego Technology Immersion Group monthly meeting track. This book was the focus of the first meeting in May, and will continue for the next couple months. Continuous Delivery and the related, DevOps, are fascinating concepts, but mostly I find the discussions really interesting for me to see and hear how others view the DevOps, Continuous Integration (CI), and Continuous Deliver (CD), and where they apply in each person’s organization.

    One of the things I’ve noticed is that lots of people get overwhelmed with DevOps. They hear stories of how others are running an extremely efficient software development shop and think they couldn’t implement CD in their organization. Or they think that CI/CD/DevOps is just a way of building software that’s so far removed from their own experience. However, DevOps isn’t really anything new or special. It’s not even one thing, as there are many divergent views of what DevOps means. However, plenty of people have been applying and adhering to the various definitions of DevOps for decades, just viewing their process as efficient, effective, and empowered.

    I use DevOps and CD somewhat interchangably, as they often proceed along the same path and each is a intertwined with the other. Note that going to a CD process doesn’t mean that you release every day/hour/whatever. If means that you release when you want, which could still be every few months. It’s just that you have a process set up to allow smooth process flow.

    As I talk to people who are looking to build a DevOps process, and I hear about more and more of them all the time, one thing I stress is to take things slow. You won’t build your DevOps process, whatever that looks like, this month. In fact, what you should expect to do is learn as you go, and whatever your vision of the process is today will change. You will learn, grow, change the way you do some things, and also change the way you implement new processes, even if you’re copying how other companies have implemented their own system.

    Most importantly, what I try to stress is to slowly move to CD (or DevOps). Look at the process you have now and just change one thing. Let’s imagine that you:

    • manually track changes to the database in email
    • then build a change script manually
    • then have a DBA apply those changes
    • then let QA run tests with an application
    • then modify the script that goes to production

    Leave that process as is. Let’s implement one thing. Add some version control that will let you build a base for other changes. Your new process would be (changes in bold).

    • manually track changes to the database in email
    • then build a change script manually
    • store the change script in a VCS
    • then have a DBA apply those changes
    • then let QA run tests with an application
    • then modify the script that goes to production

    Or change to this instead?

    • manually track changes to the database in email
    • then build a change script manually
    • then have a DBA apply those changes
    • then let QA run tests with an application
    • then modify the script that goes to production
    • track all changes to production in an automated fashion (hopefully using a tool)

    Over time, you can make another step. Maybe you use something like SQL Source Control or ReadyRoll to build change scripts and put them in a VCS. Or perhaps you build a process that takes the changes from the VCS and applies them to a test database without requiring the DBA to check things out and execute them. Both are valid ways to evolve your process. (Disclosure, I work for Redgate Software, maker of those products). I’m giving you a few suggestions here, but there are multiple ways, methods, and tools you could use.

    And that’s the key. You evolve your process, evolve your communication, learn to work together to accomplish goals. Not big goals, little goals. Add small items to your process as you realize they can be automated. Change communications in ways that help ensure everyone knows where to find code and how to deploy it. Or everyone knows how to set up environments (hopefully automated).

    Slowly get better and better, and before long you’ll find that you can make changes in smaller batches, and ensure those changes can be deployed at a rapid pace. Or maybe you’ll have confidence to begin moving to a feature flag architecture so that you deploy changes well in advance of users being aware of them. Note, they’re not perfect or the solution.

    DevOps is about learning and growing. This is really the same basic principles that have driven Six Sigma and Kaizen and various other philosophies that recognize that we can do better over time. Make small steps, measure, learn, and improve.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.9MB) podcast or subscribe to the feed at iTunes and Mevio .