Voice of the DBA

Tag: DevOps

  • The Journey to DevOps

    Netflix is one of those companies that I find amazing. I was an early subscriber to their mail DVD service, and thought it was amazing how they processed both orders and physical objects with technology. As they pivoted to streaming, I continued to be impressed with their technology growth, from the chaos monkey to their DevOps deployments. They have been an organization often looked to as a model for other technology companies.

    I certainly think there are things to learn from Netflix, as they’ve scaled and build quite a resilient system. They aren’t necessarily worth copying, however, as the problem domain they solve is both narrow and also quite different than that many of us work in. If someone can’t watch a movie, it’s annoying, but they can pick another one. If a customer can’t transfer money, communicate with another user in an app, or schedule a ride, it’s a bigger deal for other problem domains.

    Still, Netflix didn’t build this system overnight. They didn’t come up with amazing DevOps techniques for building and deploying their software from the beginning. It’s been a journey, and they talk about some of the full cycle developer challenges in a recent blog post. This looks at one team’s journey across 6 years, from 2012 to this year. The piece is an interesting read, and it’s not advocating for their particular approach, but rather trying to explain the value that they received from moving to a DevOps model, where they have a group that must run what they build.

    As I try to help customers and clients move to Database DevOps models, I see lots of similarities to what’s in this post. As we look to optimize the entire software development life cycle, this requires a changing of roles and closer cooperation between groups. As I look at the evolution of Netflix, using a centralized group to build tools and then ensuring you have a better staffed development team that works to both build and support their software, ensuring clients get the value (or features) they need quickly.

    I’ve worked in orgs that did this in groups, and for the developers to be involved in operations is an eye opening experience. Developers will learn ensure they think about their design and test more because they don’t like getting woken up. They listen Operations and learn about the ways in which they can better build a system that works. They start to realize “works” is a feature, perhaps the most important one.

    I’ve also found my role as a DBA can facilitate DevOps. I’m often between development and Operations, with a foot in both camps. I help developers build better tools and techniques to work with databases. I understand the impact on production databases, meaning I can help ensure that deployments are smoother, and we avoid risky changes. We build indexes early, install primary keys, test queries for performance, and more. The DBA is the one of the ways in which DevOps can grow in an organization, if they work with the developers instead of against them.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.7MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • No More Downtime

    At one point in my career, I worked for a wealth management firm. We managed funds for various customers, a part of which was making trades in the financial markets. Various brokers and companies used our platform to run their business, and performance was always an issue. At the time we were an NT 4.0/SQL Server 6.5 shop, though we moved to Windows 2000 and SQL Server 2000 while I was there.

    While we were looking at upgrades, a number of potential customers asked why we weren’t using AIX or Solaris or Linux. Management would come to a few of us technical leads to ask, and we usually had to provide some justification. Our success was hit and miss, though we did run into a few companies that were doing real time trading on the Windows platform, and a few of us had the chance to talk to them about how they managed their systems in an age when Windows hosts often needed patching, a requirement of which was rebooting. If you’re interested, the company actually had a fairly server oriented architecture built on top of Windows, essentially managing work by just connecting to whichever boxes were running.

    Things have changed. These days Microsoft has been working hard to build a better Windows OS to power Azure and they’ve done some amazing work. I saw a post on one of the things they’re doing, which is finding ways to patch the underlying OS without disturbing applications, including VMs that are running on the host OS.

    The work is amazing to me, and this is where we should have been going with all operating systems. We ought to be able to patch these without downtime, and certainly without disturbing guests or programs that might be running. As this rolls out to Windows hosts in our data centers, I could imagine an era when we have monthly patches for Windows that never cause downtime for SQL Server.

    Well, I guess there is some downtime. Applications such as the hypervisor are paused, which some of us might consider downtime. However, if the time frame were in the single digits of seconds, I’m not sure many clients would this as downtime. It’s along the lines of a network hiccup or a momentarily busy server. In some sense, this would be a great move forward for HA.

    On the other hand, this would raise expectation. Clients and customers would be less tolerant of downtime, which is something we can’t control with Windows and SQL Server patches. However, it is something we can control with our deployments. A system that is almost never down because of patches would put pressure on us to ensure that our enhancements to the database itself also didn’t cause downtime.

    While there’s no magic in how we make changes to objects in SQL Server, there are techniques that can make changes in stages, perform additive work, and use automation in a DevOps style workflow to limit downtime and lower the risk of making changes. To me, this might be a bigger change than cloud systems. With that in mind, I’d urge many of you to learn techniques to avoid interrupting applications when you deploy changes. There are ways to do this, but it takes some effort and practice to build better skills and habits.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.8MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • More SQL Server Updates

    This was a busy week for updates. I finished adding new builds to the build lists for 2017, 2016, and 2014, the first time I’ve done all three platforms on the same day. To be fair, two updates were on the 19th and one on the 20th, but that’s pretty close to the same day. In the past I’ve had updates for different versions during different months.

    That will not be the case in the future. SQL Server 2017 is on the CU model where we get monthly updates to the product. No more service packs for this version. If you hadn’t heard that, be ready for the new servicing model that consists of just the monthly updates. I wasn’t sure if I liked this at first, but I do appreciate that we don’t get confused with trying to determine which CU1 is needed as there were different patches for RTM, SP1, SP2, etc. Now I’ve gotten used to expecting an update each month and applying them after a few days. So far, Microsoft has mostly been sticking to monthly releases, though the December release got delayed into January. With holidays, that’s expected.

    This week also saw the update of SSMS to v 17.6. You can freely use this version and download the full install or just the update package if you are running 17.x. We are seeing updates on a fairly regular basis to the tool, every couple of months. I’m hoping we continue to see more fixes and improvements aside from new functionality as SSMS has been neglected for too long. I don’t know how stable this version is, but 17.4 was on my machines for a long time without an ill effects, and I think this seems fine for now.

    Updates have always been a part of software, but often they are fairly rare, with interruptions occurring just a few times a year. That’s changing. Redgate releases software every week, with almost every product updating once or twice a month. For someone that uses relatively few pieces of software, all of a sudden it seems like I have a regular set of tools to update across my SQL Server development platform between SQL Server, SSMS, VS, Redgate tools and more.

    That’s good, as I get fixes and increased functionality, but across a set of tools, I might find myself updating something every day. While the software often updates itself if I click OK, I have to make a decision and the process is disruptive and time consuming, especially when I run into required updates, which happens more than I’d like. I like the idea of DevOps, but one of the unintended consequences is that if everyone does it, there are a lot of changes and updates for users to deal with.

    I don’t have a solution, but I do find myself delaying updates more and more, often moving to only take every 3 or 4 updates. Maybe if everyone had an easy way to apply patches with PoSh across systems, or smoother built-in mechanisms I’d feel differently, but for now I probably click “skip” more often than “apply”.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.7MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Are You a Traffic Cop?

    There’s a joke that a DBA’s favorite word is “no”. I think that was certainly true for the individual from whom I first learned about Oracle and SQL Server administration. He constantly pushed back on developers and management who requested changes. He wasn’t rude or abrasive, though I’ve met my share of those individuals. Instead, he patiently would explain and debate the changes and his reasons for not wanting to make them. He had come onboard after a particularly bad deployment affected a significant portion of the company and while management listened to him, development wasn’t pleased.

    One of the analogies I’ve seen for DBAs is that they are a traffic cop. They implement the rules necessary for the safety, protection, and availability of databases systems. This is sometime at odds with the pressure developers feel to implement new features and functions in their applications that require a database. This can, and often is, resolved by using some methodology like DevOps, where the DBA, the developers, and operations staff discuss changes in advance and collaborate to ensure that functionality will not negatively impact data quality, performance, or availability.

    However, this can easily take a negative turn. People often have short memories, and we are all usually busy at work. When a DBA lets a change go through that causes problems, they are reluctant to let the next one go without a thorough review. They slow down and want to double check every new line of code, or worse, they start to second guess their own decisions to allow changes, which can further exacerbate the issues.

    Are you a traffic cop as a DBA? I saw this referenced as a great analogy recently, where the DBA enforces rules, perhaps taking on a bit too much power, and using these rules to limit change. They lock down too much access, even preventing metrics and data from providing feedback to developers. They may even seek to strictly enforce security by the letter of the rule, not the spirit, upsetting and angering others.

    I think good security is important, and often bypassing security for the sake of convenience or expediency is a source of issues, including data breaches. There are policies that are outdated or may not apply in specific situations, and I’d say the sign of a good DBA is that they know when an exception is needed, they document and manage the process, and may even seek to have the guidance changed. They work with others to get things done, constantly looking to push work out to customers, while protecting systems. They may slow work slightly, but a good DBA works with everyone to minimize delays.

    If you’re a traffic cop, I’d argue your employment days are likely numbered. The world is changing, and while security grows more important, the ability to get work done securely and quickly, while working with others, will become more important over time.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.8MB) podcast or subscribe to the feed at iTunes and Libsyn.