Voice of the DBA

Tag: DevOps

  • Requiring Technical Debt Payments

    I was working with a customer recently that is trying to improve their processes. This was a large company, over 100,000 employees, though most of them aren’t in the technology area. However, across many divisions and groups, there are a lot of developers and operations personnel who have tended to work in silos, managing their own applications and systems in disparate ways.

    In other words, doing software development the way most companies do it.

    I had been working with one group to streamline and standardize some of their software practices to implement more of a DevOps flow to smoothly build, operate, and update their systems. They’ve had some success and other groups noticed that this set of teams is very efficient. They aren’t DevOps like a lot of the articles you read. They still have development and operations, but the groups work closely to ensure efficiency.

    They started to get requests to onboard other teams into their flow as the management of this group has been advertising their success. Other groups want to implement Continuous Integration, get database unit testing and static code analysis setup, implement gates for approval, and more. The Operations team manages most of this and is happy to help other groups.

    But

    They require some things to be in place, some of which are cleaning up technical debt. Not all debt, but certain things that create additional risk or instability. Before they onboard anyone, they don’t want to take on a codebase that is difficult to manage. A lot of this debt isn’t difficult, but they want some good coding practices implemented. They require integrated security or a waiver from InfoSec. They want explicit index names, not system-generated ones. They want permissions granted to roles, not users. Not big things, but little items that make a system less maintainable and understandable.

    The same things a lot of us let creep into our codebase over time.

    On one hand, I thought this was an idea that would slow adoption and allow many groups to continue to operate inefficiently. They won’t clean up code. On the other hand, this might be the lever that helps create a better run environment across the organization. This might help them smooth their upgrade cycles, let staff change between projects, and more importantly, reduce the overhead of communication and work between teams.

    I don’t know how this will work over time, but I am interested to see what happens.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • What is a Failed Deployment?

    When talking about DevOps, the goal is to produce better software over time. Both better quality as well as a smoother process of getting bits to your clients. There are a number of metrics typically used to measure how well a software team is performing, and one of the things is Change fail percentage. This is the percentage of deployments that causes a failure in production, which means a hotfix or rollback is needed. Essentially we need to fail forward or roll back to get things working.

    For most people, a failed deployment means downtime. I’ve caused a service to be down (or a page or an app) because of a code change I made. This includes the database, as a schema change could cause the application to fail. Maybe we’ve renamed something (always a bad idea) and the app hasn’t updated. Maybe we added a new column to a table and some other code has an insert statement without a column list that won’t run. There are any number of database changes that might require a hotfix or rollback and could be considered a failure.

    However, some people see an expanded definition. If a service is degraded (slower), is that a failure? Some people think so. If we change code in a database (or indexes) and see performance slow down. In that case, is this a failed deployment? Customers would think so. Developers might not like this idea, at least not without some sort of SLA that might allow for some things to be a little slower. After all, slow is still working, right?

    What if I don’t notice a problem? Imagine I add a new table/column, and the app starts accepting data and storing it. What if we are supposed to use this data downstream, and we don’t notice it is being aggregated incorrectly by a process until many days later. Perhaps we’ve performed some manipulation or calculation on our data and the result isn’t what we wanted. It might not be incorrect, but maybe it’s ignoring NULLs when we want NULLs treated as 0s.

    Is that a failure? If I deploy today and Bob or Sue notices next week that the data isn’t correct, that’s a failure. I don’t know I’d count downtime from today until next week, but from when Bob/Sue files a ticket, the clock starts on calculating the MTTR (mean time to recovery).

    I don’t often see database deployments failing from the “will it compile on the production server” standpoint. Most code gets tested on at least one other system, and with any sort of process, we catch those simple errors. More often than not, we find performance slowdowns or misunderstood requirements/specifications. In those cases, some of you might consider this a failure and some may not. I suppose it depends on whether these issues get triaged as important enough to fix.

    While I might have a wide definition of deployment failures for most coding problems, I don’t for a performance slowdown. Far too few people really pay attention to code performance and are happy to let bad code live in their production systems for years.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • A DevOps Workshop Tomorrow in Atlanta

    Tomorrow is the Redgate DevOps Day in Atlanta. You can still sign up, so do that if you can make it. Here’s the rough outline

    • Vision Session – w/ Advocate Steve Jones
    • Redgate Flyway hands-on Session
    • 6 ways to Elevate Database Monitoring
    • Whiteboarding Session
    • Test Data Manager Demo Session
    • Networking/Happy Hour

    Lunch is provided and I’m looking forward to this. I’m traveling from Baton Rouge today to Atlanta (probably right now), coming from a customer visit.

    Register and come see me in Atlanta tomorrow

  • Database DevOps Recommendations

    The DORA organization is constantly researching how to better produce software at any organization. This is similar to work done by Carnegie Mellon Software Engineering Institute many years ago. Both groups are trying to determine what things help engineers work better and produce high-quality software.

    On the DORA site, there is a database change management page, where they recommend approaches to managing database schemas. The main thing they talk about is treating all schema changes as migrations, which is something some people do. However, many teams also like a desired-state-configuration approach, where they just deploy all the changes from dev (or QA) to prod in a state-based flow. Both can work, but I do think as software matures (and becomes legacy), migrations are preferred. The article lists lots of frameworks in different languages. Flyway is among them, which is the product on which I work and sell at Redgate. If you haven’t looked at a migrations framework, I’d recommend you do so. They do really give you a tremendous amount of control and flexibility. There are trade-offs, so I’m not blindly recommending this approach, but it’s worth educating yourself on how migrations frameworks work.

    The other thing they recommend is using a zero-downtime change approach. They list a few strategies that you can use make changes without impacting clients. None of these are hard, but they require a little coordination with application developers, as well as some patience in splitting changes across multiple deployments. They also require good coding practices, so many of us have a few habits to unlearn.

    DORA does also recommend considering a NoSQL database since you don’t have downtime from schema changes, but my view is that you’re likely making trade-offs that aren’t worth in. If you read that paragraph in the article, my guess is many application developers read everything but the last sentence, or they dismiss that last sentence. Then they just start developing against a NoSQL database platform and expect it to magically be better. It might be, but it might not work as well.

    The one thing I wish they had a better answer for was how to measure our work as database developers. This section doesn’t give you any concrete things to measure, just some suggestions, which are often hard for people to implement in their environment. Most of us don’t have salary or time numbers for others, and need more help in trying to determine how to measure changes. Maybe the one good measurement to aim for is 100% of database changes made by automation, rather than manual execution.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.