Voice of the DBA

Tag: DevOps

  • Concurrency Challenges Around Schema Changes

    I saw a great question on Twitter from Frank Pachot, a developer advocate of Yugabyte. He wrote: Without thinking how your preferred database deals with it, what do you expect if:

    • session 1 starts to reads table T
    • session 2 drops table T
    • session 1 continues to read

    The choices in his poll were: session 2 waits, session 2 fails, session 1 fails, both fail. My first thought was SQL Server and the default need for session 2 to get an exclusive lock. In that case, session 2 would wait. Most people answered that same way, but then Frank posted a follow-up with a link to his blog. The answer for Yugabyte is that session 1 fails as it gets the message that the table was deleted.

    Leaving aside the decision to drop a table, imagine this is some schema change instead. In the blog, some good points are raised about how to handle high concurrency changes, and the potential problems with having session 2 wait. On a busy system, this could cause lots of blocking as threads stack up behind session 2.

    It’s an interesting read about the challenges of distributed system design and how to handle changes. In some sense, I get that this makes sense, but I wonder where this causes issues. If any schema change on the table by session 2 were to cause an error in session 1, that would be bad. However, does this mean that the database engine must now evaluate whether a column change impacts a query in flight? Then decide to send an error? What about evaluating views or procedures/functions that depend on

    Does this mean that all nodes need to sync up the schema changes quickly, and at a higher priority than data movements? I don’t know exactly how Yugabyte distributes data, and if there are copies on multiple nodes, but I assume there are. This adds complexity to the communication between nodes, which is likely needed. Honestly, if someone drops a table and they should have, we probably don’t want clients getting results. If they do this accidentally, I’d like to know about it quickly.

    The question is interesting, and there are multiple ways to look at this, but I found it fascinating to spend a few minutes thinking about the complexities of data in distributed systems and the challenges involved. This also made me think that the people who keep data safe and fix problems when they occur are invaluable in the modern world.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Agile West–How Do I Get Started with Database DevOps

    This week I had the chance to deliver a talk at Agile West in Las Vegas. I linked some resources on my blog, and feel free to check them out.

    Tl;Dr – Start with deployments

    After the talk, I had an interesting question from someone in the audience. This person had a lot of developers writing code and then sending to a DBA, who executes it in SSMS. This was for SQL Server development, and that’s a common way many companies deploy database changes.

    Even companies that have software developers who embrace DevOps will still tend to work this way.

    This is error prone, inefficient, and it’s not the way to build better software. We know that DevOps produces better software if you adopt it, and we know that you can’t forget the database. That’s integral if you want to be a high or elite performer.

    My advice for many companies is that they start with deployments. If you have some manual process or DBAs involved, often they’re just running a script you produced in some way. What I’d do is start adding Flyway Community  (or Teams/Enterprise) in and putting those scripts into a folder (hopefully in Git) and naming them as appropriate. That’s a small change for DBAs or Ops people deploying code, but it starts to enable a process.

    From here, I can alter this process to use tooling (Azure DevOps, Bamboo, Jenkins, etc.) and continue to put scripts in a location. Then I can work backwards and start getting developers to build better processes for capturing and saving code.

  • Bringing DevOps to the Database – SQL Saturday South Florida 2023 Demos

    With some projector issues at SQL Saturday South Florida 2023, I had to lecture without demos this past weekend. I told everyone I would record the demos, so I’ve done that. The three demos are shown below, and if you download the deck, you can see where they fit into the talk.

    Powerpoint: Bringing DevOps to the Database.pptx

    Demo 1- Getting code into a VCS
    Demo 2- Continuous Integration

  • A Quick Trip to Agile West

    Redgate is sponsoring Agile West this year, and the marketing team asked me to give a talk on database DevOps technologies.

    I have a keynote talk, called Don’t Forget the Database, that I’ll be delivering on Wednesday. This covers some of the challenges of why the database is harder than software, but doesn’t need to be. I show some demos of zero downtime deployments as a part of this talk.

    Likely I’ll record the demo and talk over it, as I have seen that technique work well and it keeps me from fumbling around.

    My wife is coming with me. This is a quick Tues-Thur trip for me, and easy. Las Vegas is about an hour flight for me, and I only have a few commitments, so this is a chance for the two of us to see a show or two and get out of town for a short break.

    I’m lucky that some of my business travel allows my wife to come around, and it’s worth the expense to bring her and make the trip more enjoyable. That makes it easy for me to handle the 20-30 trips a year I make.

    If you’re at the show, stop by the Redgate booth or my talk and say hi.