Voice of the DBA

Author: way0utwest

  • The Four Different Types of Flyway Files

    This is documented, somewhat, but I wanted to put this down for myself, as the I don’t love the docs and they are hard to sort through.

    Flyway is open source software owned and maintained by Redgate, my employer. There are also paid versions with additional features.

    The Main Files

    There are essentially 4 different types of files you can use for scripts in Flyway. These are:

    • versioned scripts
    • repeatable scripts
    • baseline scripts
    • undo scripts

    I’ll discuss each of these in light detail below. This isn’t intended to replace the documentation, but give a short explanation of each.

    For each type of file, there is a naming standard, which is the same for all. Essentially, there is are four parts. As an example, I’ll use V8.2__add_new_table.sql to explain this. The four parts are:

    • The prefix for the type of script, in this case, a V script.
    • The version. for my example this is 8.2.
    • The separator, always two underscores
    • The description, add_new_table, in my case.
    • The suffix, which is an extension

    Versioned Scripts

    These are the V scripts, which are named with a V to start. These are scripts which are run once on each target, and intended to be those that change the schema. The “V” is the default, but this can be configured. I wouldn’t change this.

    These are typically the CREATE or ALTER scripts that you run to change objects. You can have as much SQL code in here as you’d like. Think of these as a SQL Compare deploy script. One or my objects changes.

    The run once is nice because once you’ve deployed these, you don’t want to run them again. If you write idempotent scripts, you wouldn’t care but most people don’t do that well.

    Repeatable Migrations

    These are used less in the current Redgate paradigm, but these are repeatable scripts. They are designed to run every time you run “flyway migrate”. Again, the default is an “R” script, but this is configurable. Again, don’t change this.

    These scripts don’t have versions because they execute over and over. This could be used for creating or recreating programmable code objects, like view/procedures/functions/packages. They could also be used for places you need to ensure data is always there or has certain values.

    These scripts run after the migrate scripts, so be aware of that.

    Baseline Scripts

    By default, baseline scripts are “B” scripts. This can be changed. These scripts have a database version that you want to start with for some purpose. Like a new development effort or new deployment project.

    These are run for new environments only, and usually contain all the base objects you might need. If you start a Flyway project on a database that has objects, put all the code to recreate this state (CREATE objects and add data) in a baseline script.

    Undo Scripts

    The undo scripts are the “U” scripts. This can be changed, and these should have a version that matches a versioned script. We can run one of these scripts corresponding to those versions to “undo” changes. Since you can’t really undo anything in the database world, these should include code that reverses the action of the V script.

    Be careful with these, especially if you run these more than a few minutes after the V script. Flyway isn’t checking that you won’t lose data.

    I’d only use these after extensive testing in a pipeline and then only if my deployment broke immediately. These are good for putting in a previous version of a view/proc/function and quickly rolling back.

  • Concurrency Challenges Around Schema Changes

    I saw a great question on Twitter from Frank Pachot, a developer advocate of Yugabyte. He wrote: Without thinking how your preferred database deals with it, what do you expect if:

    • session 1 starts to reads table T
    • session 2 drops table T
    • session 1 continues to read

    The choices in his poll were: session 2 waits, session 2 fails, session 1 fails, both fail. My first thought was SQL Server and the default need for session 2 to get an exclusive lock. In that case, session 2 would wait. Most people answered that same way, but then Frank posted a follow-up with a link to his blog. The answer for Yugabyte is that session 1 fails as it gets the message that the table was deleted.

    Leaving aside the decision to drop a table, imagine this is some schema change instead. In the blog, some good points are raised about how to handle high concurrency changes, and the potential problems with having session 2 wait. On a busy system, this could cause lots of blocking as threads stack up behind session 2.

    It’s an interesting read about the challenges of distributed system design and how to handle changes. In some sense, I get that this makes sense, but I wonder where this causes issues. If any schema change on the table by session 2 were to cause an error in session 1, that would be bad. However, does this mean that the database engine must now evaluate whether a column change impacts a query in flight? Then decide to send an error? What about evaluating views or procedures/functions that depend on

    Does this mean that all nodes need to sync up the schema changes quickly, and at a higher priority than data movements? I don’t know exactly how Yugabyte distributes data, and if there are copies on multiple nodes, but I assume there are. This adds complexity to the communication between nodes, which is likely needed. Honestly, if someone drops a table and they should have, we probably don’t want clients getting results. If they do this accidentally, I’d like to know about it quickly.

    The question is interesting, and there are multiple ways to look at this, but I found it fascinating to spend a few minutes thinking about the complexities of data in distributed systems and the challenges involved. This also made me think that the people who keep data safe and fix problems when they occur are invaluable in the modern world.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • The Best Career Advice

    I don’t know that I have the best advice, but this month’s T-SQL Tuesday is asking for people to share what they think is the best advice they’ve been given or have for you. I wrote my own piece, where I noted that learning to say “No” was one of the best things I’ve ever done. Not that I say no to everything, but I do default to no, especially when someone asks for me to tackle something new.

    Actually, it’s slightly more nuanced than that. As I’ve gotten used to my workload, I will say yes to things, and certainly, I’m more likely to commit to one-off things. It’s the longer-term, larger things that I don’t want to agree to do unless I’m sure I can deliver.

    There are lots of other things people wrote. Deb said that you should trust your instincts and realize you can contribute, even if you’re new. Hugo notes the user is often right. Mala is more cautious with work and practices discretion. Rob got the advice to take it slow, learn his job, and figure out what he likes and doesn’t.

    There are lots of other advice, from Pragati telling you to get a mentor to Mikey saying you should find a job you love. If you check out the comments in the invitation above, you’ll see plenty more responses, many with interesting back stories and more details. If you only read through one set of T-SQL Tuesday responses, this might be the one to pick.

    I’m a big fan of actively managing your career. Make the decisions that move in the direction that matters to you. As noted in a few posts, we spend a lot of our lives at work. At times more than we spend with family, so be sure you have a career you enjoy.

    This takes work, but it’s an investment that can repay itself over many years. Both in financial rewards and less stress on a regular basis. Every job is a job some days, but when you enjoy your work, it doesn’t feel like work.

    Steve Jones

  • A New Word: Volander

    volander: n. the ethereal feeling of looking down at the world through an airplane window, able to catch a glimpse of far-flung places you’d never see in person, free to let your mind wander, trying to imagine what they must feel like down on the ground – the closest you’ll ever get to an objective point of view.

    I snap these types of photos all the time. This was me leaving Denver for Fort Lauderdale recently:

    2023-06-05 17_11_39-Photo - Google Photos

    Here was last year above Hawaii:

    2023-06-05 17_13_08-Photo - Google Photos

    and leaving Melbourne:

    2023-06-19 11_45_08-Photo - Google Photos

    As I fly, I sometimes think about the world below me, what it’s like from the ground, what it’s like for people who’ve walked there. Sometimes I plan to go someday and see.

    Often, I know I’ll never get there.

    I flew to Las Vegas a few weeks ago, and there are lots of places I could visit that I see, and some I have. However, much of the mountains above Colorado, the desert outside Las Vegas to the East, these are places I’ll never go.

    But I can dream.

    From the Dictionary of Obscure Sorrows