Tag: Flyway

  • Friday Flyway Tips: Searching Migrations

    When a Flyway Desktop (FWD) project (or Flyway project) has been around for a long time, there can be a lot of migration scripts. That can be a pain for users, but there is a way to find your changes or limit what you see. This post looks at how to do this starting with Flyway Desktop 6.5.4.

    I’ve been working with Flyway Desktop for work more and more as we transition from older SSMS plugins to the standalone tool. This series looks at some tips I’ve gotten along the way.

    A Busy Screen

    I’ve got a lot of migrations in this project. You can see below in the image, noting the scroll bar goes up and down here.

    2023-08-10 12_30_54-Flyway Desktop

    Many customers have many more migrations, which could run into the 100s. I’ve worked with large Oracle migrations in the past where there were 100s for an upgrade from v4 to v5. The project would have been 1000s in its lifetime.

    Use Search

    At the top of the Migrations tab, there is a new search box. This was added somewhere in Flyway Desktop 6, and I just noticed it. However, it’s very, very handy. If I start typing in there, like “New Table”, then I see just those migrations.

    2023-08-10 12_33_47-Flyway Desktop

    I can even search versions, like the specific migrations in the 3.8 release that I am wondering about.

    2023-08-10 12_35_55-Flyway Desktop

    This works by checking the version and description fields. The rest aren’t as useful, but it does mean if you come up with some standards for the description in your team, you can easily use this to find changes in your project.

    Try it today. If you haven’t worked with Flyway Desktop, download it today. There is a free version that organizes migrations and paid versions with many more features.

    Video Walkthrough

    I made a quick video showing this as well. You can watch it below, or check out all the Flyway videos I’ve added:

  • The Four Different Types of Flyway Files

    This is documented, somewhat, but I wanted to put this down for myself, as the I don’t love the docs and they are hard to sort through.

    Flyway is open source software owned and maintained by Redgate, my employer. There are also paid versions with additional features.

    The Main Files

    There are essentially 4 different types of files you can use for scripts in Flyway. These are:

    • versioned scripts
    • repeatable scripts
    • baseline scripts
    • undo scripts

    I’ll discuss each of these in light detail below. This isn’t intended to replace the documentation, but give a short explanation of each.

    For each type of file, there is a naming standard, which is the same for all. Essentially, there is are four parts. As an example, I’ll use V8.2__add_new_table.sql to explain this. The four parts are:

    • The prefix for the type of script, in this case, a V script.
    • The version. for my example this is 8.2.
    • The separator, always two underscores
    • The description, add_new_table, in my case.
    • The suffix, which is an extension

    Versioned Scripts

    These are the V scripts, which are named with a V to start. These are scripts which are run once on each target, and intended to be those that change the schema. The “V” is the default, but this can be configured. I wouldn’t change this.

    These are typically the CREATE or ALTER scripts that you run to change objects. You can have as much SQL code in here as you’d like. Think of these as a SQL Compare deploy script. One or my objects changes.

    The run once is nice because once you’ve deployed these, you don’t want to run them again. If you write idempotent scripts, you wouldn’t care but most people don’t do that well.

    Repeatable Migrations

    These are used less in the current Redgate paradigm, but these are repeatable scripts. They are designed to run every time you run “flyway migrate”. Again, the default is an “R” script, but this is configurable. Again, don’t change this.

    These scripts don’t have versions because they execute over and over. This could be used for creating or recreating programmable code objects, like view/procedures/functions/packages. They could also be used for places you need to ensure data is always there or has certain values.

    These scripts run after the migrate scripts, so be aware of that.

    Baseline Scripts

    By default, baseline scripts are “B” scripts. This can be changed. These scripts have a database version that you want to start with for some purpose. Like a new development effort or new deployment project.

    These are run for new environments only, and usually contain all the base objects you might need. If you start a Flyway project on a database that has objects, put all the code to recreate this state (CREATE objects and add data) in a baseline script.

    Undo Scripts

    The undo scripts are the “U” scripts. This can be changed, and these should have a version that matches a versioned script. We can run one of these scripts corresponding to those versions to “undo” changes. Since you can’t really undo anything in the database world, these should include code that reverses the action of the V script.

    Be careful with these, especially if you run these more than a few minutes after the V script. Flyway isn’t checking that you won’t lose data.

    I’d only use these after extensive testing in a pipeline and then only if my deployment broke immediately. These are good for putting in a previous version of a view/proc/function and quickly rolling back.

  • Flyway with Multiple Folders

    A customer asked about how they could organize their migration scripts in different ways to manage them and worried about it being complex. I decided to test a few things. This post looks at using multiple folders for scripts.

    Setup

    Flyway is controlled by a number of options. These can be in a configuration file or they can be passed into the command line. I’m lazy, and I like things self-contained, so I tend to drop things into the flyway.conf file and specify this in the command line.

    I’m also lazy about command lines, so I created a fw.cmd file with this in it:

    flyway migrate -configFiles=”./flyway.conf”

    Then I set up a few folders on my system. These were subfolders in a repo called SQL1, SQL2, SQL2. I know, I’m not creative. Here are the folders I set up. SQL1 has 3 scripts, each containing a CREATE TABLE statement. In this case, I am creating Table01, Table03, and Table04.

    2023-05-18 12_28_18-sql1

    I do something similar in SQL2, with table Table02, Table10, and Table11.

    2023-05-18 12_28_22-sql2

    SQL3 gets the 5th and 6th tables.

    2023-05-18 12_28_25-sql3

    I did this to check order of operations, as well as using multiple folders. For reference, each file looks similar to this, with numbers changed:

    2023-05-18 13_04_21-V010__tenthtable.sql - ARISTOTLE.foldertest (ARISTOTLE_Steve (60))_ - Microsoft

    I then edited my flyway.conf file. You can name this differently, you just need to pass the appropriate file to flyway when you execute it. By default, flyway.conf is looked for. I changed the variable for flyway.locations to be a comma separated list. In my case, I entered this:

    flyway.locations=filesystem:sql1, filesystem:sql2, filesystem:sql3

    Make sure you uncomment the # from the line.

    Once this was done, I created an empty database, called foldertest. I then ran my fw command and saw this:

    2023-05-18 12_30_45-cmd

    Success, with all my tables created.

    Flyway is set up to search the locations path and load the files and execute them. You can organize your scripts into different subfolders if you want to, and flyway will sort out the ordering, if you’ve named them correctly.

  • Flyway Desktop PoC–Adding a Shadow and Baseline Script

    In the last post, I created a baseline marker for Flyway in each database. This set the version in the dev and QA databases to v1. However, I also need a baseline script, at least the tool asks for one, so this is the process if you have objects in your production or other downstream databases.

    I’ll do this for SQL Server and then PostgreSQL.

    Why do this?

    The main reason to create a baseline script is to note which objects already exist in production. For these objects, I don’t want to track these are changes in their current form.

    For example, if I already have a CountryCodes table in production, when I create a project, I want to tell Flyway Desktop that this table exists in production, so if the dev version matches, don’t add this to scripts. If it doesn’t, then I’ve done something in development and need an ALTER script deployed to prod.

    What was the Other Baseline?

    The first baseline in this post, is a version marker. I hate that this is the case, but both Flyway (pre-Redgate) and Flyway Desktop (evolved from SQL Change Automation), had the concept of a baseline, but these were somewhat different things.

    Flyway Baseline – The initial version of the database. Don’t deploy any scripts that are <= to this version.

    Flyway Desktop Baseline – A script that has the structure and code of all objects that exist in the target database(s).

    We can create a baseline script for Flyway, which looks for a B script, but the baseline command expects that you create this script manually. This is used to populate a new database with the baseline migration script prior to running all other scripts.

    Setting up the Baseline Script

    Flyway Desktop makes it easy to create a baseline script, and in fact, prompts you to do so.

    In my project, if I go to the Schema Model (first) tab, I see there is an object in Development. This was the table I created when I set up the database. The goal is to get this table to other environments.

    2023-04-04 16_10_58-Flyway Desktop

    This table doesn’t exist in QA. I do have the flyway_schema_history table, which was the result of the baseline command.

    2023-04-04 16_15_58-SQLQuery4.sql - ARISTOTLE_SQL2022.FWPoc_1_Dev (ARISTOTLE_Steve (77))_ - Microsof

    If I go to the Generate Migrations (second) tab, I see this. The first thing that the tool wants is a Shadow database.

    2023-04-04 16_12_55-Flyway Desktop

    The shadow is essentially a development V-1 (v minus one) version. This is where I test all migrations, compare the state with development, and then determine what’s changed. This is just a regular database, but I create this outside of Flyway Desktop. For me, I created a database (FWPoc_1_Dev_Shadow) and then clicked Set up shadow database to get this dialog. You can name this anything.

    I enter details, and test the connection before saving this. In general, this ought to be saved to my user settings as I’ll have my own shadow different from other developers. I DO NEED to click the “ok to erase data” box.

    2023-04-04 16_14_09-Flyway Desktop

    Once this is done, I now see another prompt on the Generate Migrations tab. Now I need a baseline script. I don’t have anything, but I will click the button.

    2023-04-04 16_14_28-Flyway Desktop

    This gives me a dialog to pick a target database. This target is used to get the initial set of objects to populate in the baseline script. You can use production or a copy (recommended) as the target database.

    2023-04-04 16_14_41-Flyway Desktop

    My QA is the same as prod, so I add that with the proper connection string and then I see the target here for the Baseline. I am ignoring static (or lookup/reference data for now). I’ll click the Baseline button.

    2023-04-04 16_15_25-Flyway Desktop

    This runs and … nothing.

    Which makes sense, as there is nothing in my target database. I actually get an error after this, which tells me that it doesn’t make sense to baseline an empty database.

    2023-04-10 12_38_51-Flyway Desktop

    I wish that were surfaced earlier. In any case, if I close this, I get the same image above, saying I don’t have a baseline. For now, I’ll ignore that.

    Summary

    Not much happened in this post. I added a shadow database, which I’ll use to generate scripts. I tried to baseline, but that errored, as it should. I really don’t need a baseline, so I’ll come back to this later in another format.

    For now, I’ve advanced the SQL Server project. I’ll actually repeat these steps for the PostgreSQL one, but it’s really creating a new database for the shadow and setting a connections string. Everything else looks the same.

    The next post will actually generate a script and deploy this to QA.