No Flyway Baseline No Migration

In my experiments with the Flyway CLI (fwcli), I’m finding some interesting behavior, some of which is catching my by surprise.

This post looks at the baseline command and the issues with not having one. I also cover a naming issue. This is a bit long, but I wanted to document what happens as I experiment. I’ll condense down what a baseline does in another post (or two or three).

The Scenario

I created a FWTest database and put a small table in it. I can see in my Object Explorer (OE) that there is just one table. Assume there are no views, functions, etc.

2022-12-27 16_17_44-SQLQuery2.sql - ARISTOTLE_SQL2022.FWTest (ARISTOTLE_Steve (83))_ - Microsoft SQL

My flyway.conf file points here, and if I run Flyway Info, I see the results below. The only important part is the “schema version”, which is empty, and the table, which shows one versioned migration called getone.sql.

2022-12-27 16_18_35-cmd

The migration is in my SQL folder, which is below the location of the flyway.conf file. The configuration file is in the smoketests folder and it has only two lines uncommented: 1 for the SQL Server connection string and one for the location of the migrations, which is the SQL folder. The relevant lines are:

2022-12-27 16_20_52-● flyway.conf - flywaysimpletalk - Visual Studio Code

That’s the basic start. I’ve got a database, and I created a new script. The script in the file is shown here:

2022-12-27 16_22_20-V1__getone.sql - ARISTOTLE_SQL2022.master (ARISTOTLE_Steve (64)) - Microsoft SQL

Running Flyway

Now, I have a migration script I want to apply to a new database. What happens if I run flyway migrate. Will this create my procedure? Let’s see.

The output shown below runs and give me a green line and a red like. One success, one error.

2022-12-27 16_23_10-cmd

The success is that the script was named correctly, so it passed validation.

The error is that there is no flyway_schema_history table. This is where all Flyway activity is tracked inside the database. Without this, there’s nowhere to stick the data on script execution.

The error does note that we need to run flyway baseline or set the baselineonmigrate option to true. The default for this is false.

Adding a Baseline

Let’s do the baseline thing. The documentation for baseline is very poor (as of Dec 2022) in my opinion. I’ve sent a few notes around the company, as I think this needs to be cleaned up.

In any case, this will do a couple things. First, it creates the flyway_schema_history table (under dbo for SQL Server) as the place to store history for Flyway. Next, it will add a row as the baseline version for this database. I don’t have a baseline script, a “B” script, but that’s OK. I don’t need it for now.

Let’s try this. I’ll run this on my database and we see the results below. I’ve captured the text, ignoring the licensing and connection string part. The results are really here:

Creating Schema History table [FWTest].[dbo].[flyway_schema_history] with baseline ...
Successfully baselined schema with version: 1

This shows me we have created the table and added a baseline of version 1. If we look in the database, we see this: the new table, but no proc.

2022-12-27 17_05_53-SQLQuery2.sql - ARISTOTLE_SQL2022.FWTest (ARISTOTLE_Steve (83))_ - Microsoft SQL

If I run flyway info, I see these results (again, ignoring the licensing and connection stuff).

2022-12-27 17_10_54-cmd

What I see in here is that my database version 1 is baselined with no scripts. However, the getone.sql script is noted as a versioned script but ignored because of the baseline.

The baseline is supposed to be a level that includes all scripts up to that number. It’s not well explained in the docs, but this means that any scripts up to the baseline are assumed to have been executed in the database.  This is the baseline. The baseline.sql script also is supposed to include the contents of all previous migration scripts, but as my test showed, I don’t need that.

The Problem with Baselines

The thing I have to know here is that this baseline version means no scripts at this version or lower will be executed. Since my script was a v1 script, it gets ignored, as you see below:

2022-12-27 17_10_32-cmd

Why this says 2 migrations, I don’t know. There’s only one script. I assume this is counting the non-existent baseline script.

However, nothing migrated, and my procedure isn’t created.

Be aware, that you want baseline scripts and other scripts to have discrete numbering.

Summary

Getting started with Flyway means we need a baseline to get going. We can do this without a script, but we do need to run flyway baseline, or set an option. I’ll look at those two items in a future post.

I also need to be careful with naming of scripts, as a script that matches the numbering of the baseline will not get executed.

Posted in Blog | Tagged , , | 1 Comment

Forgoing Tech Investments

The US was hit with a number of storms over the Christmas holiday weekend. This disrupted air travel for many airlines and their customers, but one of the worst hit was Southwest Airlines. They accounted for most of the cancellations, over half of their scheduled flights at one point.

A number of places reported talking with Southwest employees who blamed the lack of tech investment by Southwest over time, noting this caught up with them. The Chief Operating Officer disagreed, saying that their scheduling system is the best in the world, even as the CEO noted that their scheduling software couldn’t keep up and they fell back to manual operations.

Most of us likely have no idea of how Southwest software works or the scope of the problem. This airline does tend to operate differently than many others in that they mostly fly point to point, rather than using hubs. Possibly they have the best point-to-point scheduling software in the world, but it still couldn’t keep up with the storms covering much of the US.

There’s an interesting perspective on Facebook, supposedly from a pilot with 35 years of experience with SouthWest. If you don’t want to click, his view is the hands-on CEO retired years ago and accountants were appointed as CEO and COO. They improved the money flow, but neglected investments in tech and weren’t aware of how the business really runs day to day. The infrastructure and software deteriorated, and they’ve had many small issues, but issues that were bigger than other airlines. They’ve started turning around with a CEO that is more hands-on, but they’re digging out of a hole.

Like many of you, I’ve built and operated software over the years. I sometimes realize just how hard it can be to keep up with the demands of customers for adjusting how our systems work. I also know that it’s easy to slow your investment in a system that appears to works and limit your efforts to just maintenance work. Allan Hirt wrote about this.

This does bring up the issue of investing in systems and maintaining them over time. I see why many companies would prefer to purchase software and let someone else manage the investment in ongoing development. I also know that for companies that see software as strategic, likely there needs to be regular investment, upgrading and refactoring code, as well as finding ways to scale higher and use resources more efficiently. Especially for databases.

The battle between enhancing software and reducing technical debt is a constant one. I see this struggle being one that project managers and developers never agree on, but in the companies that seem to thrive, there is a balance. Perhaps it’s splitting the sprints, perhaps it’s allocating regular time during each development period, or maybe there’s another way.

One thing is certain. We need to find a balance. Otherwise we might get into the situation where a complete rewrite or replacement of software is warranted; a situation that is almost always very costly.

Steve Jones

Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

Posted in Editorial | Tagged , | Comments Off on Forgoing Tech Investments

Daily Coping 16 Jan 2023

Today’s coping tip is to learn something new and share it with others.

I do this all the time. It’s a part of my job, and I did this recently with a PostgreSQL container. I also did this with a piece of my coaching training, helping kids understand where to serve from /to for better chances of success.

Sharing is a great way to ensure to engage with the world and give back.

I started to add a daily coping tip to the SQL Server Central newsletter and to the Community Circle, which is helping me deal with the issues in the world. I’m adding my responses for each day here. All my coping tips are under this tag.

Posted in Blog | Tagged , , | Comments Off on Daily Coping 16 Jan 2023

The Programming Languages We Use

Many of you reading this probably work primarily in SQL. Even if you are a developer whose main language is something else, you write a lot of SQL. Even if you have an ORM writing the SQL that goes into production, I bet a lot of you are writing queries against a database to check that the data coming back in your application is correct.

As for me, I mostly work in SQL, with PowerShell and Python being second and third. I tried R for a while, but I think Python does everything R can do and it’s much cleaner. I find R very cumbersome. I rarely write C# or experiment with anything else, but that’s the nature of my job. PowerShell is important, as I do a bunch of DevOps and PoSh is a good choice to work with on the command line for gluing processes together.

There was a set of the top articles on programming languages from 2022 that I saw recently. I found it interesting to see what was popular. The top one was about Python being the most popular, but it shouldn’t be. This one feels like clickbait, and I find many of the conclusions not making an argument against python in a meaningful way.

There are some other links on the “hotness” of various languages. I think these are clicked on as many developers are just curious about what others are doing, and what they might experiment with. While I like curiosity and experimentation, I do think that many of our important systems in organizations need to be built with mainstream technologies. Support and staffing are a challenge, and while Golang might be great, finding people to read and code in it is hard. I don’t know how to balance the growth of new tech with the safety of old tech, but I wouldn’t stray too far from the mainstream for anything important.

I do find it interesting that COBOL makes the list. I know there are still lots of COBOL systems, and while there aren’t a ton of jobs, there are jobs and little competition. If I were 10-15 years younger, this would be tempting. Of course, I’d have to be willing to adapt to the jobs, but it is tempting. I know a few people making well into the six figures because of COBOL jobs.

It’s nice to see SQL is one of the top 10 languages in use, according to this survey.. It was #9 in 2021 and #8 in 2022. I don’t know it grew in popularity so much as assembly declined compared to other skills. I certainly can’t see SQL going away, but it’s not as popular, clickbait-y, or exciting as other languages. Instead, it’s a core, required skill for any serious software development. Whether you use relational or NoSQL databases, likely you need some SQL skills.

Steve Jones

Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

Posted in Editorial | Tagged , | 1 Comment