Rolling Back a Broken Release

We had an interesting discussion about deployments in databases and how you go forward or back from the point when you discover a problem. You can watch the episode and see what you think, but one thing that Pat asked was about rolling back a broken release.

I’ve seen a few broken releases that were rolled back immediately in my career. However, in a lot of cases, I’ve also been a part of semi-failed releases where we had to roll things forward.

I learned early on to smoke test the system post-deployment. Either get an account, or run known queries after applying a patch to ensure things worked BEFORE I let someone know the deployment was complete.

In one case, we applied a patch, restarted the application and started receiving errors immediately. We knew then that whatever things had changed, those changes were not in sync with the application. In this case, the application code and the database code had a small typo in a name, but late at night, we didn’t realize it was this simple.

In the days of outage windows, we couldn’t debug for long, so I decided immediately that getting back online was important. We replaced the new .exe with the old one and I looked at each of the database commands and wrote a reversing one to reset the database. Since the system was down, there weren’t any data changes.

In recent years, a few clients have had an easier time as they use feature flags to enable new functionality. When they’ve had an issue like the wrong name in code, they just flip the toggle to disable the feature. This rolls back the code.

I highly recommend using feature flags to anyone working with database software changes. Coupled with zero-downtime architectures for database changes, this lets us rollback things quickly.

Posted in Blog | Tagged , , | 4 Comments

Doing Good at SQL Server Central

This is part of a few memories from the founders of SQL Server Central, celebrating 25 years of operation this month.

We did photoshoots at Redgate many years ago. We had a bunch of props, including some phrases written down. We could create our own, but my handwriting is atrocious (likely why I never became an architect), but I ended up with this one:

Steve Jones caption "Doing Good"

I couldn’t really explain why I picked “Doing Good”, but the more I thought about it over the years, it’s been a lot of what has driven my life forward, both here at SQL Server Central and in my personal life. I’ve been a Boy Scout and Girl Scout leader, have been and am a sports coach, and I have given up no shortage of time to help others with speaking efforts. I put time into the SQL Saturday charitable foundation, trying to get more SQL Saturday and Day of Data events that teach, train, and inspire data professionals.

SQL Server Central has a huge reach. We’ve been very popular, and millions of people have seen our newsletters. Our mission early on was to help database professionals get better every day, and we provided a lot of free (as in beer) resources: articles, quizzes, and answers to your questions. Many community members have also volunteered their time and expertise to help, and I thank them as well.

There have been a few times when I’ve tried to use that reach to do good. I don’t want to point out the specifics, but a few times I’ve known people with very sick children. One was a local friend I’d worked with (tech pro, but not database) and another was a speaker in our community. This started when I was sitting at dinner with a fellow speaker, who was talking about this family struggling and needing help. I thought, surely we as a community could raise money and help them. I used the reach of our newsletter at that time to ask for donations and managed to raise substantial amounts of money to help a family. I did it again a few years later, raising well over USD$10k in both cases.

In those cases, as well as with SQL Saturday, I have done what I thought was in the best interests of everyone and ensured all the monies raised went to the intended recipients. The families got all the money donated, and all sponsorship monies to to events. If there are bank charges or other fees, I absorb them. In the case of SQL Saturday, I go begging for donations to support the admin costs and fees.

Those might be some of the things I’m most proud of in my life, which is an interesting thing I learned. I really like helping others. As much as I enjoy living my life, I get more satisfaction that I would have expected by helping others grow, learn, succeed, and enjoy their own lives.

I think everyone should volunteer some of their time/resources/knowledge/etc. at some point in their life. It might not be now, but think about where you can help make the world better. You’ll gain way more than you ever spend from helping others..

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.

Posted in Editorial | Tagged | Leave a comment

Advice I Like: In 100 Years

In 100 years a lot of what we take to be true now will be proved to be wrong, maybe even embarrassingly wrong. A good question to ask yourself today is “what might I be wrong about?” – from Excellent Advice for Living

In this age of AI, which is an incredible disruptor, it’s interesting to think back about past changes. I heard someone recently compare the AI revolution to the change from assembler to C (high level languages). That’s interesting, but I think it might be even more impactful.

And I’m a little worried.

In any case, 100 years ago we were in the boom of the stock market, the roaring 20s. The teletype was in use, people thought after WWI that life would be amazing and we’d never go to war like that again.

I can’t imagine 100 years, but I have been skeptical of AI, especially writing good code. At this point, after more testing, I think I was wrong about AI being a disruptor for tech. With advances in robotics, I truly worry that AI might cause serious challenges for humanity.

I hope I’m wrong.

In terms of databases, I constantly think relational systems are the best all-around store and a good choice for most work. I constantly ask myself if I could be wrong. I don’t think I am, but I keep asking the question.

It’s good to ask yourself this type of question. Have those strong opinions, but loosely held.

I’ve been posting New Words on Fridays from a book I was reading, however, a friend thought they were a little depressing. They should be as they are obscure sorrows. I like them because they make me think.

To counter-balance those, I’m adding in thoughts on advice, mostly from Kevin Kelley’s book. You can read all these posts under the advice tag.

Posted in Blog | Tagged , | 4 Comments

Engineer Lessons

Many of you reading this have a number of years working with technology. You might have 1 year or 20 years, but you’ve likely grown and learned along the way. Some of you may also know someone who has several years of seniority in a position but not that many years of experience. In this case, a person might have been working at this job for 5 years, but they really have one year of experience that’s been repeated 5 times.

That’s been a common complaint over quite a few years from people who interview others. They find candidates often have very limited experience, yet are applying for senior roles. These candidates are ones who have just a few years of experience, but have ended up repeating those few years over and over.

I ran across an interesting piece that contains 21 lessons from the author’s 14 years at Google. You might not think Google is the place where great software engineering takes place, or where great careers are made, but Google has been a place where it is challenging to get a job, they work on truly large-scale problems, and Google has had many engineers who have gone on to success elsewhere.

The first few lessons in here are things that I’ve learned in my career. Quite a few of them point to the value of being a team player, and remembering that being effective and efficient matter. It’s easy to be smart, or depend on one’s code to speak for our ability, but in many cases, we are working in teams. Our code is a team game, and others need to understand our code. It’s easy to forget that someone responding to a problem at 2 am might not understand the code. It’s also easy to forget that person might be you, and you might get confused.

Others also need to understand and believe in us. Lessons 2, 14, and 16 talk about the value of working with others and not creating resentment or ill will in others. Even with AI help, most software will require multiple people working together, so building those skills and habits is important.

I found this to be an interesting list, and there are other topics I want to discuss in the future, but the main thing I see here is that these are lessons from someone who has had to work as a technical engineer with others, while having success both at his job and outside of it. That resonates with me, as do these lessons. Individuals and teams that work in similar ways to those described in the piece have tended to have more success than those that don’t, at least in my experience.

Learn to be a team player as an engineer, developer. DBA, or any other job, and you will have more success than if you expect to only be judged on what code you write.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.

Posted in Editorial | Tagged | Leave a comment