Voice of the DBA

Author: way0utwest

  • T-SQL Tuesday #202: 100 Hours

    It’s time for T-SQL Tuesday again and this is a great prompt to start writing. I might have written about this before, but it’s a story that sticks with me. It’s a memorable one, and it was quite the outage, one that lasted for weeks.

    Thanks to Marlon for this invite on unforgettable outages..

    T-SQL Tuesday is the monthly blog party, and if you want to participate, feel free to write on a blog, Linked In, or anywhere else and link your post to the invite. If you’d like to host, ping me. You need to write an invite and a recap. I’m always looking for people to host, so jump in and join the party.

    New Year’s Eve

    I was a relatively new hire, a former intern, at a large electrical utility in Virginia. I worked as a network admin at a nuclear power plant in Surrey, VA. I showed up at work on New Year’s Eve at 5:00pm. We were planning on deploying a new database server running SQL Server, along with a new application to track radiation exposure for workers. This was a mandated change to our tracking, which needed to go live at midnight. I was supposed to be a bystander, helping developers from our internal group implement the server and then take over administration for the future.

    New hardware had arrived, and we had installed OS/2 1.3 already. This was the “hot” new OS to us and to the plant, which would run SQL Server 4.2, a real database. Instead of the previous BTrieve database on our Novell network. The developers installed SQL Server, showed us how to add logins and users, and we verified this was running. The devs then went into the plant (with escorts) to install their software on various machines with radiation badge readers. These machines were the “clients” that connected to the new database server to look up user names, verify their total exposure, and record new exposures.

    We finished installations well before midnight and then went to eat some food. At midnight, we turned the old system off, and the new one went live. This was accomplished by stopping one application on the client machines and running a new one. A nuclear plant is busy, with people constantly moving in and out of spaces where they could be exposed outside the reactor. We saw workers getting badges, logging the device to their account and then entering the plant. Quick and easy.

    After 30 minutes or so, some workers would return from the plant, run the app, log in and have their badge scanned, recording their exposure. Many were 0 rads, but a few might have picked up minor exposure.

    Then the crash. Once we started getting a little busy, with 5 or so people logging in the application stopped responding. I worked with the devs in the server room, where we saw the new OS/2 server hung. We rebooted it, thinking something went wrong.

    This happened again.

    And again, and again.

    We ended up seeing either the entire server hung, or SQL Server not accepting connections. In the v4.2 version, we didn’t have a Dedicated Administrator Connection, when the server was hung, we could only see external, OS-level counters. Our network staff (4 of us) stayed until around 4am, at which point my boss sent a few people home. They had families, so he and I remained behind. We alternately caught naps and were on call to reboot the server. Sometime during the morning the plant rolled most people back to paper, manually tracking workers and exposure on cards, one per worker. We kept one machine on to keep testing. That worked, but when a second client was added, the server kept hanging.

    My boss and I stayed, trying all our tricks, reading lots of OS/2 and SQL Server  docs, and trying to help developers figure things out. The plant wasn’t that busy on Jan 1, but busy enough that their system kept freezing up. Lots of arguments amongst our staff, our managers, and the software development group. I stayed overnight, rebooting servers, napping in the server room (a cold room) with a hat and coat. When the “normal” day started on Jan 2 and coworkers showed up, I went home.

    I returned late afternoon to relieve the other staff, working through the night again, mostly rebooting a server and trying to find a cause. I learned about the “probe” user, which was a backdoor account used for accessing data. I never found a reason why the application didn’t work as expected, other than the developers didn’t really test with multiple clients.

    In any case, it didn’t matter. I worked 100 hours that week, from Dec 31-Jan  6. Since I was being paid hourly, I was happy to do so. My partner, less happy.  I continued working over 80 hours every week that month. The developers stabilized things enough for the system to be used, but it still was rebooted every few hours. Since the plant runs 24×7, that meant the 5 or 6 of us needed to ensure someone was there all night long.

    A profitable month (and the next), but a hard one.And an unforgettable outage.

  • Labor Day 2026: Fun on the Ranch

    It’s Labor Day in the US and I’m not working. Not for Redgate, at least. I’m not even traveling today, which is something that happens all too often on holidays.

    Today I’m going to be working on the ranch, catching up on a few things. I was up in the air last week, trying to replace a pulley. The bolt in this pulley sheared off. Both ends and the nuts were still there, somewhat rusted to the side. The wheel and middle were on the ground.

    2026-08_0307

    This lasted 15 years, which is good. Bad that this was 30ft in the air at the top of a building. And I was it Scotland when it broke.

    2026-08_0338

    My daughter secure the door, so she could get hay out. I had to rent a lift last week as this couldn’t wait until Labor Day. We had a spare and I ordered another one as well. I got up in the air and unbolted this and replaced it with a new one.

    While I was up there, I took the opportunity to replace the lifting cable. This had broken near the bottom a few years ago and I added a new section, but it didn’t go up and down smoothly. Since I had the lift, I was up here, undoing the stays for the cable on the left and then running it down 30ft to the bottom of the door.

    2026-08_0308

    My daughter had noticed part of the upper cable was also worn, so I ended up replacing that since I had the lift. My wife also asked that we tighten down the straps on this end of the building, as they were looser than others. We replaced the roof last year, and if you want to see part of this, I have a video of us pulling this section over the frame.

    It was a long, hard 24 hours of having the lift and being up in the air for 8-9 hours while working on a Saturday afternoon and Sunday morning.

    I’m sure I have something else in store for today just as hard. It’s a wild life for a database professional who lives on a ranch.

  • The End of Summer

    Growing up, this was the last week of summer for me every year. When I was a child in Virginia Beach, school started the day after Labor Day every year, which is next Monday in the US. This was our last chance to sleep in, enjoy the ocean, and stay out late without commitments. Starting next week we’d be back in school, up early, with homework to do every night.

    As an adult, the end of summer doesn’t really stand out anymore. Corporate changes are usually at the beginning of the year or in July, so it’s the end of June or December that are slower times. September is often the beginning of conference season for me, and I’ve usually been preparing for travel in September, as I am this year. I’ll be in San Diego in a little over a week for VS Live (if you want to join me, use the code Jones).

    I hope that most people have busy times and slow times at work. If you’re constantly busy and fighting fires, life is not good. That leads to stress, unhealthy living, and burnout. It’s not sustainable over time, at least not if you want to keep producing high quality work. I’ve lived in those situations and they’re neither enjoyable, nor the times when I’ve been at my most effective.

    When work is slower, I naturally slow down a little and appreciate that I can take more time to write, answer emails and messages with more detail, and think more carefully about situations. I also use the time to get ahead administrative tasks that I sometimes put off. For me, this is producing and scheduling things I’ll need later. This is often a time when I’ll write questions or editorials for the future.

    These are often the pieces I’m more pleased or proud of. I have time to work through details and spend more time reviewing my work. These are often the best questions I write.

    As a developer, this was the time I might refactor code I’m not pleased with, or tackle some niggling piece of work that I know is needed, but never has priority. As a DBA, this is when I refactor scripts and alerts, do more maintenance, maybe add some automation for tasks that I’ve just been doing manually because I don’t have time to do anything else. If I were  still a dev or DBA, this is when I’d be experimenting and practicing using AI as a lever to make myself more effective.

    When you have slower times, what tasks do you tackle? Documentation? Improvements? What else might not get done when the pace is a little more hectic? Let me know today.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Advice I Like: Lots of Ideas

    “A multitude of bad ideas is necessary for one good idea” – from Excellent Advice for Living

    Whenever we read about the success of someone or something, we often get the impression that someone had a great idea and executed on it. Especially in computing, it can seem like it’s the trajectory of many of the companies we know well.

    The reality is that a lot of companies have various bad ideas and then discover one working well. They pivot into the one good idea. Microsoft started writing programming languages and then pivoted to providing DOS. Netflix went from DVD rentals to streaming.

    The reality is that often we have lots of ideas and it can be hard to decide what a good idea is. That’s why we brainstorm, why we game, debate, and discuss options. We don’t have any hindsight when we are trying to move in a new direction, so we need to come up with various ideas to find a great one. Artists do this, musicians do this, plenty of creative people do this.

    I used to want to try and wait to have a great idea before discussing it with others. Andy Warren taught me to just throw things out there and then examine them with others.

    I’ve been posting New Words on Fridays from a book I was reading, however, a friend thought they were a little depressing. They should be as they are obscure sorrows. I like them because they make me think.

    To counter-balance those, I’m adding in thoughts on advice, mostly from Kevin Kelley’s book. You can read all these posts under the advice tag.