T-SQL Tuesday #202: 100 Hours

It’s time for T-SQL Tuesday again and this is a great prompt to start writing. I might have written about this before, but it’s a story that sticks with me. It’s a memorable one, and it was quite the outage, one that lasted for weeks.

Thanks to Marlon for this invite on unforgettable outages..

T-SQL Tuesday is the monthly blog party, and if you want to participate, feel free to write on a blog, Linked In, or anywhere else and link your post to the invite. If you’d like to host, ping me. You need to write an invite and a recap. I’m always looking for people to host, so jump in and join the party.

New Year’s Eve

I was a relatively new hire, a former intern, at a large electrical utility in Virginia. I worked as a network admin at a nuclear power plant in Surrey, VA. I showed up at work on New Year’s Eve at 5:00pm. We were planning on deploying a new database server running SQL Server, along with a new application to track radiation exposure for workers. This was a mandated change to our tracking, which needed to go live at midnight. I was supposed to be a bystander, helping developers from our internal group implement the server and then take over administration for the future.

New hardware had arrived, and we had installed OS/2 1.3 already. This was the “hot” new OS to us and to the plant, which would run SQL Server 4.2, a real database. Instead of the previous BTrieve database on our Novell network. The developers installed SQL Server, showed us how to add logins and users, and we verified this was running. The devs then went into the plant (with escorts) to install their software on various machines with radiation badge readers. These machines were the “clients” that connected to the new database server to look up user names, verify their total exposure, and record new exposures.

We finished installations well before midnight and then went to eat some food. At midnight, we turned the old system off, and the new one went live. This was accomplished by stopping one application on the client machines and running a new one. A nuclear plant is busy, with people constantly moving in and our of spaces where they could be exposed outside the reactor. We saw workers getting badges, logging the device to their account and then entering the plant. Quick and easy.

After 30 minutes or so, some workers would return from the plant, run the app, log in and have their badge scanned, recording their exposure. Many were 0 rads, but a few might have picked up minor exposure.

Then the crash. Once we started getting a little busy, with 5 or so people logging in the application stopped responding. I worked with the devs in the server room, where we saw the new OS/2 server hung. We rebooted it, thinking something went wrong.

This happened again.

And again, and again.

We ended up seeing either the entire server hung, or SQL Server not accepting connections. In the v4.2 version, we didn’t have a Dedicated Administrator Connection, when the server was hung, we could only see external, OS-level counters. Our network staff (4 of us) stayed until around 4am, at which point my boss sent a few people home. They had families, so he and I remained behind. We alternately caught naps and were on call to reboot the server. Sometime during the morning the plant rolled most people back to paper, manually tracking workers and exposure on cards, one per worker. We kept one machine on to keep testing. That worked, but when a second client was added, the server kept hanging.

My boss and I stayed, trying all our tricks, reading lots of OS/2 and SQL Server  docs, and trying to help developers figure things out. The plant wasn’t that busy on Jan 1, but busy enough that their system kept freezing up. Lots of arguments amongst our staff, our managers, and the software development group. I stayed overnight, rebooting servers, napping in the server room (a cold room) with a hat and coat. When the “normal” day started on Jan 2 and coworkers showed up, I went home.

I returned late afternoon to relieve the other staff, working through the night again, mostly rebooting a server and trying to find a cause. I learned about the “probe” user, which was a backdoor account used for accessing data. I never found a reason why the application didn’t work as expected, other than the developers didn’t really test with multiple clients.

In any case, it didn’t matter. I worked 100 hours that week, from Dec 31-Jan  6. Since I was being paid hourly, I was happy to do so. My partner, less happy.  I continued working over 80 hours every week that month. The developers stabilized things enough for the system to be used, but it still was rebooted every few hours. Since the plant runs 24×7, that meant the 5 or 6 of us needed to ensure someone was there all night long.

A profitable month (and the next), but a hard one.And an unforgettable outage.