I saw a comic from Kendra Little recently, which reminded me of my first SQL Server crisis. I got hired as a contractor to support a large Novell network. In my first chance to work with SQL Server, a new database server running SQL Server 4.2 on OS/2 1.2 was installed on our network to support a new data entry application. This was mandated by a government legal change on Jan 1.
As the junior person, I was supporting the installation by corporate developers on Dec 31 at 5 p.m. We finished getting things set up, ate dinner, and then I watched the developers do some smoke testing and prepping the application and database servers for go-live at midnight. It was exciting to me as a young professional to be part of this deployment.
At around 12:30 a.m., as I was getting ready to leave, the system locked up. We found the database server unresponsive, so we rebooted it. This continued to happen, with my eventually paging my boss and having him come in the early morning. We worked through the night and into the next day. I didn’t leave until early morning on Jan 2nd, with the need to return that night. We had an overloaded, unstable system that required us to work long hours babysitting the server. It was worse for our clients, who had to manually record data and then try to perform data entry during slow times to catch up the system. I was a hero that January, as were my co-workers, with all of us logging around 100 hours each week that month.
It was a rough time, though I learned a lot about SQL Server, OS/2, and eventually, Windows NT. I also learned about poor software testing, especially load evaluation. I learned how easy it was to overwork yourself, and how heroics are needed but can’t be business as usual. Being a regular hero isn’t something that is conducive to being effective or efficient with your staff or your business. I saw this later when I came to Denver and a poorly designed system that needed patches or reboots every week overloaded me and my staff.
Much of what we do as developers or operational staff is to support others. We provide them with systems to get their work done, whatever that work might be. We work hard, and we find solutions to challenges, but we ought to not only provide stability to customers, we need to do that over time. That means that we need to have a staff that we can count on over time. If we overload them or struggle to retain them because of burnout, we can’t easily do that. We also can’t get new, perhaps more important work done if we are constantly dealing with the same issues.
I’ve seen management do this, and they end up with staff that isn’t efficient, can’t tackle new work, or turn over constantly. Some of you might know of a company like this in your area where they are always hiring because they aren’t a place anyone wants to work. Someone will work there because they need a job, but that company is never very efficient. Eventually, they’ll struggle with their competition and lose business. Or maybe not, maybe they’ll linger on with lots of missed opportunities.
I’ve also seen IT professionals live like this and thrive on emergency situations. If you’ve read The Phoenix Project, you’re familiar with Brent, who is the go-to person for anything. I’ve been that person, and I’ve seen that person in different organizations. While I always appreciate them, I’m often frustrated because they become too busy to handle my requests and there is a constant stream of requests because things aren’t being fixed or solved. It’s never a good situation.
We need to be heroes at times, but those ought to be emergencies, and they ought to be rare. Most IT professionals (and others) I know will work longer or harder when needed, but the need can’t be constant or even regular. It should be an unusual situation. If it’s not, then something has to change, at least for me. Either the technology or the staff. As I gained experience and savvy, this has usually meant I go find another job because the situation has become “emergencies are status quo.”
Life’s too short to live like that.
Steve Jones
Listen to the podcast at Libsyn, Spotify, or iTunes.
Voice of the DBA 