Voice of the DBA

Category: Editorial

  • A Staffing Disaster

    There was a failure recently at an Azure data center in Australia when a utility power sag caused equipment to trip offline at one of the Azure data centers in Australia. You can read about it here, but essentially the headline is that there were only three people on site when the incident occurred, and that caused them to be unable to restart the equipment in time before an outage occurred.

    In a little more detail, there weren’t enough people to quickly restart the equipment chillers after the incident. The staff had to access the equipment on a roof when 13 of the units didn’t restart. They were able to get to 8, but when they got to the last 5, the temperature of the water had risen to a level that wouldn’t allow a restart. So they had to power down some computer equipment and go through a more lengthy process to get everything running.

    This sounds bad, but in reality, this is exactly the type of thing I’ve seen in private data centers, who almost never have all the staff they need, or the knowledge necessary, to deal with large-scale failures. While I haven’t seen the chillers, I have seen people trip electrical systems and be unable to restart or reset UPS’s or generators for hours until qualified staff could come in. If you read the incident history, there is a good retroactive of what happened, and then some actions taken to try and prevent this in the future. They increased staff levels but also identified some places where the previous staffing level would have been fine with some equipment and protocol updates.

    I wish more organizations would review incidents and examine them with an eye towards not only what happened and where there were failures, but how to prevent issues in the future. Too often I see people going through this exercise in order to blame someone and “prevent this from ever happening again”, which usually means we fire someone and don’t change anything else. We need psychological safety in reviews of actions to get better.

    As we build more complex systems, or even more complex organizations with lots of teams, people, equipment, procedures, etc., it’s easy to build in lots of points of failure without realizing there will be problems in the future. My goal often these days is to assume I’ll have some inexperienced or less capable staff and design processes and systems to survive issues. To keep things simple, and not get too cute with engineering. I like robust, resilient systems that anyone can operate, not those that require me to ensure my senior superstars are always on call.

    Of course, it often takes the senior superstars to design and test these systems and protocols, which is a good use of their time.

    Many businesses struggle with staffing, in many areas. Technology groups are no different, and we have to learn to work smarter, not assume we will just get more staff and solve our problems.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

  • Running a Service

    Many of us work for some organization and we’re responsible for managing a database or coding some application or some other data related position. Perhaps we ensure reports work for our users or move data around. In many cases, we have customers, whether they are internal to our organization or perhaps external from another business or even the wider public.

    However, many of us see our jobs as a part of the org/business and not that we are actually providing some service to others that we are responsible for. While I’ve certainly felt that way, I learned at some point that if I were an app dev, I ought to treat people using my app as customers, even though they don’t pay me in currency. They often pay me in praise, or better yet, with praise to my boss. As a DBA, I treated both app devs and users as customers. That helped me take pride in my work and learn to view my position from other points of view.

    There was a post on learnings from running a SaaS service for a year. While few of us maintain a SaaS service, we could think of the app we build as Software-as-a-service, albeit without any revenue. We could see databases we manage as DBaaS (databases as a service) or PaaS (platform as a service) items.

    I think that the way we view the world, or view our situations, can be helpful to us in managing our workloads. We can better triage what is critical, what’s important, and what’s nice to have. We can decide when an issue deserves some to cause some stress and we might give an extra effort, and when something can wait until tomorrow. We can decide when we push back on demands and when we accommodate them.

    This approach has also helped me to think about ways to improve the service/software/platform I provide to others in ways that helps my customers, or makes my job easier. This way of thinking leads me to make the situation better when I can. It also helps me to discuss the decisions with others, especially management, in terms that can matter to them. Are we finding ways to improve the overall system?

    It doesn’t always work, but it often has for me.

    And if you’re interested in your own side project, the journey continues in part two. Perhaps you can spend an hour or two a day and build something that becomes your own business. That’s what happened to me with SQL Server Central.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

  • Are You Worried About AI?

    The AI revolution is happening. I don’t know how successful it will be, but regardless, people are trying to get AI into everything. That might be good or bad in various situations, but it has people worried about AI. This piece looks at a survey from the American Staffing Association that says 47% of people somewhat or strongly agree that their job is replaceable by automation with an AI. There is more data to dig into, but that’s a lot of people who worry about their jobs.

    Most people do think that tech is generally a good thing. I do find that many of us in tech, and some other industries, are buried in work. There’s never a shortage and we could easily work 60, 70, or more hours a week and not complete everything. The flip side of that is a lot of work doesn’t necessarily need to be done now and could be delayed, so I think it’s hard to decide whether we need more help or more patience.

    The various places I’ve seen AI used are often helping people get started or unblock their creativity struggles. Recently someone posted this link on some prompts a person had used to get moving. I found a few of these possibly helpful. Not writing, but maybe having an AI proof something, or even get me started in some new area. I worry about book summaries, or even text summaries, as I’m not always confident the AI understands which pieces I might find important. However, that’s also a danger with humans. I’ve seen no shortage of people asked to summarize something for others. When I understand the subject well, I sometimes disagree with another’s summary (or think it’s just plain wrong).

    I don’t think AIs are a big danger to many people’s employment. I do think that as people learn to use AIs are assistants we might find that we grow or enhance staff less. We might also find that the 10x engineer knows how to engage 10 AIs to help them get things done, especially simple, busy work. The low performers will likely be those working in the same way in the future as they do today. That might not threaten your job, but it might mean lower bonuses and less opportunities in the future if you can’t adapt your skills to this new world.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

  • If You Build It, Will They Use Linux?

    There’s a great scene in Field of Dreams where James Earl Jones says that people will come. It’s in response to the voices Kevin Costner hears early in the movie. This is the climax of the movie, where Costner makes a financial decision to trust his instincts and hope his farm will be saved.

    In many cases, organizations do just this. They build something, assuming people will come. They may have some data, research, or other reasoning as to why why people will use what they build. However, that’s not always the case. Sometimes they build something and hope people will come, much like Field of Dreams.

    I wondered about this recently with some friends when discussing SQL Server on Linux. Quite a few people I know haven’t seen any Linux installations inside their organizations. Others are consultants and haven’t experienced any work on Linux, which is interesting. I’ve heard people at Microsoft state there are plenty of installations out there, but I’ve found few people who have moved from Windows.

    I know a lot of people who work in the Microsoft data stack aren’t familiar with Linux. This is despite all the articles and writings the last few years trying to teach people about Linux and how SQL Server runs well. I also see articles like this one, which claims Linux uses more CPU. This is despite Red Hat saying Linux runs SQL Server well and leads in benchmarks from TPC, at least the TPC-H results.

    I learned at the various command lines in school, with a lot of time spent on Unix systems. Even with X-Windows, I often used the interface to open multiple shells to get work done. However, a lot of people have not worked at the command line very often. Despite the popularity of PowerShell, I find no shortage of Windows-based knowledge that struggles with certain concepts, like quoting and piping results between commands.

    I don’t know how many systems run Linux v Windows. Azure states that over 50% of their VMs aren’t Windows, but Linux. I suspect a lot of those are likely web servers or other systems and not database platforms. They don’t seem to publish how many databases are SQL Server on Linux, and I have no idea if Azure SQL Databases (or MI/Synapse/Fabric/etc.) run Windows or Linux. I suspect it could be either, but most likely Windows.

    Do you feel comfortable running SQL Server on Linux? Is your organization considering it? I don’t know I think it’s worth the savings in license costs. Not that Windows is cheap, but the effort to train, learn, and work on a second OS might not provide any savings for years. Maybe not ever if you can’t get work done smoothly and quickly.

    They built it; now, will you come run it?

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.