Voice of the DBA

Category: Editorial

  • Bare Metal

    At the first job I had as a DBA, I had to build a new server. This was in the days of SQL Server 4.2, and I was combination DBA, sysadmin, and general help desk at a small company. With a software developer consultant, we ran some tests on various machines and then ordered a collection of parts from Compaq. Back in this time, they would only customer parts of the server. We unpacked our boxes with the server, extra drive bay, various SCSI drives, and extra RAM. We assembled and tested the machine and eventually put it into production.

    In the years since then, I got out of the hardware business and left that to others. For awhile I worked in organizations with IT staff dedicated to building machines, but at some point we stopped doing that. The growth of VMWare and other hypervisors changed the paradigm for most organizations. For more than a decade, all the servers I’ve connected to are virtual machines running on hardware that my employer or a cloud provider owns and manages as a node in a cluster.

    Early on there were concerns about the overhead of using hypervisors and virtual machines. When many early workloads moved to VMs, lots of organizations left database servers on bare metal to squeeze every bit of performance out of the system that was possible. Over the years, improvements in hypervisors as well as the software used to connect storage and networks together seem to have rendered those conversations obsolete.

    Or have they?

    Today I’m wondering if any of you still have servers on bare metal. Are there systems that you continue to install the OS and database server software directly on the hardware? Or are there systems where you still ensure that there is only one VM, the database server VM, on a hardware node?

    The cloud has changed a lot of these conversations, since everything is a VM. Even many local data centers will rent you a VM, something that wasn’t possible when we first moved SQL Server Central out of a friend’s basement into a co-location facility. Back then we owned our servers and I installed them in data center racks. I would like to think we’d have moved our site to the cloud in some way over time, though maybe not. Maybe I’d still be managing a couple servers in a room in Denver.

    Are any of your organizations still running legacy systems in the same way you would have in 1999? Let us know today.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Planning for a Bad Day

    Are you ready to get a new job? I’m not talking about you deciding you want a new job and working towards that goal, but rather getting surprised by a layoff or termination today. If your boss called you into his/her office in the next 10 minutes and told you that you were being terminated today, are you ready for that? Kendra Little wrote a nice post recently asking who might be prepared.

    Most of us aren’t. In fact, in my branding presentation, one of my slides has a screenshot of a post from LinkedIn. A friend noted that they walked into work and found out his company had been purchased. It wasn’t a good day because they were also notified they were being laid off, hence the post on LinkedIn letting others know of their need to find one.

    The modern work world, especially for technology workers, isn’t stable and certain. While we have lots of positives in our jobs (work environment, low physical needs, remote possibilities, etc.), many of those can be detractions. Outsourcing IT jobs is something many companies do, and whether it’s a good idea for the company or not, it’s a reality. We also have lots of acquisitions in different spaces, and when sets of IT groups come together, there isn’t always the need to keep everyone employed. Even with double the workload, many in management would rather save money and give more responsibility to fewer people.

    Side note, this is one reason a good knowledge of DevOps, ITOps, GitOps, etc. is valuable for your career. When a manager is trying to decide who to keep, those with these skills may stand out.

    Unless you are ready to retire and merely padding your assets, you ought to have a plan for changing jobs. Even if you like your job, circumstances might force that upon you and when lots of people are scrambling for new jobs, competition can be fierce. Like Kendra, I don’t know you need a detailed plan, but documenting some skills growth and keeping your resume or CV up to date is always a good idea. I recommend everyone touch their resume every quarter. At least spend 5 minutes looking over it and thinking about it.

    Kendra has other suggestions, and I like them. In fact, I’m looking to think about a few of these. Not that I want another job, but you never know what might happen, and I’m not that close to retirement. Close, but not close enough. Certainly thinking through a few options at a high level is something I want to do with my wife.

    And make sure my resume is up to date.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Securing All Your Connections

    I read an interesting blog from the cyber security team at Microsoft, noting you should use TLS for your SQL Server connections. I would assume most professionals know that using TLS and secure protocols across the network is important. I would also assume few of us are willing to get real certificates for all of our SQL Servers, especially those in dev and test environments.

    Setting up a certificate for a server instance isn’t hard, but it’s also not easy. It’s also something that I don’t know how easy would be to automate in many environments. I know that you can use APIs from somewhere like Let’s Encrypt, but integrating that into a server setup process would be something. What about integrating this into instances in containers? I don’t know that I think many SQL Server organizations have DNS integrated for most of their database servers, much less asking them to also get certificates set up.

    I do think that Microsoft likely has better infrastructure in place, and better staff resources, than most of us. I can see this being something needed for Azure SQL DB and Azure MI. Heck, this might be something we want for all services that we get in the cloud. Since we don’t control much of anything other than the service, and Microsoft can automate the process of generating certificates, this makes sense.

    I think this might be something that is a good idea for production systems as well, ensuring that when a DBA or system makes a privileged connection to a database server in production, the person or process is sure of which database server is at the other end of the connection. I don’t know that I’ve seen an exploit in the wild where someone impersonates a prod server and captures credentials in some man-in-the-middle attack, but I wouldn’t be surprised if it happens.

    Do you think you also ought to avoid trusting the server certificate? I don’t think this is practical in many orgs, but I’d be curious what you think today.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Concurrency Challenges Around Schema Changes

    I saw a great question on Twitter from Frank Pachot, a developer advocate of Yugabyte. He wrote: Without thinking how your preferred database deals with it, what do you expect if:

    • session 1 starts to reads table T
    • session 2 drops table T
    • session 1 continues to read

    The choices in his poll were: session 2 waits, session 2 fails, session 1 fails, both fail. My first thought was SQL Server and the default need for session 2 to get an exclusive lock. In that case, session 2 would wait. Most people answered that same way, but then Frank posted a follow-up with a link to his blog. The answer for Yugabyte is that session 1 fails as it gets the message that the table was deleted.

    Leaving aside the decision to drop a table, imagine this is some schema change instead. In the blog, some good points are raised about how to handle high concurrency changes, and the potential problems with having session 2 wait. On a busy system, this could cause lots of blocking as threads stack up behind session 2.

    It’s an interesting read about the challenges of distributed system design and how to handle changes. In some sense, I get that this makes sense, but I wonder where this causes issues. If any schema change on the table by session 2 were to cause an error in session 1, that would be bad. However, does this mean that the database engine must now evaluate whether a column change impacts a query in flight? Then decide to send an error? What about evaluating views or procedures/functions that depend on

    Does this mean that all nodes need to sync up the schema changes quickly, and at a higher priority than data movements? I don’t know exactly how Yugabyte distributes data, and if there are copies on multiple nodes, but I assume there are. This adds complexity to the communication between nodes, which is likely needed. Honestly, if someone drops a table and they should have, we probably don’t want clients getting results. If they do this accidentally, I’d like to know about it quickly.

    The question is interesting, and there are multiple ways to look at this, but I found it fascinating to spend a few minutes thinking about the complexities of data in distributed systems and the challenges involved. This also made me think that the people who keep data safe and fix problems when they occur are invaluable in the modern world.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.