Voice of the DBA

Author: way0utwest

The Invisible Disk

I saw a post recently about the incredible shrinking computer, where a software consultant had written down his predictions for the future. In this case, he writes about a computer shrinking smaller than today, essentially to one chip, with docking stations wherever you need them. We’re already at the place where many of us have an extremely powerful computer in our pockets in the form of a mobile phone. I’m still amazed that I bought a phone recently with half a terabyte of storage.

A few years ago I gave a keynote where I looked at the changes in disk storage, starting with an IBM hard drive being loaded onto a plane. The capacity was 5MB. I remember working with KB floppy disks and MB hard drives, and lots of different connection technologies: IDE, ESDI, SCSI, and more. I’ve seen disks shrink from a 5 1/4″ form factor to mere wafers with today’s NVMe drives.

However, what might be even more interesting is that I don’t even think about drives anymore as being a physical thing. Certainly, my mobile, my laptop, and my desktop have physical drives, but they’re all connected together, essentially giving me invisible disks. I can take a picture on my phone and it shows up on my laptop or desktop in seconds. The disk might as well be invisible.

As I left full-time Operations work, we were starting to see this in our servers, which were essentially invisible VMs stored in some location, on storage that we never laid eyes on. We never saw disks fail because the storage arrays had redundant disks. There were people replacing failed drives every week to ensure that service wasn’t interrupted.

These days we rent storage in the cloud without ever working with the hardware. Most of us who specify the configuration in the cloud don’t think about disks. We think about the total capacity and the IOPs that we need for our system. We care about the results, not the hardware.

Most of us. Some of us care and worry about it, but a lot of the people I know in data work, even as operational DBAs, don’t worry about hardware. They need a high level of service and low latency to ensure clients can retrieve data. Even when requesting more hardware, most of us are asking for more capacity or throughput, not specific models of hardware.

I find myself rarely caring about disks anymore, though recently I was glad I had a flash drive in my bag. I needed to transfer a file to some air-gapped computers, and a quick copy on an old 2019 PASS Summit recording flash drive worked well. That drive was certainly visible, and full of memories.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.

September 9, 2024
Redgate Clone Graduating Images
I wanted my own custom image in our internal Redgate Clone cluster, so I decided to make one. This is an overview of how this works.

This is a series of posts on Redgate Clone, which is a way of quickly and easily creating ephemeral developer images of databases.

Overview

Usually a custom image comes from production, which is masked and subsetted, then available for devs. However, if you are working in greenfield development or want to experiment, you want to start somewhere. I decided to start with an empty instance and customize it.

The process I followed is this:
- create a blank container (create)
- make changes to the container
- save the changes (save)
- create an image from the container (graduate)
I’ve added the RGClone commands in parenthesis after each of these steps.

Creating a New Container

We have an internal Redgate Clone cluster that many of our developers and teams use for spinning up database environments. I connected to this and created a data container using this command.
```
rgclone create data-container –image empty-sql-current
```
This is a blank image we have that contains SQL Server 2019 on it (current was a poor choice for naming) and no databases installed. As you can see below, this created the container and then returned the endpoint in a connection string, including credentials.

When I then listed out the containers running, I could see mine below. It’s the last one, where the container name is the image with a random set of characters appended.

Once this was done, I decided to connect to this instance and then customize things. This container is just a plain SQL Server instance, and appears like that on the network, so I connected to it in SSMS.

Customizing things

Once I connected, I could see this was a standard SQL Server instance, just as if I’d installed SQL Server on a VM.

I had a script to create a database, so I ran that and created a new database and added some data. When I was satisfied with my changes, I decided to save this as a new image.

Graduating the Image

I don’t know why graduate was chosen as the verb, but it was. This means we’ll take an existing container and use it as the source for a brand new image. To do this, we follow a two step process: we save the container state and then create a new image.

The first step is saving, and we use the save verb with the container who’s state we are saving. This creates a revision of this particular moment in time of what the container looks like. I didn’t include a version, since I was learning. The default is rev.x, where x is the next numeric version.

Here’s the command and output.
```
rgclone save data-container empty-sql-current-wyzpthav
```
Now that I have a saved state, let’s use that to create a new image. I used the graduate verb here, but this didn’t work. There is an error that ran off to the side, which noted there was no revision number 1. That’s because the docs didn’t quite explain this, and I wasn’t sure.

The better command is:
```
rgclone graduate data-container empty-sql-curent-wyzpthav –revision rev.1 –name sql-2019-cu22-Northwind
```
where we give the contained and the actual revision used (or returned from the save command. In this case, rev.1 is the revision. I also gave this a better name, which reflects the actual version of SQL Server.

Once this was done, I could see my sql-2019-cu22-Northwind image in the list of data images. I could easily create a new container instance from this, which would be my starting point for more development work.

Summary

This is a basic look at how to use Redgate Clone to create custom images based on your work. Ideally, a bunch of this work would be automated, so we could get updated images of the current state of our development systems as we evolve and change our schema and data.

There are ways to automate things, often starting with a backup, but if you have developers that might want or need to customize a container and want to deploy that to others, this is a way to do that.

Redgate Clone is a part of our Test Data Manager solution, which includes the ability to also mask and subset your data.
September 9, 2024
Branding Yourself for a Dream Job Slides

It was very cool to come back to Syracuse for the first SQL Saturday. It was neat for me, since I used to live here, Shaw Residence Hall at Syracuse University:

In any case, the slides for my talk are here: Branding Yourself for a Dream Job.pptx

If you have questions, please reach out to me or leave a comment below.

September 7, 2024
Container Development Work

On my new laptop, I only use containers as database servers. I made the decision not to install SQL Server or PostgreSQL and instead work on containers only. I’ve written lightly about this, but I set up docker-compose files to load different instances of SQL Server and PostgreSQL (and others) and batch files to start and stop them. I’ve also set dedicated places on my disk where I can drop backup files and access them from the host.

It’s 2024. I moved to containers on my laptop exclusively for databases for the first time this year. This is despite the fact that I like containers, am comfortable with them, and find them handy. Moving from installed database server software to containers took a conscious effort, and it took time to configure everything. Really, it took me a bit of time to think about how I’d want to configure my system so that my work in SSMS went smoothly.

I saw an blog recently from Microsoft on some of the devcontainer work they’ve done. I talked with a few people, who showed me how easy this was to do in ADS or VS Code and ensure your database was included as a part of your project. On one hand I was impressed. On the other, I don’t see many people with projects in ADS/VS Code and the need to spin up/down containers and connect through that tool. Plus, how easy is it to get connected with SSMS or another tool to the container?

If there is any friction in using a new technology, most of us won’t adopt it. Even if we’re forced, we’ll be upset (and less productive) for quite some time if using something is a hassle. I believe in containers, but spinning one up from go-sqlcmd is far different from easily being able to grab a backup file from a friend and get it restored.

While I see lots of companies where developers are excited to use containers, I see relatively few where containerized is the default, or even common, method of working with something. I see even fewer where containers are used for database work. Certainly some people use them, but not most.

Local installs, dev servers, and VMs seem to still be very common. They’re tried, true, and familiar, Most of us like things that are familiar and we fall back to them quickly.

Do you use containers for anything? Testing out software? Actual work? Are you even allowed to use Docker or something similar to run containers? Maybe less likely, but I’m curious, how many of you actually deploy containers in production and with what tech installed in the containers?

Let us know today. I think containers have lots of possibilities, but they haven’t caught on as quickly or widely as I would have thought. Primarily because of friction.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.

September 6, 2024