Tag: storage

There was a time when I knew a lot about the various storage technologies available for a database. It was important when designing a server around the various requirements for size and performance balanced against the limitations of hardware.

The rapid growth of solid state storage and the adoption of storage area networks have changed the game for many of us. We no longer care or think about storage. It’s just a service that we consume in our databases, and while we might demand more IOPS capacity, we often don’t worry too much about how that’s provided to our systems.

I caught an article about IBM and a few enhancements to their offerings. They have a FlashSystem 5200, which is a way of taking their enterprise-class systems and packaging it in a way that organizations of all sizes can afford. In this case, they can get 1.7PB in a 1U package, which seems insane. I can remember working on an Enterprise 2U system that maxed out at 200GB worth of SCSI HDDs.

I wondered if this was a lot in today’s world. A few friends work for Pure Storage, and I’ve met some customers who were very impressed with their arrays. When I looked at a FlashArray//X70 from them, they get 2.4PB in 3U, so the IBM offering is more dense. In terms of IOPS, I’m sure they are both very fast, likely fast enough for most places I’ve worked.

What’s interesting is that I’ve found sysadmins way more concerned about specs and loving the Pure arrays more than the DBAs. DBAs (I think) are stuck with the hardware, and so they are always looking for what the bottlenecks are on a system and can I tune them down to a reasonable level. Or can I just get more storage or IOPs from whoever is in charge of storage?

The world of storage (and networking) has come so far in the last 30 years that the numbers today seem mind-bogglingly fast compared to even a decade ago. It makes me wonder why we have any performance problems at all. Unfortunately, all the hardware in the world can’t compensate for keeping too much data in expensive storage and writing bad code. A little query expertise and archiving structures would likely make all that hardware look as amazing in the real world as it appears in spec sheets.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.

I saw a post recently about the incredible shrinking computer, where a software consultant had written down his predictions for the future. In this case, he writes about a computer shrinking smaller than today, essentially to one chip, with docking stations wherever you need them. We’re already at the place where many of us have an extremely powerful computer in our pockets in the form of a mobile phone. I’m still amazed that I bought a phone recently with half a terabyte of storage.

A few years ago I gave a keynote where I looked at the changes in disk storage, starting with an IBM hard drive being loaded onto a plane. The capacity was 5MB. I remember working with KB floppy disks and MB hard drives, and lots of different connection technologies: IDE, ESDI, SCSI, and more. I’ve seen disks shrink from a 5 1/4″ form factor to mere wafers with today’s NVMe drives.

However, what might be even more interesting is that I don’t even think about drives anymore as being a physical thing. Certainly, my mobile, my laptop, and my desktop have physical drives, but they’re all connected together, essentially giving me invisible disks. I can take a picture on my phone and it shows up on my laptop or desktop in seconds. The disk might as well be invisible.

As I left full-time Operations work, we were starting to see this in our servers, which were essentially invisible VMs stored in some location, on storage that we never laid eyes on. We never saw disks fail because the storage arrays had redundant disks. There were people replacing failed drives every week to ensure that service wasn’t interrupted.

These days we rent storage in the cloud without ever working with the hardware. Most of us who specify the configuration in the cloud don’t think about disks. We think about the total capacity and the IOPs that we need for our system. We care about the results, not the hardware.

Most of us. Some of us care and worry about it, but a lot of the people I know in data work, even as operational DBAs, don’t worry about hardware. They need a high level of service and low latency to ensure clients can retrieve data. Even when requesting more hardware, most of us are asking for more capacity or throughput, not specific models of hardware.

I find myself rarely caring about disks anymore, though recently I was glad I had a flash drive in my bag. I needed to transfer a file to some air-gapped computers, and a quick copy on an old 2019 PASS Summit recording flash drive worked well. That drive was certainly visible, and full of memories.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.