Voice of the DBA

Tag: Database Weekly

  • The State of Data Growth

    About a decade ago I did some presentations on Filestream and Filetable in SQL Server.  These were technologies that Microsoft added with the idea that lots of data would need to be linked to data in a database, but wasn’t necessarily a good fit for a relational table. At the time, I looked at the growth of data, which was incredible back then. Statistics I had from that time were from the invention of the printing press for the next 500 years, we created 1 EB of data. In the 2011/2012 time frame, we created 2EB each day.

    That’s changed. I saw a blog on the story behind the explosive growth of data, which looked at a recent IDC report that showed we created and replicated (a strange term) about 64ZB of data in 2020. With a ZB being 1000 EB, we can see that a decade has brought us multiple orders of magnitude of data growth.

    That doesn’t mean we need to store all this data. The report notes that a lot of this is consumption data, which is transferred, but not saved. This means that we do need strong networks, but that most of the time the data is being viewed and not actually stored. That’s certainly the case with things I see on Netflix, where I may end up transferring part of a show multiple times to get through it, usually because I fall asleep.

    The estimate for the next 5 years, through 2025, does indicate that we expect to need 200% more storage capacity and that a lot of data storage is going to be at the edge, in devices and server machines, as well as in the core, made up of data centers, cloud and private. While plenty of this data will be outside of databases, and plenty will be in non-relational stores, this does tell me that working in the database business as a career is one that’s going to be in demand for a while.

    I’ve enjoyed my career working with data, and I expect to keep doing so for the foreseeable future. I am confident there will always be work for those that know how to work with data in a variety of situations. I also expect a very high demand, and high compensation for those that do it well.

    Steve Jones

  • Deploying SQL Server Automatically

    I’ve had to install SQL Server many times over the years. Often it has felt that most of these installs were one-offs, a dev server, a new QA instance, a production server for a brand new application. A few times I’ve had to recover from disaster, including restoring master, but often, I just installed SQL Server manually because that was quick and guaranteed. I knew what needed to be done for most of the instances.

    However, I couldn’t be sure. In the past there were relatively few settings that were ever changed outside of the master database, but today there are more, and perhaps more importantly, the tolerance for making mistakes with any security missteps, is low. I’ve spent a bit of time learning to do unattended installs, and I’ve tried systems like FineBuild for installing SQL Server. In any size organization that might want certain standards set up, using one of these systems is important. Perhaps one of the better reasons to do this is ensure that your dev and test systems are configured the same as production to prevent any silly mis-configuration problems.

    This week I saw a post on using Ansible to install SQL Server on Linux, with all the various configuration. While I do think that it’s easier to install SQL Server on Linux programmatically, there are a number of items to set and configure. Following along the install, it’s interesting how much more this feels like something a developer would do, which is how many modern Operations groups approach installs and upgrades. Everything needs to be done without a human directly involved, often because of scale. There are so many systems we manage, often a mix of VMs, cloud, and local systems, which mean that a consistent, programmatic way of installing instances is needed.

    That’s likely one of the most important skills for a modern system administrator. We need to learn to use tools to get our work done. Whether these are free, open-source, or purchased, using a tool to work at scale is more important today than ever before. Even if you are part of a development group, learning to manage systems with code can help ensure you can scale and grow quickly, and more importantly, hand off this job easily to someone else when you reach that point.

    I think there are plenty of homemade scripts and tools that can help here, but almost every time I’ve seen one of these, it’s not very portable to other staff, especially if the author isn’t available. Too often these tools take shortcuts or are specifically tailored to the current environment and not the future one. These days, for many functions we tackle there are extremely well built tools available at no, low, or modest costs. I’d encourage anyone that is handling these tasks to learn about the tools available and pick one. I don’t know that I think any of these are necessarily easier to use, but they are all well documented and capable of handling most of your install, update, and configuration chores.

    Steve Jones

  • Deceptive Visualizations

    I’m sure there are plenty of examples, but I’ve seen deceptive data visualizations far too often in mass media. I don’t often watch the news, but at times I’ve been at the gym, saw a graph on a television and been a little surprised. The issue is similar to what’s discussed in this SAS article on deceptive graphs.

    In this case, the problem is likely what the author describes. An axis was moved, which what I often see in the news. Rather than starting at 0, often the image might start at 20 or 30 on a scale of 1 to 100. As a result, the differences between two bars, lines, or whatever is on the graph is distorted.

    This might be unintentional, as more and more graphing tools try to “size to fit” in a space, and can alter graphs, but in many cases, I think the author of the report is deliberately emphasizing something to evoke a reaction in an audience.

    I suspect this also happens inside companies. Someone might change scales or axis starting points to emphasize or de-emphasize some part of the data. I could see sysadmins wanting to de-emphasize downtime, but they might want to emphasize a cost savings to get management to make the decision they would prefer.

    The article linked above makes a good point. If you do this, your credibility is undermined if someone notices. Maybe the media doesn’t care, but I know that trying to deceive the person that signs your checks, or has some impact on your future employment, is a bad idea. These days, it is hard to find good help, but that doesn’t seem to stop a lot of companies from getting rid of bad help a times.

    Present data the way you’d want to see if your positions were reversed. Or if you wanted a decision to be the opposite of that you are hoping for in this instance. You might get a short term win, but in the long term, I would argue your credibility and reputation are worth more than a short term win.

    Steve Jones

  • Back to Basics

    This week I saw a post from Randolph West that covers binary and hexadecimal. It seemed basic and simple to me, but I’m old. I started learning about binary and hex in high school, where we used to tap our fingers to count in binary. Up was “0”, and down was “1”, and with one hand, you count to 31 (from zero). At the time, I thought it was important to know binary and hex to work with computers.

    These days, I’ve rarely had to deal with binary and hex. They aren’t useless in programming, but they are much less used in these days of GB of memory and large data structures. We don’t encode things as often as we used to. Usually now hex just means I’m dealing with varbinary and need to cast it (or render it in some other way).

    During the last few years, I’ve seen some efforts to get back to basics and publish more information that helps people get started or fill in gaps for areas that someone might have missed. Years ago Ed Leighton-Dick started the SQLNewBlogger challenge. I participated, and continue to write some basic posts every week or two. Grant had his database fundamentals series, and SQLServerCentral has published the Stairway Series for some time, giving people a way to get started in some new area of the data platform.

    I think it’s important that we continue to cover the basics in all aspects of the data platform, as well as other areas. The world of technology is so wide, and new people are always starting their journey in their industry. Even for those of us that have worked with SQL Server or some other area for a long time don’t know everything about the platform. We often have as need to learn more about some niche and need some basics to help us. I think Azure Data Factory as a perfect example of this. Many people who worked with SSIS for years might need a few primers on the differences with ADF to understand a slightly new paradigm. Much of cloud computing needs basics, and constant updates to those pieces.

    I know that the basics are always popular at SQL Saturdays and other conferences. The basics often get the most reads as articles, and I am hopeful that we will continue to see those with knowledge remembering that the basics are often the most important part of learning to build better software and systems for the future.

    Steve Jones