Voice of the DBA

Category: Editorial

  • The Value of Metadata

    A long time ago, nearly two decades ago, I had a manager come ask me about metadata, specifically, what it was. This was a technical manager, who understood software well but wasn’t a data expert. I tried to explain data about data, but at that time, the world primarily dealt with client-server type technologies, with data being something that was primarily about a think. Metadata like types, scales, etc., weren’t seen as that useful. Descriptive items, like we might store in the MS_Description extended property, were rare.

    Today, we use metadata quite often, and I think in many ways we’ve “consumer-ized” metadata with tagging and other structures that let the users of our software add descriptive attributes to the data they store in our systems. In some sense, we get metadata at a deeper level, almost the row or cell level from actual users, as if we were tracking column or entity level markers.

    There is a lot of value in this metadata, both to the technical staff and to business users. This article talks about the value of having detailed metadata for data scientists and similar roles, where understanding the meaning of the data can be very valuable. The more focused data scientists and analysts can be, and the easier with which they can find data, the more productive they can be.

    At Redgate, we work with customers all over the world, but it’s outside the US where we’ve seen the value of metadata grow. Primarily in response to the GDPR, we find many business people want to know about their data, it’s meaning, it’s sensitivity, and really, the risk of data loss. This has expended into our Data Catalog and Data Masker products, which exist to help organizations keep track of their data. It’s a big job, one that isn’t fun, and often has debatable value when starting from the beginning. It can be a large time sink when one person has to track down the meanings of all the data in databases. It’s also not a very interesting job.

    Many find, however, that knowing this information can be quite useful when it’s available. Archive plans, decisions on masking data, even choosing technologies for protection become more focused when you know what data you have. We continue to research and work on finding ways to improve the help software gives users in applying this metadata, as do many others. Privacy and compliance have driven a lot of work in this area, but potentially data science, data lineage, security, and more could find ways to use this information in all our organizations.

    The one thing I think is important is that we maintain this information as a part of our regular work. Customers might do some metadata work themselves (which can be extremely valuable information), but asking most people to classify and tag lots of information is a mind-numbing job. It’s better if we can tag sections of data as we add features to software, whether as developers, or someone else as a part of the software process. That way, when someone goes looking, most data is already partially tagged.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • Do You Know What the Settings Should Be?

    One of the research areas at the Redgate Foundry is in estate management, trying to better understand how people manage an estate of servers. These could be physical servers you own, VMs in a hosted or cloud situation, or even a platform service like Azure SQL Database or AWS RDS. In today’s world with a myriad of choices, it’s easy to lose control of your estate of servers.

    I saw a quote recently from someone that was struggling with their estate. They said: “The moment it goes red, if you don’t know what it should be, then you’re clutching at straws.”

    This particular person was struggling with rebuilding systems after a failure. If one of your VMs dies or gets removed, something that is easy to do in the cloud, do you know all the settings to rebuild it? Not just the CPU and RAM, but all the SQL Server configuration settings you might have changed? These days there are lots of database settings, which ought to be in those backups, but there are plenty of other items that could be hard to recover.

    Most of us don’t experience large disasters in our instances, but we do get regular calls, tickets, and complaints about performance. We might even find out that settings get changed in a team environment that we are not aware were made. Monitoring systems might catch this, but not necessarily every little setting that we care about. Building your own system is complex, and more importantly, I find that ensuring all new instances and databases that get deployed are in your system is hard.

    I didn’t think much of this project when it started, but I realized this is more of a problem for people when I attended a session at SQL in the City London in 2019, where our Foundry presented on a few projects. I had assumed that most people would be thrilled with the Spawn project, but most were more interested in estate management. Lots of interest in having software to ensure you not only know what your settings are, but when they might change and how to get them back.

    Part of building software, especially with DevOps, is ensuring you know how well it is, or isn’t, performing and if the things you change are useful and valuable. Certainly this is important for those DBAs and system administrators, but I think it’s also important to ensure you share those settings with developers. Having all your systems configured in the same way through the software process helps ensure more consistent performance.

    Some sort of estate management is important, and no matter how you might monitor systems, ensure that you are including the various configuration settings as a part of that.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • The Habits of Better Teams

    As a developer, I’ve often worked with a team of others, where we discussed the software we were building, what architectures and patterns to adopt, etc. While we all learned to work together, there were plenty of times that it felt as if we were working separately on code, but arguing often about how to make a decision about a technical approach or a priority.

    As a DBA, I’ve often worked alone, though usually interfacing with different teams as needed. In these cases, I’ve often had to learn to make my own decisions and then justify those to others later. The few times I worked with a team of DBAs, it seemed as though this was still the model, independent work, though with everyone deferring to the senior DBA.

    Building a team takes work, and it can be difficult to do so without the support of management. I’m always amazed at how many managers undermine the building of a high performing team, often in contrast to their words or the company’s statements. Many don’t even realize how poorly they promote teamwork, or how they actively prevent it.

    Assuming you can get support, what do you want in your team? I saw a neat post on the habits of high performing teams. These are some of the things you want to install in all your members. I’ve rarely had high psychological safety, though I think we do now at Redgate. I especially like the idea of leveling experience points. I hadn’t thought about it in these terms, but that makes perfect sense to me.

    That last section, where we assume our peers are competent and intelligent resonates with me. I haven’t always done this in the past, but I have tried hard later in my career to listen more, and to assume others are working with the best information they have. Perhaps that’s different than my information, so I should try to have a discussion, not a lecture or a dismissal of their effort.

    That’s something I wish I would have learned a lot earlier in my career.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • The New Normal Annoyances

    Recently I re-ran an old editorial on annoyances. The world of work has changed for many of us. For the last few months, many of us have previously worked in offices have had to work at home. We often have partners, roommates, spouses, kids, pets, and more that distract us from the environment we’re used to being in while working on the data platform.

    I don’t know how the world of work will change, but I do know it will. As I write this, I’m not sure when I’ll actually be able to go back to an office to work. While I don’t go often, I am looking forward to spending a few days here and there with others when I’m able. What I’m not sure about is whether I’ll see lots of people or just a few. Will there be rotations of who’s in the office during the week? Will I, as an outsider, be able to meet with the large variety of developers I’m used to seeing, or am I an infection vector to be avoided?

    I suspect that world of remote work will grow for many data professionals. It’s been years since I needed to touch a physical server, and with the changes I’ve seen in many environments, it might not be quite as critical for many of us to show up and meet in a room every week. At the same time, I do think we’ve lost some of the ability to easily collaborate and coordinate our thoughts while we’re remote.

    Now that we’ve had a few months of a different way of working, I wonder if you have any new annoyances that bother you. For me, having more people at home has been a bit of a challenge as noise is a problem at times. It’s not bad, but it’s different than it was in 2019.

    I’ll also say that while meetings seem to be starting on time more often, I find we still have plenty of audio issues, and sometimes video issues. I also find that it’s hard to decide when to talk and when I want to avoid interrupting someone. It seems that we have different delays in hearing each other at times, so we end up with a bit of a talking collision. Overall, however, I’m impressed and surprised at just how smoothly things are going.

    I’m sure many of you are finding new challenges at home. Some from virtual school, perhaps some frustration still from your setup at home, or maybe there’s something else. A few of you might not like the remote meetings and still not be used to them. I think that no matter how you feel, many of us are still going to need to adjust the way we work and learn to cope with our new annoyances for awhile.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.