Voice of the DBA

Category: Editorial

  • DR as a Service

    It’s not the first task when I start a new job, but often as a DBA or developer, I usually ask about Disaster Recovery (DR) plans sometime within the first six months. If I’m a DBA, of course I need a plan. If I’m a developer, however, I still need to understand how this might work as it can affect how I build the software and prepare for networking, machine changes, etc. Even if I don’t concern myself with production DR, I usually do want to make sure the VCS repos are being protected, which is something I’ve found isn’t always being handled.

    I have had to build and test DR plans as a member of an Operations team in the past. While my plans and practice are nothing like Google’s large exercises, they often reveal some issues, even when we duplicate the service without touching production. I’ve usually found the brainstorming and debating of the various ways to build a plan to be fun. Arguing for money and then actually implementing plans is less interesting, but the testing is a great challenge. I’ve had some fun days offsite where we try to recover systems and find all the little things that we take for granted in our production environment.

    These days there are companies offering DR as a Service (DRaaS), which is an interesting concept. I found an article from Michael Otey that talks about the features you might want to look for if you contract with a vendor. In the past, I would never consider this, but the more we advance in the world with cloud infrastructures and even full service co-location vendors, the more I think DRaaS makes sense.

    I wouldn’t necessarily take anyone’s word that their service meets my needs, so thinking about the requirements, and then working through a few PoCs (proof of concept) is likely very important. We do a lot of PoC work at Redgate to help customers evaluate whether Compliant Database DevOps is a good fit. I think this is important for software development, but even more important for DR plans. After all, downtime is expensive, and the last thing you want to find out when troubles arise is that some critical piece of infrastructure can’t be easily duplicated.

    I’ve used DR companies in the past, with their own physical facilities. They have impressive capabilities and marketing, but the mixing of their skills with my systems has often been rocky and lead to changes in our plans, contracts for new or fewer services, and often updated documentation for junior staff. After all, I usually expect DR situations to occur when I’m on vacation, so I plan for that.

    You never know when you’ll need to execute DR plans. It pays to think about this ahead of time and periodically test yourself and your staff.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • The Value of Metadata

    A long time ago, nearly two decades ago, I had a manager come ask me about metadata, specifically, what it was. This was a technical manager, who understood software well but wasn’t a data expert. I tried to explain data about data, but at that time, the world primarily dealt with client-server type technologies, with data being something that was primarily about a think. Metadata like types, scales, etc., weren’t seen as that useful. Descriptive items, like we might store in the MS_Description extended property, were rare.

    Today, we use metadata quite often, and I think in many ways we’ve “consumer-ized” metadata with tagging and other structures that let the users of our software add descriptive attributes to the data they store in our systems. In some sense, we get metadata at a deeper level, almost the row or cell level from actual users, as if we were tracking column or entity level markers.

    There is a lot of value in this metadata, both to the technical staff and to business users. This article talks about the value of having detailed metadata for data scientists and similar roles, where understanding the meaning of the data can be very valuable. The more focused data scientists and analysts can be, and the easier with which they can find data, the more productive they can be.

    At Redgate, we work with customers all over the world, but it’s outside the US where we’ve seen the value of metadata grow. Primarily in response to the GDPR, we find many business people want to know about their data, it’s meaning, it’s sensitivity, and really, the risk of data loss. This has expended into our Data Catalog and Data Masker products, which exist to help organizations keep track of their data. It’s a big job, one that isn’t fun, and often has debatable value when starting from the beginning. It can be a large time sink when one person has to track down the meanings of all the data in databases. It’s also not a very interesting job.

    Many find, however, that knowing this information can be quite useful when it’s available. Archive plans, decisions on masking data, even choosing technologies for protection become more focused when you know what data you have. We continue to research and work on finding ways to improve the help software gives users in applying this metadata, as do many others. Privacy and compliance have driven a lot of work in this area, but potentially data science, data lineage, security, and more could find ways to use this information in all our organizations.

    The one thing I think is important is that we maintain this information as a part of our regular work. Customers might do some metadata work themselves (which can be extremely valuable information), but asking most people to classify and tag lots of information is a mind-numbing job. It’s better if we can tag sections of data as we add features to software, whether as developers, or someone else as a part of the software process. That way, when someone goes looking, most data is already partially tagged.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • Do You Know What the Settings Should Be?

    One of the research areas at the Redgate Foundry is in estate management, trying to better understand how people manage an estate of servers. These could be physical servers you own, VMs in a hosted or cloud situation, or even a platform service like Azure SQL Database or AWS RDS. In today’s world with a myriad of choices, it’s easy to lose control of your estate of servers.

    I saw a quote recently from someone that was struggling with their estate. They said: “The moment it goes red, if you don’t know what it should be, then you’re clutching at straws.”

    This particular person was struggling with rebuilding systems after a failure. If one of your VMs dies or gets removed, something that is easy to do in the cloud, do you know all the settings to rebuild it? Not just the CPU and RAM, but all the SQL Server configuration settings you might have changed? These days there are lots of database settings, which ought to be in those backups, but there are plenty of other items that could be hard to recover.

    Most of us don’t experience large disasters in our instances, but we do get regular calls, tickets, and complaints about performance. We might even find out that settings get changed in a team environment that we are not aware were made. Monitoring systems might catch this, but not necessarily every little setting that we care about. Building your own system is complex, and more importantly, I find that ensuring all new instances and databases that get deployed are in your system is hard.

    I didn’t think much of this project when it started, but I realized this is more of a problem for people when I attended a session at SQL in the City London in 2019, where our Foundry presented on a few projects. I had assumed that most people would be thrilled with the Spawn project, but most were more interested in estate management. Lots of interest in having software to ensure you not only know what your settings are, but when they might change and how to get them back.

    Part of building software, especially with DevOps, is ensuring you know how well it is, or isn’t, performing and if the things you change are useful and valuable. Certainly this is important for those DBAs and system administrators, but I think it’s also important to ensure you share those settings with developers. Having all your systems configured in the same way through the software process helps ensure more consistent performance.

    Some sort of estate management is important, and no matter how you might monitor systems, ensure that you are including the various configuration settings as a part of that.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • The Habits of Better Teams

    As a developer, I’ve often worked with a team of others, where we discussed the software we were building, what architectures and patterns to adopt, etc. While we all learned to work together, there were plenty of times that it felt as if we were working separately on code, but arguing often about how to make a decision about a technical approach or a priority.

    As a DBA, I’ve often worked alone, though usually interfacing with different teams as needed. In these cases, I’ve often had to learn to make my own decisions and then justify those to others later. The few times I worked with a team of DBAs, it seemed as though this was still the model, independent work, though with everyone deferring to the senior DBA.

    Building a team takes work, and it can be difficult to do so without the support of management. I’m always amazed at how many managers undermine the building of a high performing team, often in contrast to their words or the company’s statements. Many don’t even realize how poorly they promote teamwork, or how they actively prevent it.

    Assuming you can get support, what do you want in your team? I saw a neat post on the habits of high performing teams. These are some of the things you want to install in all your members. I’ve rarely had high psychological safety, though I think we do now at Redgate. I especially like the idea of leveling experience points. I hadn’t thought about it in these terms, but that makes perfect sense to me.

    That last section, where we assume our peers are competent and intelligent resonates with me. I haven’t always done this in the past, but I have tried hard later in my career to listen more, and to assume others are working with the best information they have. Perhaps that’s different than my information, so I should try to have a discussion, not a lecture or a dismissal of their effort.

    That’s something I wish I would have learned a lot earlier in my career.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.