Tag: data

I had never heard of data debt until I saw this article on the topic. In reading it, I couldn’t help thinking that most everyone has data debt, it creates inefficiencies, and it’s unlikely we’ll get rid of it. And by the way, it’s too late to get this under control. I somewhat dismissed the article when I saw this: “addressing data debt in its early stages is crucial to ensure that it does not become an overwhelming barrier to progress.” I know it’s a barrier, as I assume most of you also know, but it’s also not stopping us. We keep building more apps, databases, and systems, and accruing more data debt. Somehow, most organizations keep running.

The description of debt might help here. How many of you have inconsistent data standards, where you might define a data element differently in different databases? Maybe you have duplicated data that is slow to update (think ETL/warehouses), maybe you have different ways of tracking a completed sale in different systems. Maybe you even store dates in different formats (int, string, or something weirder). How many of you lack some documentation on what the columns in your databases mean? Maybe I should ask the reverse, where the few of you who have complete data dictionaries can raise your hands.

For most of my career I’ve heard a couple of terms that I’ve never really seen implemented. There’s the famous “single version of the truth” for a system, which seems to break down whenever we add a reporting or warehousing system. Even inside a single database, often an OLTP one, it’s hard to get a truth because values are changing so fast. The other term is MDM (master data management), which promises to ensure that every element is tracked and tagged the same way. No misspelled customer names or outdated addresses. There have been no shortage of products I’ve seen to help people tackle this problem, but ultimately I think the amount of data debt is too high. When we realize we need MDM, we’ll never pay down that debt, mostly because too many developers have too many habits and legacy ways of capturing data that will never get integrated into any MDM dictionary.

The article seems like a great academic set of principles. Make sure you label all your data. Put governance in place, with good access controls. Train workers, establish accountability to properly manage data. Invest in scalable architectures. How many of you can add scale to your system easily? It’s always taken me jumping through a variety of hoops to do that. The cloud makes it easy.

For a month. Then when the bill comes, you’ll be scaling back down.

Really, the chaos of the real world, where organizations are not one thing, but a large number of people and groups, each with their own goals and processes, just trying to get enough done to keep the organization moving forward is where we live. There’s no real time to deal with data debt.

Except if you’re the ETL person. We mostly pay you to move data around and clean it as best you can. At least then the problem remains hidden from the report readers, who trust you’ve actually done the T portion of ETL correctly.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.

There’s an article at Forbes about the Five Things Business Leaders Should Know about Databases. Disclosure, it’s by my boss, but I think it’s still a good read. These are points we’ve learned from research and work with customers and prospects at Redgate Software. These points come from you, as well as from executives with whom we work, but there are so many people in organizations who don’t think about the complexity of data, so it’s a good one to pass along.

The five things are (if you don’t want to read): data is growing, getting more complex, there are multiple database platforms in most estates, teams struggle (duh), and data is a business issue. Most of us know about the fourth one, often because we may feel overloaded with work. We might also feel a lot of stress in trying to keep up with not only the workload but also trying to learn more to support the ever-growing variety of systems it seems our employer wants to put into production. I regularly talk with customers whose developers keep wanting to try out a new, shiny database platform in the cloud (or add new features from their existing platforms).

Wait, not try out. They’ve already deployed some production data there and now want other developers or operations people to work with their system.  Often lots of the staff isn’t familiar with the platform or feature, even the people who decided to implement it. Is anyone familiar with that situation?

The digital transformation and importance of software isn’t lost on most executives, but I find far too many that don’t place the same importance on the data that powers their software or the database platforms that support all software. Data is important, and ensuring it is available to software, protected, secured, and available is critical. That also means that the process of developing and managing the database portion of your software matters.

I think Database DevOps is important, but it’s not a panacea to buy a product. I’d love to show you what Redgate Software can do here, but the main thing I stress to clients is that you need to get your database development into the modern era and match what software developers do. That means a smooth process, with version control, that ensures you deploy changes in a way that doesn’t impact customers.

This doesn’t mean just go faster. This means bringing along data modeling, good architecture, and performance testing. Those are the same things we’ve been doing (or should have been doing) for decades, but we need to ensure these are still a part of whatever process we choose and included in any automation we implement.

Pass the article along to your management, and be sure they understand that all these points are important. Particularly the fourth point because if your staff isn’t supported and trained, the rest of the business will suffer.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.