Voice of the DBA

Category: Editorial

  • Dealing with Technical Debt

    Stop for a minute and think. How much technical debt do you have in your code? Let’s make this easy, take a break from this, close your eyes, and take thirty seconds or so and think about just your code. How much would you rewrite or redo if you could?

    Now, stop and think for another minute. How do you measure the amount of debt? What’s your metric? Leave a note in the discussion after you finish this and let us know.

    There’s an article from a project manager about technical debt and how his team decided to measure this in their organization. They actually set a series of metrics for things they cared about and assigned an actually dollar, well, pound sterling, value to each. They then totaled these for all projects and came up with a total debt number. They then track this, incurring new debt or paying off older debt over time.

    I think that’s an interesting idea. I don’t know that the debt needs to be money, but I do think money is a little better than some abstract metric like tickets or requests. Money is something many of us are familiar with and we hold debt. Assigning a concrete value does let us measure where we stand in a way that we can better relate.

    Just like financial debt, not all technical debt is bad. Sometimes debt lets us get more things done sooner than we might otherwise be able to do. We do need to take shortcuts at time in development to get a prototype working, or deliver a feature or meet some other goal.

    It is easy to overdo things, and just like with finances, overextend ourselves. Paying down that debt can prove to be a struggle in both cases, perhaps even crippling our ability to tackle any other projects.

    If you think about technical debt now, do you view things differently? Do you have a measurement system? Can you tell if your incurring or paying down debt? It might be valuable to implement something and ensure you don’t keep building up more than you can handle. It might even be something to show your manager and get some time to fix a few bugs.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.6MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Badly Encrypted Databases

    I ran across a blog about encrypted databases linked from Bruce Schneier’s blog. I follow his musings and writings on security ,and he recommended we read it with this sentence: “Even the summary is too much to summarize, so read it.” Good enough for me, so I clicked the link and read about encrypted databases.

    I like the idea of stronger encryption in databases, and I’ve given a few talks on the subject. At times there are attendees that will debate that encryption in the database doesn’t do a lot of good. Often they dismiss the idea of TDE, since administrators can still read the data and break the encryption, and normal users aren’t affected. Many also note that database encryption does nothing for data on the wire, which is true. Most people want to do the encryption and decryption on the client, which has other challenges and is fairly hard to do well.

    I think that security a series of layers, and as noted by the author of the blog, most criminals are lazy. If they can copy a backup file or data file, they’ll just do that and read the data. TDE isn’t perfect, but it does limit these simple attacks. Always Encrypted was developed to try and make it easy to include encryption from the client side, but in SQL Server 2016, it has lots of limitations. In SQL Server 2019, we get secure enclaves, which should help adoption somewhat, but we will see once developers start to experiment with the feature.

    The blog talks about problems with encryption, spending quite a bit of time on approximate database reconstruction, which is essentially guessing data values with some information and by watching queries and results. It’s somewhat fascinating, and also scary, but complex and likely requiring lots of queries. To me, this is an area we ought to focus, and really an area that all our protocol libraries and possibly database firewalls (or built in limits) ought to focus efforts. We shouldn’t be most clients to make large queries of all data in a table. Really at this point, we ought to have build in limitations of queries to ensure that users are exporting all data from a table. I’d like some throttle that might prevent the return of very large result sets to clients.

    At the same time, there ought to be some way to analyze the queries coming in and see if an attacker is “guessing” values. I won’t pretend to know how we might do this, but it would seem that in the same way we detect lots of login attempts, we could have some alert being raised when we had xxx of the same type of query in yyy time. That might alert us to potential problems.

    Or users running lots of searches.

    I don’t know the best way to protect data, but I do know that too many of us aren’t doing a good job of this. We need to get better, both in production and development environments. We need to be better at protecting databases and the data within them.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 5.0MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Database Brainstorming

    I read a post recently that captured my attention for more time than I expected. The piece fascinated me, and I sent it to my son and a couple friends. In fact, reading it caused me to stop working for a bit as I thought through some of the concepts and ideas. It’s about divergence and convergence, and how you can use these ideas when trying to be creative. I recommend you take a few minutes and go through it.

    Is this something we ought to do a bit in database design? Should we spend first some time on divergent, creative thinking for how to implement some new entity or design some schema? Do you ever do this in your job, especially in a team? Or do you just push forward with your first thoughts, relying on experience and instinct? That’s what I’ve often done.

    We certainly need convergence thinking. We have to get things done and deliver software. Most of us are probably good at the convergent side of thinking and writing code or queries or configuring systems. Most of us probably do some amount of divergent and convergent thinking at the same time.

    However, I’m not sure that’s the best way to tackle creative work, especially when building complex systems or working to solve new types of problems. There is some value in separating out the brainstorming portion from the actual, let’s write some code and see if it works. Getting a good list of potential ideas and spending a few minutes thinking about them before writing code helps us to avoid getting trapped pursuing into the first thing that works and possibly see other approaches.

    In today’s world of DevOps, trying to move faster and get small chunks of work to the customer quicker, we are under pressure to do more convergent work, quicker. If we want to ensure we do model our entities well and follow good database design practices, we need to spend some time with divergent thinking without doing work, perhaps speculating on potential future design challenges in advance. If we periodically spend this time, then we can rely on our previous brainstorming when it is time to converge and get things done.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.3MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Common Data Challenges

    Working with data has proven to be a challenge for most of my career. It’s been fun, and certainly fulfilling, but there are constant challenges involved. Let’s take away the hardware and admin challenges of keeping systems running, backed up, and performing well. I’m thinking today of the struggles of just data.

    There is a short article that talks about three common data issues, and these are some of the same ones I’ve struggled with for most of my career. Are these challenges any different than they were 20 or 30 years ago? I’m not sure, and I was working with databases and software nearly 30 years ago.

    ETL is a constant challenge, even today with tools like SSIS and Biml that make it much easier to build flows that migrate data from one database to another. ETL is such a challenge that many people make a very comfortable living helping organizations meet their every changing needs to move and prepare information for end users.

    The other challenges noted in the article are getting a complete data picture because of missing data and not trusting or believing in data. The latter hasn’t been as much of a problem for me.  I might describe it differently as more often we aren’t sure what weight to place on certain data. The world is messy, and often we collect data that we think might be valuable only to realize later that it doesn’t mean what we thought or our our hypothesis was incorrect in the first place.

    I think the challenges are part of what makes this work interesting. Our employers and clients might view the effort and time involved as frustrating, and I wish I had solutions to make our process quicker and smoother. Actually, I do think that advances like SSIS have made things quicker, but the world has grown more complex. We deal with more data from more systems, in the still chaotic, messy formats of the world.

    What are your challenges with data today? Are they getting better, worse, or still the same as they always were?

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.4MB) podcast or subscribe to the feed at iTunes and Libsyn.