Voice of the DBA

Category: Editorial

  • Normal Data

    I thought this was an interesting post: Your data is not normal. It talks about the fact that we often assume some normal distribution when working with data. We seem to think we can extrapolate, often in a linear fashion, based on past experience. We often use functions and write queries that tend to think of normality as having some standard distribution.

    However that’s not often the case. It’s not necessarily even the best way to begin looking at data for patterns. We might need to apply a number of different types of algorithms to a set of data to see what types of analysis might be appropriate. Not sure what I mean? Let me give you a few examples.

    As a speaker, I often get feedback from my sessions as a series of forms where people have rated the session on a scale and possibly included comments. Many speakers receive the data in the same format, but I’ve often seen speakers get one form that rates them poorly, or has a complaint. The speaker will then view that one item as somehow representative of the session. This despite the fact that they might have ten forms with high ratings. They’re somehow assigning more weight to the negative rating then the all the others. There’s perhaps some value here in doing so, since there might be a legitimate complaint, but often we need to discard this one form as an outlier.

    As another example, I worked with an import firm that was trying to estimate future purchases based on past ones for ordering purposes. Initially they looked at an average of the last three months (rolling) to place future orders. However I pointed out that we had some seasonal fluctuations in our business. There were a few times a year that the rolling average would dramatically under or over estimate the product we needed. As we dug deeper into our historical data, we learned that a better estimate required some level of complexity in our formulas, but we also realized we needed to allow for human overrides when other information was available. That was a fun project, but one that was quite complex when we analyzed the data.

    There are numerous other examples I could give, but the point is this. Learn about your data. Dig into patterns, and don’t assume that normal for one set of data looks anything like normal for other sets.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.0MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

  • Time for Learning

    I was chatting with a friend recently about our careers and how much we have learned over time. We are both interested in continuing our education in technology and debated how we might proceed. I mentioned that I was thinking to spend some time following Colleen Morrow’s journey to pass the 70-463 exam, which is one exam I need to complete a certification. I think that it is hard to work through a topic without having some plan for doing so, and I prefer to follow a book or another person’s plan to guide my learning.

    My friend talked about their experience with Pluralsight, which I’ve also used. They mentioned that it’s exciting to get started, and you certainly can work through a number of courses over there, but at some point you’re just going through courses. You really need to stop and practice the skills you’ve listened and to and been quizzed on, otherwise much of that knowledge won’t be retained.

    I’ve found that to be true as well with classroom training. A week’s worth of knowledge is too much to learn at once, and only with a lot of practice, and substantial documentation available after the training, does the class really do you a lot of good. However, that’s the way that so much training is organized in the technology space.

    Ideally, I think that the best way to actually teach someone a skill is the way that many college classes are organized. Learn over time, slowly, with a few skills taught and then practiced. Repeat that for months and you’ll gain knowledge. However who has time to spend six months learning something like SSIS, a skill at a time.

    You do, and I do. In one sense, our Stairway Series provide this, as does Pluralsight. These methods give you the ability to work through a subject slowly, in a guided way, at your pace. What these methods don’t do is push you to keep moving, which is perhaps the most difficult part of learning a new skill for most people.

    Email reminders, pings, and all the possibilities available with technology likely won’t work for most of us. However I might suggest the best way would be to find a few friends or co-workers looking to acquire the same skills and press each other to stick to a schedule. It’s worked for exam study, and I bet it would work here, applying some pressure for you to keep up with your friends, and continue your learning.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.9MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Deploy the Database First

    One of the patterns I’ve seen in some environments is people are trying to deploy changes rapidly to their database backed applications. That’s not news, but what is interesting is some of them are staging the deployment of the database changes first. Not as in I deploy database changes at 8:00pm and then application changes at 8:30pm. These people try to deploy the database changes on Monday, and their application changes will follow on Tuesday, Wednesday, or even a month later.

    It’s an interesting concept, though I think it requires a lot of forethought in your designs, as well as very tight coding from your front ends that won’t be disturbed by extra columns in a result set.  That’s not easy to do, but it’s certainly possible, and it can even be useful if you deploy feature flags extensively in your application.

    As we become more dependent on databases for our applications, and our customers expect systems to be running constantly, I think it behooves us to find ways to make alter and enhance our applications without downtime. While there are patterns to keep applications running when the database goes down, I expect that the reality is that we need to find ways keep the database up as we alter it, which for me means making non-breaking changes.

    I think it’s possible in many cases to upgrade a database over time by carefully planning your schema changes and accounting for those changes in your front end architecture. However it’s not easy, as I mentioned, and you do need to commit to very stable and careful programming practices for your developers.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 1.9MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Which Problem Are You Solving?

    One of the problems I’ve seen for years in computing is people seeing the forest or the trees. Not that either is good or bad, but far too often I’ve run across people that only see one of these, meaning they examine problems from a high level perspective, or a very low level perspective. While either of those can help you solve your issues, there will be a problem or a time when you need to examine the situation from another perspective. Knowing when to look at a higher or lower level can be invaluable in troubleshooting.

    This past week I ran across a great post from Grant Fritchey called Understand the True Source of Problems. In this post, Grant talks about some strange DBA “best practices” that were implemented to solve problems, but aren’t necessarily the best of anything. It’s an interesting list, and you might read it for entertainment, though I hope none of you are following these ideals at your place of employment.

    However the thing I liked about this post is that in the beginning Grant begins with an old doctor joke, which is still funny to me, but it often is the approache taken by people trying to solve problems. They just avoid the issue, and never solve it. That can work in the short term, but I’ve seen this build up over time to the point where people are afraid to make any changes to a system or code because of the fear of causing some unknown issue.

    I don’t think that every issue needs to be solve to the root cause. There are plenty of times something goes sideways on one of my computers and a reboot fixes the issue. I’m fine with that, but if it occurs more than two or three times, I do need to solve the problem. I need to be able to dig deeper to diagnose what resources are failing, what performance issue exists, or maybe what line of code is broken. Perhaps I need to step back and examine if some other part of my system, like the disk or network, is causing an issue that appears to be isolated to code. In other words, I need to be able to look at the wider environment (the forest) or dive into the particulars of a process (the trees).

    Knowing how to strike this balance is a bit of an art, and it’s what really makes the best problem solvers experts. I can’t teach you that part of computing, but I can teach you this. When you don’t think you’ve completely solved the issue, stop for a minute and take a wider or narrower view. Or maybe both. Look for what you might not have thought of. If you’re like me, there are probably plenty of possibilities that come to mind when you stop and re-examine the issue. Then work your way through those items, discarding the ones that aren’t relevant. Sometimes opening your mind a bit is the best way to find the exact cause of an issue.

    Steve Jones