Voice of the DBA

Tag: data quality

  • Generating Our Own Bad Data

    I’ve worked in a few different industries and for a number of companies throughout my life. In almost every position, employees were given goals to improve the performance of the company. Often these were revenue-related goals that were supposed to drive additional sales. The one thing I learned when these goals directly related to sales was that there would always be unintended consequences that might skew business in a way that management didn’t expect. This is especially true when the rewards for meeting goals lead to direct compensation for employees. In fact, additional compensation has always led some percentage of the staff to work towards improving their own pay, even with fraudulent or unethical activity. Hopefully this has usually been a small percentage of people, but perhaps that’s not always the case.

    What does this have to do with data and databases? I recently read about fraudulent activity at Wells Fargo, where a sales policy caused a number of problems. Employees used real customer data to open fake accounts in order to meet sales quotas, generate fees, and earn bonuses. While certainly criminal and immoral, this also shows what I’ve believed for a long time. While computers allow work to take place very quickly, computers also allow mistakes to be made at incredible rates. While these weren’t mistakes, the computing systems allowed this fraudulent data activity to take place at a large scale. One that resulted in not only fraudulent charges to customers, but fraudulent reporting to investors, unearned bonuses paid out, likely unearned promotions, and who knows what else.

    To be fair, this type of criminal behavior could have happened before the age of computers, and likely could have lasted just as long with employees forging paper documents to split accounts, open credit lines, or any other activity. However, I’d hope that computer systems would have detected this sooner. Too detect the bad data, customers really need to be tied into some sort of reporting system that can audit their accounts, that can examine credit bureaus, and warn them if anyone is making unauthorized changes.

    We have seen similar unintended consequences when companies have offered bug bounties to developers. More bugs appear, and not surprisingly, more bugs are fixed, with more payouts ensuing. I’d like to think that most people would be driven to be professional, but time and time again we find enough of a percentage of employees aren’t, and the programs are doomed to failure. We need to do better, though I’m not sure what would help.

    Protecting data from insiders and outsiders is our responsibility as data professionals. However, insiders can cause more problems, which is why I would hope that the auditing and transparency of data platforms would be a priority for vendors across the next decade. In my mind, not enough attention has been paid to these areas, and our systems are woefully immature in these areas.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.4MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • A Nice Refund

    Why don't these kinds of mistakes happen to me?

    This editorial was originally published on Sept 12, 2006.

    How’d you like to be the DBA for a system thatmistakenly gives out $318 million? It probably wasn’t the database, but more likely the code for some application that caused this to occur in the IRS system, but it’s something that becomes more likely in SQL Server all the time.

    SQL Server 2005 is the first time that SQL Server can be easily used as a complete application development platform. With the addition of the CLR, the Service Broker, Database Mail, web services, and more, SQL Server 2005 could be your database server, application server, and only server, with no other software running.

    While application developers will likely write most of the code, including lots of the CLR functions, procedures, and other constructs, who do you think will be responsible for the system? That’s right, the DBA. You’ll be the first and last line of defense and the one that will likely shoulder a lot of the blame for issues. Fair or not, I expect that DBAs will become more likely the scapegoats for issues.

    This means that as a DBA, you really want to beef up your skills as well as clearly delineate lines of responsibility. You should be able to read the code behind assemblies you add to your server, as well as understand how the SQL Server subsystems fit into an application. But you also need to be sure that you let management know when you are overloaded, when you can’t review code, or when you are concerned about possible issues with code.

    I’ve felt for years that it was a matter of time before some computer professionals will need to become bonded because of their responsibilities. I’m not sure when or where this will start happening, but I wouldn’t be surprised if it starts with DBAs.

  • Data Quality

    Beautiful Colorado

    Today we have an editorial reprinted from Dec 12, 2005 as Steve is on vacation.

    My wife and I were out looking at new houses this past weekend and I was amazed at the data quality issues that we found in various listings. These weren’t new houses as I’m not quite in the place where I can afford a new house with some land, at least not the 30+ acre plots we were looking at.

    What we found as we went from place to place was an amazingly inconsistent amount and quality of data in the listings. Before all the IT guys that work in real estate come after me, I’m not blaming the applications, the developers, etc. This is really a data quality issue probably from the data entry people.

    As we’ve researched properties and looked at the information online, we’ve often encountered conflicting information between what we’ll actually see in the listings and what the property looks like. More often, we don’t see the same information in each listings. We tend to look for acreage, and it’s amazing how many listings leave off the size of the property, or leave off the garage spaces. It’s usual in Colorado to have 3 spaces in newer homes and it seems that unless the house has more, the count is often left off, even when there are only two spaces.

    My Mom is in real estate and I’ve been around agents my whole life, so I’ve seen the issues first hand. Each agent writes up their own listing, using forms that are preprinted. Most agents seem to still work on paper, which means transcribing the data into the computer. Which means the chance for lots of mistakes as the human factor is involved. I don’t think it’s all laziness or lack of effort, but we all just make mistakes. And the more of us involved, the more chances it will occur. Just think of how many times you mistype something in email.

    I think this business could get a lot better, especially with more standardization and more IT tools in use. Using PDA-type applications or Java powered phone apps that could take listings in real time where the owner could double check things and ensure all the data is accurate. After all, better data quality will likely help the owner more than anyone.

    This isn’t limited to real estate, as many industries and processes suffer from data quality problems. There isn’t much that DBAs and other data professionals can do about the quality of the data we get; we can only ensure that the quality and integrity remains the same.

    But we sure get the blame when it’s not accurate.

    Steve Jones

    (published at http://www.sqlservercentral.com/articles/Editorial/72364/)