Tag: software development

  • When Should We Be Cleaning Data?

    I was reading Grant Fritchey’s update from the Business Analyst Conference  and noticed this quote: ” There were lots of sessions on the need to clean data, which implies that we need to do a better job at data collection in order to support this wild new world.” Grant was talking about the fact that many of the sessions seemed to imply that it’s mostly the processes around data that cause issues with data, not the people.

    However is that really what we should do? Do we want to do more and more cleansing at the point where people enter data? I’m not so sure that’s the case. The more that I deal with various applications and forms, the less I want to see too many fields and values required. Certainly there are pieces of data that are necessary for action, but I often find there are lots of additional fields that analysts want, but are more of a burden than a necessity.

    Most of us as data professionals design tables to handle the needs of an application. We put fields in tables that we expect to fill with data, based on a specification. However the real world is messy, and often the data we want to collect and store isn’t always available. The question is, do we force failures in the application or do we handle missing data?

    I don’t want to eliminate DRI, or go to an EAV models for lots of data. However I do think that we need to ensure we allow defaults and work with developers to allow data in our systems that might not be complete now, but perhaps will be in the future. We should work with analysts to help them build reports that can handle defaults or missing fields. We can work with developers to allow applications to request updates to data later and then design ETL that can efficiently fill in the updated information.

    Applications and software need to be flexible to work with problematic data. We have the ability, as data professionals, to help our clients still find meaning in data that might not be as complete as we’d prefer, or they’d prefer. However we can still find valuable information in what data they have..

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.5MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Your Job

    This quote is a great:  “I don’t really care if your code runs locally. If your code just runs locally, then my only option is to sell your computer so that our users can use our software, and that really doesn’t scale.” That quote is from an open letter to engineers and talks about what the job of a software engineer really is. It’s not just to write code, but to make sure our code and the entire system works for users.

    That’s important. It doesn’t matter how well you code if the application only works on your machine. We traditionally haven’t been great at getting code from one system to another and ensuring it works, at least not in a smooth fashion. We haven’t really followed an “engineering” process when we write, and deploy, software.

    The DevOps movement is changing this a big, bringing automation, scripting, and repeatability to both software development and operational deployment. All of these things should help us adhere to a more repeatable engineering practice. The commoditization of powerful hardware has also helped as many client environments might be nearly as powerful as development systems these days. It also means that our test systems can be good substitutes for production systems, at least if we can quickly build up and tear down large virtual environments.

    However we still need to be sure that we always remember that our code needs to work on other systems. Whether it has to scale down to meet the needs mobile devices or slightly older browsers and desktops, or our code needs scale up to handle thousands of times more transactions on a production database server than it processes on our laptops. We have to always consider the needs of the end user environment as we develop software. We need to test it in those situations, and broaden our skills to ensure that our applications satisfy the needs of the clients. And, of course, we should be able to quickly and easily deploy our changes to any other environment.

    Steve Jones

     

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.1MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Documenting with Tests

    Documenting code isn’t a task that many people enjoy. Those of you that have been through a computer science curriculum probably added lots of verbose, and obvious, comments to code. It’s probably unlikely that you ever found much value in the comments in a paying job.

    There are all sorts of ideas on how to document your software, writing good comments, and even refactoring code to remove unnecessary comments. I’m sure that if we put five developers in a room, we’d end up with eight different ways to comment code that would be debated, with no agreement on how to proceed.

    I ran across an interesting approach from Ed Elliot on documenting your code with unit tests. I hadn’t thought about the tests providing some documentation, but it’s an interesting idea. I’d have to work with the concept a bit, but I’m skeptical I’d get enough information from unit tests to ensure I understood what a stored procedure was doing, especially if I had 10 tests for a long procedure.

    I do think unit tests are important, and perhaps in conjunction with some type of code header that gives the requirements the procedure fulfills, I’ll get enough information to understand the code.

    I don’t have a universal solution that I think will work in most situations, but I do think that having a tool like intellisense or SQL Prompt helps you self document code with expressive names for columns, variables, and aliases. If nothing else, those phrases for variables can clue the next programmer in to what is happening better than single letter names.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.2MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Idempotent

    I ran into the word idempotent in the Stairway to Integration Services. I had heard the word, but I hadn’t really considered how important it can be for a DBA or developer until that time. It’s a term used in computer science, as well as other sciences, but I think it’s one that many of us don’t consider when we’re writing code, especially code used to deploy software to other systems.

    Most of us have written scripts like this:

    if exists (select object_id from sys.objects where name = 'uspGetSales')
      drop procedure uspGetSales;
    create procedure uspGetSales
    as
    ...

    At the end of running this code, we have the system in a state. If we run this over and over, we’ll regularly return to the same state, which is often exactly what we want to occur. That works great when deploying code, but what about this:

    insert into states select 'CO', 'Colorado';

    If I run that multiple times, what will happen? Either I’ll get errors if one of these fields is a PK, or I’ll get multiple inserts. If I’m running this as part of a software deployment, do I want either of those conditions? Do I want my end user to experience either one?

    No, I don’t. Certainly if I’m manually making changes to systems I can probably avoid issues, but that’s not what I want to do. Maybe you do, but I don’t. I’d like to be able to restart my deployment if something fails, without causing other issues. I’d like idempotent code, like this:

    if not exists (select abbrev from states where abbrev = 'CO')
      insert into states select 'CO', 'Colorado';

    That way if I happen to run this code twice, or ten times, I arrive in the same place.

    I realize that building scripts and deployment processes that are idempotent is a pain. It’s work, but it’s also scriptable and repeatable work that is easy to automate over time with a few patterns. I also realize that a little extra work to prevent issues is often an investment that’s worth making for both my customers, and my reputation.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.1MB) podcast or subscribe to the feed at iTunes and LibSyn. feed