Voice of the DBA

Tag: software development

  • It’s the Engineers

    There are all sorts of articles and blogs on the Internet that try and teach you how to use a particular technique or technology to solve a problem. Plenty of this information is well written, and showcases the knowledge of that individual (or team) and how they’ve built a fantastic system. Every vendor building software platforms has reference case studies that show how well their system has worked for some customer. These are all good models that might give you confidence why SQL Server or MongoDB or Entity Framework or GoLang are good choices for your company.

    In reality, it’s not as simple as just choosing a platform or framework or language to make your application perform better. You really need the staff that understands how to use your choice of X and has some skill in building a system in that manner. It’s why I think that the cost of the software is a pittance compared to the cost of your people. If you have to train them, or they need to learn how technology X works, then you’re going to pay more in salary than you’d save in software licenses.

    There’s an article that I think illustrates this well, and might be worth passing along to your developers. It’s called Why Amazon DynamoDB isn’t for everyone and it’s worth a few minutes of your time to read. The gist of the article is that a NoSQL system, like DynamoDB, isn’t as simple and easy as you expect. It also says that for many applications, a relational database is a better choice because it’s often better understood and will solve most problems at small scale. For most of us, we don’t really get past what I’d consider small to medium scale, and so we should stick with relational systems.

    This isn’t to say that DynamoDB will be a problem for you, but there will be a learning curve for your developers if they haven’t used it in production and at your scale. The same thing occurs if a development team decided to switch from MongoDB to SQL Server because they like automatic tuning. They won’t necessarily configure and use SQL Server efficiently, and might not have their application perform well. Switching your development paradigm or technology is hard, and it will take time for your staff to get up to speed.

    Ultimately it really comes down to having good engineers that can write good code and understands the ins and outs of administering that particular platform. The database is extremely important here as it is the central location of your information, but the same arguments apply to frameworks and languages. As much as we want to learn and try new technologies and incorporate them into our work, we need to realize that it takes time to learn to use them well. Make small experiments, try POCs and slowly build up skills before you decide that your mission critical system needs some new technology that you read about and are excited to try working with.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.1MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • A Christmas Bug

    It’s not really a big deal for me, despite the fact that I travel to Europe in a few days for two weeks. At least, I’m fairly confident this won’t be an issue for me. A few of my fellow #sqlfamily might not be so lucky, as they use a different airline.

    Apparently American Airlines has a bug in their scheduling system. It’s not a safety bug, don’t worry about that, but it’s potentially a huge bug for the company in terms of profit and cost. The bug is a part of the system that lets pilots request holiday time off. The system allowed too many pilots to take vacation, and as a result, American Airlines (AA) doesn’t have enough pilots to fly their planes.

    In case you wonder about the scope of this, it’s 15,000+ flights that are affected. AA flies about 6,700 flights a day and are the world’s largest airline by some measure. They are offering pilots 150% of their normal rate, which is apparently the max allowed by contract. I’m sure some pilots will take the offer, but I’m also sure there may be some flight cancellations. After all, pilots have likely made their own plans, which they may not want to change. Apparently some have as the news as of this writing is a few hundred flights.

    The complexity of airline scheduling software has to be quite high, and while this won’t really endanger lives, it will affect the profitability of AA. This is definitely a development mistake, and one that should have been caught in testing. I don’t know how often this software is updated, nor what their process is. Someone suggested this is outsourced software, but that doesn’t matter. We clearly have a software development failure here.

    I certainly think developers are responsible for testing their code and meeting requirements. This includes not assuming the happy path is followed. However, this is also a place where we need (as an industry) to learn to better test the bounds and limits of our software. We need to be sure that we account for not only security, but pushing the limits of use by legitimate users. Software mistakes cost money, and as we continue to use computing more and more, we need to become better at testing our code. Our systems are very complex these days, more complex than a single person can handle. We need better testing everywhere, including the database.

    Please, make an effort to test better next year, in both your application and database. Use larger data sets at some point (both numbers and dates), add automated testing as a part of your development routine, and assume that users will do stupid, silly, and malicious things with inputs.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.4MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Avoiding Development Taxes

    When I first worked in an environment where multiple developers needed to release software, I found too many rules and constraints for smooth development. The process was developed by one person, or for one application, and somehow expanded to encompass all work being done in the organization. It was if we weren’t trying to think about our creative work as developers and DBAs, and instead viewed our systems composed of widgets built in a factory, each one a copy of the others, perhaps just a bit bigger or smaller. We used the same set of rules for the mission critical finance program as we did for the department vacation scheduling application.

    While working inside an overreaching command-and-control mentality, the idea of moving faster or releasing software more often was seen as really dangerous. Two decades of Agile methodologies, and the rise of DevOps have started to change this for many companies. Each year the State of DevOps report seems to show more and more organizations finding ways to build better software, often by having their operations people embrace the concepts and processes used by software developers as a way of building and managing their environments.

    Databases still lag behind, and as Donovan Brown says in this piece, if you’re not automating the back-end work of code deployments to the database, you’re still “faking” some of your DevOps process. Databases must maintain the state of data, even as transactions take place around code changes. Ensuring that we properly handle those data changes is a challenge. However, by treating the database changes as similarly as possible to application changes, we can minimize risk and learn database development techniques that help us push forward without being reckless.

    Tools have gotten better for database development. Microsoft has some tools for SQL Server, and various other vendors such as Redgate have others. Good tools are essential, but each new one adds some complexity to the environment. That’s one reason why having a minimal number of tools and platforms reduces the friction of getting things done. I see this as a big reason SQL Server on Linux might take off. Plenty of companies don’t want Windows in their infrastructure when most of their systems are on Linux. Not that Windows is hard, but consistency makes everything easier for a staff.

    This is the same reason why I’d say that it’s worth sticking with a database platform or two and not experimenting with each new type of system that comes out. If your staff knows SQL Server, then adding in MongoDB or Cassandra means there’s a learning curve, or a “tax” as the piece notes, to getting things done. This same tax gets paid with each new vendor, platform, language, or technology you take on.

    Database work is hard, and once your staff gets good at building and deploying changes, you want to take advantage of their knowledge. For companies that have solid development and deployment practices, stick with the things that work well. However, if you have staff that turns over regularly and don’t have a mature process, you can reduce the “tax” you pay for database development. There are companies that have paid some of that integration tax, that learning effort, and they’ve got tooling to help you build code better and faster. Just as I wouldn’t want to code my own build server, I don’t want to code my own database deployment tools. I could, but I’d rather spend my time solving problems and use the tooling that someone else has built to make my job easier, and provide a consistent coding experience for my developers.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 5MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Don’t Use Code Coverage

    I got asked in a seminar recently how to perform code coverage for T-SQL. There aren’t a lot of options, but there is a project from Ed Elliot called SQLCover that looks to examine how much of your code gets called from a test. There may be other methods, but my question back was why does code coverage matter? There were a few reasons given by various people, but ultimately the original questioner said that their boss wanted a report.

    That’s fine, and it’s a valid reason to try and measure code coverage, but is that an effective use of developer time? I don’t think so, especially as trying to meet some code coverage goal is fraught with all sorts of issues. Apart from various good points on Stack Overflow, no shortage of people have blogged about the problems of relying heavily on a code coverage metric. I won’t repeat all the arguments, but I’ll give a few of my own.

    First, someone pointed out that you want to be sure all your code is tested. I disagree with that, especially in many  long dev cycle applications. There are all sorts of enhancements I’ve seen in many applications that never get used. While you should certainly check them for security issues, if the features aren’t being used, remove them. Since most of you don’t implement tests for security issues, I’m not sure there’s value in telling you to write a unit test for logic or functionality.

    The other issue with code coverage is that anyone other than a developer using this metric will put pressure on the developers to write more tests. Some developers may write good tests, but I wouldn’t be surprised to have tests written that always pass and aren’t checked for correctness. Tests like these are misleading and potentially create more bugs since anyone refactoring code might think their changes haven’t caused an issue if the test passes. Bad tests, IMHO, are worse than no tests.

    I do think that code coverage can be valuable. If a developer is looking to ensure that some part of their code is being checked, especially in long stored procedures, then code coverage can guide them to a place that might need  testing. Since most code needs more than one test, covering more than one case, code coverage doesn’t help. I’d prefer a developer spent more time thinking about how to build better tests (or experimenting) than looking at coverage reports.

    And if you need a code coverage report, try my technique for building one

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.8MB) podcast or subscribe to the feed at iTunes and Libsyn.