Voice of the DBA

Tag: software development

  • What is Continuous Integration for Databases?

    I have a talk that I give on Automated Build and Test for databases. This was put together by one of the engineers at Red Gate Software, and I’ve presented it at a number of events. I enjoy it, and it’s one of those presentations that really gets people thinking. I enjoy getting people to think about software, and there are always good questions that come out of the session.

    The talk is essentially about how to set up continuous integration for databases. Continuous Integration is an extension of the daily build, which companies have been using for years to try and ensure some discipline in their software development practices. They want to limit the amount of code that can potentially break software.  The whole idea is to test changes developers commit to version control as quickly as possible and give feedback developers immediately so developers can fix issues while the issue is fresh in their minds.

    For databases, this can be a challenge. We have this stuff called data that has to be maintained between, during, and after builds. We can’t drop our structures and reload them as a developer does with an executable or DLL. As we work in a CI environment, we need to handle new objects, as well as alterations to existing structures. That means our scripts and code need to handle creation as well as upgrade scripts.

    This means that a CI process for databases must not only check out the DDL code from a Version Control System (VCS), but it must also handle the CREATE or ALTER code based on the state of the database. You can always build a database from scratch in your CI process, or you can restore a known version (usually the last version) and apply upgrade scripts. The way this happens depends on how you version your DDL code in a VCS and how your build process is structured.

    At Red Gate, we’ve built a number of scripts to help with this process for various CI servers. We have articles that discuss CI for Team City, Jenkins, Bamboo, and probably more by the time this piece appears. We’ve learned a lot about CI over the last few years and we are trying to share both our knowledge and code with you, as well as incorporate those ideas into our products.

    Whether you use our products or not, we do believe that you will build better software if you implement CI. We just know that we can make the process a lot smoother with our tools for developers and DBAs.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( MB) podcast or subscribe to the feed at iTunes and Mevio . feed

    The Voice of the DBA podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.

  • T-SQL Tuesday #51 – Place Your Bets

    tsqltuesdayIt’s T-SQL Tuesday time again and Jason Brimhall is hosting this month’s event. He used to live in Las Vegas, and with that theme in mind, he calls for you to Place Your Bets.  That’s a time when someone has gambled, or risked something on an application, venture, process that relates to SQL Server? When have you done something that caused an issue you could have avoided?

    It’s an interesting topic, and I’m looking forward to reading what people write.

    T-SQL Tuesday is a monthly event, the second Tuesday of the month, started by Adam Machanic (blog|twitter). You can participate by setting up a blog and writing on the month’s topic. Check Twitter for the #tsql2sday hash tag or look for the blog postings with a search.

    If you want to host, contact Adam.

    The Big Bet

    I didn’t place this bet, but a bunch of application designers and management did. I came in to the company late, and realized we had issues, but wasn’t able to effect change in any short order.

    I worked for a large software company. Large as in thousands of employees, and certainly hundreds of developers. Most were engaged in writing the software we sold, but there were plenty of people engaged in managing our internal systems and writing software that we were using internally. One of these was our internal Intranet site for employees.

    This was a large project, replacing a number of applications, and designed to be a central point of information where employees could turn and various departments could publish information. Being a software company, we knew that it’s worth paying for software that’s written and we chose a framework that would cover many of our needs, but allow us to customize parts of the site for use by different departments. All in all, a CMS-type system that would fit our needs.

    The project started before I did, and no one consulted me or my group, as the production DBAs, as to the architecture, hardware spec, or anything else. It’s not that we should have, or would have, led the way, but we could have offered opinions about how things might perform.

    The day of the roll out came. I was told which day it was and never participated in any testing. The global policy was changed to set everyone’s default browser to the intranet, and I came into work expecting to find a new site where I’d see information posted, resulting in (I hoped) less emails from various departments. Not long after I arrived, I was pulled out of a meeting because of performance problems.

    I had no idea what to look at on the site, but we found CPU pegged and very high I/O on the server. I decided to run a short Profiler trace from my machine to get an idea of what the workload was and what we might look to run. I found fairly simple queries, lots of SELECT * to a few tables. What was interesting was that the queries were scrolling rather quickly on the screen. I found that the default trace values in SQL Server 2000 were storing around 25MB/sec worth of data on my local hard drive.

    In 2001, this was a fairly large load on a SQL Server 2000 instance. When we checked the tables and calculated result sets, we found very little data from each query. There were missing indexes, but even adding those to tables that were a few hundred rows of data didn’t help. The problem was simply that so many employees, each hitting the server by default, with a series of sidebars on the web pages, each of which created its own connection, ran its own query, and ran lots of them on each page, was simply overloading the hardware.

    If was bad design, poorly tested, and not well thought through. Even simple caching mechanisms as Brent Ozar has talked about would have dramatically removed a lot of the load. In the end, those weren’t really needed. What we found was that the majority of the information being returned by queries changed less than once a week. A few weeks of development time removed most of those queries and had them replaced with static XML files that were loaded by the application for display, and could be updated from the database by an administrator.

    Ultimately the site worked well, and we did get less emails from various departments since announcements could be made on the Intranet, which most of us learned to check once or twice a day.

  • Just use Version Control

    There are free systems out there. If you have no budget, and want to get started, download one of these:

    There are good ones to pay for as well, and some of them have other features and integration you may like:

    There are plenty more. Please, please, please, start using Version Control. There are compelling reasons from many successful developers, including those that want your database code under control.

    There are ways to do this, but please pick one. Version control is a must for professional software developers. That means if you write code, and get paid for it, use version control. If you write code, and don’t get paid for it, value your time and use VCS anyway.

  • Too Much Choice

    As this piece shows, more information can be bad. As drivers get more choices, options, and opportunities to interact with information, they may not be driving as well as they otherwise might. Restaurants and retailers have seen something similar. Too much information either prevents decisions or slows the process. I know when I go to a restaurant that has too large a menu, I often find myself taking a long time to decide on an item.

    I think this can certainly be a problem for our clients in technology, as we discuss specifications and requirements with them. If we allow clients too many choices, and too many options, it can be hard for them to make a decision, much less understand the implications of their choices. As software professionals, we need to give clients two or three options, but recommend one and explain in a fairly simple manner what the advantages and disadvantages of each solution are.

    Too much choice can also be a problem for us in technology. If we have too many architectural options, we can debate entirely too long. If we have too many requirements, we may be unable to get work done if we can’t focus or distill them all down. If we have too many concerns, we may not find an effective solution because we’re looking for a perfect one. The latter is a situation I find all too often with developers that want a perfect, 100% solution when a 90% one will work fine because the edge cases so rarely occur.

    The exceptions to this, both in my restaurant experience, and in research, are when the person already knows what they want, or they have some preconceived idea. That can be good for us in technology, as we can move quickly when we have some idea of what is needed. However this is a double edged sword as it can lead us to work with tunnel vision, not thinking about the alternatives or possibilities we may face.

    In general I think more information is better than less, but we can’t let the abundance paralyze us from moving forward and accomplishing work.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.5MB) podcast or subscribe to the feed at iTunes and Mevio . feed

    The Voice of the DBA podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.