Author: way0utwest

  • The Danger of Custom Software

    The Movie Vanishes
    My kids enjoyed this DR tale from Pixar.

    There’s been a great little movie short making the rounds of the Internet from Pixar. It’s called “The Movie Vanishes” and it’s worth a few minutes of your time. Toy Story 2 was almost lost because of a mistake and some bad luck at Pixar.  This was at a time when the company was successful, and certainly should have been able to better prepare for a disaster. If you want a touch more background, there’s a few other notes at Quora.

    A lot of the software that Pixar uses was written in house. That’s a double edged sword because there isn’t anyone that can stand behind the software, other than the people that wrote it. There might not be adequate testing and there are certainly bugs in the software that may lie dormant for years. I have no idea of any of the bugs inside Pixar’s software caused this disaster, and I’m not implying it did.

    The positive side of building your own software is that you know how it works. You have the source code, and if you have a developer that can understand it, you can fix problems, patch issues, and customize it to suit your needs. As long as you have the time and resources to do so.

    I saw someone write recently that building their own monitoring solution for a set of SQL Servers was easy, but that was the smallest part of the job. Maintaining and enhancing it over time were much larger jobs than setting up monitoring. This person said they’d rather buy a package in the future than build their own again.

    If you have a system set up, it probably makes sense to use it, but as you look to develop new software, whether for monitoring servers or handling sales, it might be worth spending a bit of time trying to determine if there is something out there you can buy, which might be well tested, vouched for by other customers, and be easier to integrate than your own system.

    Steve Jones

    SQL Monitor from Red Gate SoftwareIf you don’t have monitoring set up, you should. SQL Monitor from Red Gate software is an easy way to get notified when something in your environment needs to be looked at further.

    If you want to get monitoring setup without minimal effort and immediately, think about downloading a trialof SQL Monitor and testing it with your servers.

    If you’d like to see SQL Monitor working on the live SQLServerCentral database server, go over to monitor.red-gate.com.


    The Voice of the DBA Podcasts

    We publish three versions of the podcast each day for you to enjoy.

  • T-SQL Tuesday #31 – Logging

    TSQL2sDay150x150It’s T-SQL Tuesday time again, and this month Aaron Nelson (blog | @sqlvariant) is hosting. The topic is logging, and if you’re like to participate, read Aaron’s post and learn the rules. We do this on the second Tuesday of every month.

    If you’d like to host, contact Adam Machanic. It’s easy to do. Get on the schedule, pick a topic, and then write a post.

    A list of previous posts is here,

    Logging

    I’ve found documentation of events to be one of the most important things I can do in my career. Finding out what happened, what changed, or what I did has been important many times, and often helped me come through difficult situations.

    Logging is the automated version of documentation. All kinds of applications, including SQL Server, produce logs of the various activity on the system. In SQL Server, we are moving to an eventing system, and if you haven’t looked at Extended Events, you should.

    One of the times when I found logging to be lacking was in a startup I worked at a decade ago. We had a number of developers that were working on various development servers. They had full rights, and they were allowed to build their own objects. That was a little concern to a controlling DBA like me, but I allowed it since they often wanted new objects quickly, and if I allowed them to write their own, they’d use stored procedures.

    A good compromise, if you ask me.

    However in the hectic pace of development, I found that the developers didn’t often keep good notes about what was being built for which features and functions. Since we had to produce a build script fairly quickly every Monday in order to update our QA systems, we would find that developers invariably would forget objects and we would not have a well tested QA script on Monday afternoon.

    I decided that we needed to better log the changes on our development server. I didn’t care about every change, especially intermediate changes to objects, but I did care about the gross changes made each day.

    This was in the SQL Server 2000 days, with limited tracking of changes outside of SQL Trace. Since I had no desire to move through lots of trace files, even in an automated fashion, I decided on a much simpler method.

    In sysobjects (now sys.objects), there was a crdate field, which tells you when the object was created. However that doesn’t change if an ALTER TABLE is run (or any other ALTER). That stumped me briefly, but I decided to search further.

    I found that there was a schema_ver field, which is incremented every time the object is changed. Since the majority of our developer changes were ALTERs, I could track the version number and then compare this each day. I tested this out, and it worked well.

    The outline of the solution is that I grabbed a copy of the sysobjects table every day and stored it in a temporary table. I then used a left join to compare this with the previous values stored in a table I’d created to store the data. When I found differences, I logged them in a table, along with the date, and sent myself an email. I would then overwrite the stored version of the objects with the version from the temp table, giving me a baseline for the next execution.

    At the end of the week, I’d have an aggregate list of all objects changed, which I could then compare against our build script.

    At the time we were in an agile environment, releasing new code every Wednesday, and operating on very short timelines. The logging I did cut down on mistakes and allowed us to have a smooth release process that functioned for over 18 months, with code releases nearly every Wednesday outside of holidays.

  • Slipping in a Break

    I know some people like long holidays, but I tend to prefer short ones. Today is one of those breaks for me. I’m coming back from Pensacola and SQL Saturday #132, taking the day off as I spent Sunday and the morning with my daughter.

    This was a combination of work on Fri night/Sat for the event, and then a mini-vacation on Fri morning, Sunday, and Monday morning for the two of us. A little Daddy/daughter time that we both enjoy and look forward to.

    Remember to get those breaks in when you can, and take advantage of the chance to spend time away from work if you are traveling and can fit it in.

  • 13 Disasters

    Hopefully you don’t have any server disasters because of coffee spilled on a machine.

    This editorial that was originally published on Aug 15, 2007. It is being rerun as Steve is traveling.

    This is an interesting list of disasters that can befall a production system. It comprises more than database servers, but certainly can happen to them as well.

    The list is pretty good and of these I’ve had more than a few of them happen to me. There were a few that I’ve never seen and it makes me worry a bit that this guy doesn’t have a good staff or set of vendors working with him. But one of these is very interesting.

    The second item on the list, a controller going bad and corrupting disks, is an interesting one in today’s world. What would you do if this happened on your SAN? Actually I know what you’d do. First you’d be in denial, and I don’t mean the river in Africa.

    Then you’d tell the SAN guys. They wouldn’t believe you. You’d argue and they’d check things, then they’d wonder how it could happen and why? You’d scream at them to fix it, probably using 4 letter, not 3 letter words. They’d start looking for ways to recover data and your hair would be slowly thinning as upper managers started calling down looking for answers.

    It shouldn’t happen, but it could. Some marketing VP is showing off that nice piece of SAN equipment with it’s high speed switches, dozens of drives, and lots of colored wires. He sloshes some coffee onto the system, freezes, but when the flashing red lights and siren from the Enterprise don’t go off (Red Alert!), he continues his tour while the controller scribbles on your disks.

    Restoring from backup might be easy. Of course if you have a system like some I’ve seen that shares physical disks among multiple LUNs, it might not.

    Disaster Recovery is rarely the hurricane Katrina type of issue. Usually it’s something like a disk drive, raid controller, cut wire, etc. that you have to deal with. So think about all of the minor disasters that you want to be sure you can handle and get some practice in on those. Make sure you have spare parts, you can rebuild a server (QA machines are handy for this), you know how to perform a restore.

    And most of all, be sure that you know where the backup files are stored. Preferably on different disks than the production data.

    Steve Jones