Voice of the DBA

Tag: administration

  • Hooks

    I read this sentence recently, and it really caught my eye, mostly because I’ve rarely seen hooks being built into the software systems that I’ve written, or that have been deployed to my production systems.

    “A quick chat with your Operations team should convince you of the necessity to log every error condition to a single wellknown location, with the appropriate severity, so that they [the Operations team] know exactly what the problem is.”

    It seems obvious, but so few people I know have taken advantage of this. Far too many software products build their own logs and display errors on screens rather than alerting operations departments. So few people have taken advantage of RAISERROR to write to Windows logs, or the custom SQL Server performance counters to disclose information to operations systems or personnel.

    That’s my experience, but I wanted to ask you this week how you’ve handled things. Do you have software installed that does this? Is it a part of your development process to include hooks?

    Do you hook your logging into operations systems to alert others?

    As I think about it, I can’t imagine not doing this now. After all, I like to sleep. I don’t need calls at 2am from Production operator trying to find a log and debug a problem. I’d rather ensure that I log information to well known places and ensure exceptions will catch the eye of operations people. I want to give them as much information as I can in order to help them solve the problem. I’d rather sleep and depend on the on-call staff then be woken up because they rely on me to solve problems.

    Personally I think that logging to the Event logs, or including configuration switches to send alerts to other software ought to be part of the best practice of developing software. It’s a maturity point where we recognize that we’re building interconnected systems, not individual bits of software that live like hermits.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.6MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

    The Voice of the DBA podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.

  • Reverse Engineering Disasters

    If you are responsible for managing systems, you should have some sort of disaster recovery plan. Even if you are only managing the one system you carry with you on a regular basis, you should ask the question: What would it take to destroy this data?

    It’s a good question, and in the post I’ve linked above, a technologist talks about some of the failings of disaster recovery plans because they forward engineer plans to recover systems. People think of specific problems and try to prevent them. They don’t reverse engineer to find out what events would cause them to lose their system.

    That’s really the key to a lot of design and architecture in computer science. You can’t think about just what you expect to happen, or even what you want to happen. You have to consider the other ways in which events could occur or the issues that could cause an problem in your system. In development, the things you expect are considered to be the happy path. A good architect or engineer will think about everything else in addition to the happy path.

    I sometimes think that far too many decisions are made while considering the happy path, but ignoring or discounting the other paths available. These other paths may be unlikely to occur, but having worked with computer for over 25 years, I can tell you that I often find the least likely events occurring far too frequently.

    As the author says, you should never say “I never even thought of that happening.” Consider reverse engineering problematic situations and then make a judgment of how likely is it that any of the events will occur. That way you have at some idea of what could go wrong and what events you are willing to protect against.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.1MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

    The Voice of the DBA podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.

  • Production Subsets

    Continuous delivery recommends developers never use production data. It’s too big, too cumbersome, and slows the process too much. Developers should have enough data to determine if their solutions work as they build them. Testing should have enough to do some tuning, but unless you plan on full performance/load tests (which you should), then you don’t need the full set of production data.

    It’s an interesting idea, and overall I agree. A subset of data, hundreds of rows, can usually tell you if you’re writing code that works if you profile the code and look for inefficiencies. Note that profiling code doesn’t mean use Profiler. It means examining the resource used by your code in terms of CPU, I/O, memory, etc. There are tools to help you, and at some point in your development process, you should be using them.

    However it can be time consuming and cumbersome to build small development data sets. There are lots of choices in how you might do this, and I thought this would make an interesting poll. For those of you that deal with development, whether that’s T-SQL, .NET, or something else, what do you think?

    Should we have a subset of production data, a custom data set, or perhaps deal with complete production data?

    Some of this depends on the size of your production data, and I hope, it’s contents. I would not want any PII, PCI, medical, etc. in any development area. However if that’s not the case, then what do you prefer?

    Whether you have  custom data set or a subset of production, it can be cumbersome to keep this up to date. Your data may evolve over time and there’s overhead in maintaining some scripts that would produce the data you need. Perhaps that’s the cost of writing good software, but I’m curious how many of you feel.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.2MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

    The Voice of the DBA podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.

  • Artist or Scientist

    Which are you, an artist or a scientist? If you automate, you’re the latter. If you are a scientist, you can go on vacation. You can be more productive. People can count on you. You get things done quickly, consistently, and reliably. Everyone knows what to expect when you’re done with a task. They can expect things to be completed a certain way.

    If you manually run installation programs, click GUIs to configure options from memory, and customize each system you work on, you’re an artist. Artists build works of art, each of them unique. I know some incredibly talented artists working in technology, people who duplicate their work over and over extremely consistently. However at some point they’ll make a mistake, and then I’ll never know what state the system or code is in.

    While we need artists to push boundaries and experiment with new techniques, we don’t want them managing production systems or writing production code.  I want production code to use well known and proven techniques, best practices, good error handling, application of standards, logging and more. I want production systems to be stable, not with a lack of change, but with a lack of issues. I need scientists that produce work that can be counted on.

    Don’t build works of art. In development you must be an artist at times, but when you solve problems, ensure that the code contains best practices (secure coding and error handling among them), and make sure that your team understands and can reproduce the code later. In production, ensure you learn automation (PoSh, scripting, templates) and can build, or rebuild, your systems quickly and consistently. Deploy your builds to QA and development so that all the environments are the same.

    Become more of a scientist and not only will people depend on you, they’ll be less worried when you go on vacation because there will be fewer surprises for the person covering your work.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.5MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

    The Voice of the DBA podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.