Voice of the DBA

Tag: T-SQL Tuesday

  • T-SQL Tuesday #15 – Automation

    TSQL2sDay150x150It’s that time of the month again, and this month Pat Wright and his SQL Asylum are hosting the T-SQL Tuesday #15 party.

    If you want to learn more about the party, read Adam Machanic’s intro post (Blog|Twitter). If you’re interested in hosting from your blog, contact Adam.

    Automation is Crucial

    I have worked in a number of different sized environments in my career. From single server, 3 person companies, to 1 many-hundred instance, 12,000 person organization. Surprisingly in most cases, the size of the DBA team hasn’t varied tremendously. While it’s often been just me at smaller organizations, at the largest company we only had 3 DBAs to manage hundreds of SQL Server instances.

    How did we do it? You probably know: automation.

    In every environment I’ve worked, my goal has been to be insurance for when things break. That means that all the routine tasks need to be handle by someone or something else. Preferably by SQL Server itself since it’s usually more reliable at the tedious routine tasks.

    In the past, mostly SQL Server 2000 and before, I heavily used custom scripts, an administrative database on each instance, SQL Mail, and SQL Server Agent to perform regular maintenance activities, monitor jobs and processes, and summarize the results for the DBAs. My goal in all automation is to log the things that did happen, and only alert the DBA for items that need his or her attention.

    More Complex and More Capable

    When SQL Server 2008 was being developer, and new features were being introduced, I was pleased to see that a number of changes were made that I thought would help make automation easier. Policy Based Management, and the Powershell links to SMO objects, along with the new XEvents, Change Data Capture, the Data Collector, and more, there are many more hooks and ways to use automation to manage and monitor your SQL Server.

    However there also is a tremendous amount of complexity in using these tools. PBM sounds easy, but once you start creating policies, you can easily end up with hundreds of policies that result in an environment that is harder to understand.

    Watch the Watcher

    The ultimate goal in automation is to handle tasks in a more efficient way, using code and scheduling applications. This should result in less work for the DBA, but it also means that the DBA might count on certain tasks being completed, which might not get done.

    No matter what method you choose for implementing automation, you need to ensure you have some independent scripts and checks running to check on your automated tasks. An independent set of automation checks should be available to ensure your tasks run when scheduled, and they perform the tasks that are needed. While you might have a job to defragment indexes, you might have a second job that checks the fragmentation levels and independently notifies the DBA if the levels get too high.

    You also need to distill information down to the essence of what is important for the DBA as well. Too much routine information dulls the DBA’s senses, and makes it harder for them to find out what is broken in their environment. While all actions need to be logged to analysis and auditing, only those actions truly requiring the DBA’s attention should be raised on a regular basis.

    This sounds like very general advice, without a lot of detail, but the reality is that the balance to strike here will change with each environment. Usually I have had to spend months slowly tuning my automation and monitoring until I eliminate notification of the routine items, and only get notified of the critical “fires”.

    Which means that I don’t need a notification every day. The lack of a “look at this” note is hard for many DBAs, myself included, to get used to.

    The last piece of advice for implementing automation is this. Don’t try to account for every situation in your system. It results in too many code branches, and too complex an automated system. Handle the 80% of routine tasks and let the 20% anomalies fall through to the DBAs. It means a simpler system to build, and maintain.

  • T-SQL Tuesday #14–Resolutions

    tsql2sdayIt’s T-SQL Tuesday time again, and this month the Midnight DBAs (Jen’s blog | Sean’s blog | @MidnightDBA) are hosting the party. You can read about the whole T-SQL Tuesday blog party idea from Adam Machanic (blog, Twitter) if you want, and if you want to participate, you have a few hours left.

    Resolutions

    The theme this month is resolutions. That’s interesting because this was the first year in quite some time where I haven’t set goals or resolutions for the new year. I’m behind and somewhat unsure of what I want to do in life.

    However, it’s T-SQL Tuesday, so I need to think of something if I want to participate.

    I recently received an offer to take the MCM written test for SQL Server. There was no downside for me, so I signed up. And I pushed back the test a month since I wasn’t ready. I could say that my goal is to get the MCM this year, or to pass the written test, but that wouldn’t be right. I have reasons I’ll blog about later, but suffice it to say those aren’t resolutions.

    However I would like to make an honest effort to pass the written test. I haven’t had production responsibilities on a SQL Server instance in years, and while I know a lot about SQL Server, and understand how many things work, I don’t have to practically apply that knowledge, and so I’m rusty.

    I’d like to remedy some of that this year. I have been watching the MCM readiness videos, trying to learn some things and get prepared. Part of the reason I pushed back the test was because I hadn’t had a chance to watch all the videos and read the other prep material. I’m still working my way through things, and hope to be better prepared when the test comes around.

    My resolution is to really make an honest effort over the next month. Not cramming, and making this a single-focus-I-have-to-get-it-all-done, but make a good effort, my best effort given my busy life, and do as well as I can on the test.

    Kind of a nebulous resolution, but I can say that I didn’t make a solid effort in December, and I’m working to fix that now.

  • T-SQL Tuesday #13 Roundup

    TSQL2sDay150x150After my call for “What the Business Says Is Not What the Business Wants”, there were quite a few people that responded. Including my blog post, I found 43 comments posted on with links back to the thoughts.

    Here’s a quick summary, along with a short comment on each.

    Matt Whitfield wrote on issues with letting the customer decide how, in addition to what, a project should be built. Wise words, and often a place where trouble ensues when technical people take things literally. A great cartoon to start out the topic is in this post as well.

    Rob Farley talks collation issues. In this case, the collation issues are misunderstandings due to the different way we express ourselves in our particular industry or career. It’s a common problem, and it shows why the technical people need to make very, very sure they understand what the business person said, not what they meant.

    Mark Broadbent stresses that we need to provide alternative solutions when what is proposed won’t work, and talk in terms of value, something the “business” should understand.

    Roger Noble wants to take off the white coat and tells us to remember that communication is the key. On both sides.

    Aaron Bertrand has seen issues with outsourcing and requirements. This technique works well in some cases, but not others, and with development projects, it can be problematic.

    Reporting is the area where Jason Brimhall has found confusion in different departments wanting separate reports, but then somehow having them match for comparison purposes.

    Performance is always an issue, and Alexander Kuznetsov talks about the need to actually define carefully what types of performance, and what the metrics we use, will mean.

    I have horses, well, my wife has horses. I love my wife, so there are horses at the ranch. Noel McKinney asks what to do with the business wants a pony? Not literally a pony, but worth the ready anyway to understand what he means.

    Shrinking Databases is the scenario that Pinal Dave uses for his post. He describes a scenario where the business requested that the smallest file be used for DR, but a lack of understanding of how backups worked resulted in a regular shrink job.

    Nicholas Cain talks about this mysterious business, and has a few examples of the problems that can be caused when too much design input comes from non-technical people.

    Chris Shaw channels the Soup Nazi in a quick note about a former manager who wants the database locked down, to everyone but him.

    Jen McCown starts with a classic joke and gives a few examples of where the business has caused issues with technical systems.

    John Sansom, DBA from across the pond, has some advice for DBAs on how to better develop understanding with your business people as well as understand politics a little better yourself.

    Bradley Ball tells a story of how he has had to build trust with the business. A story similar to one I’ve had, and worth the read to remind you how to deal with mistakes.

    Joshua Feierman wonders how we can better build good software the meets the business’ needs, budget, and timelines. I can sympathize here, but also understand that the business wants things done, and not necessarily in the best way possible.

    Bob Pusateri describes the problems that come when the business has a little knowledge and tries to work too closely with technical people.

    Matt Velic reminds us to listen to the business and ask questions to be sure that the communication is understood by both sides. A good reminder that ultimately this is a job and you shouldn’t get too upset.

    Jeremiah Peschka tells us to listen in a not-so-subtle way. Listening, and understanding, what users say to you is important. We shouldn’t design the software we want, we should design the software users want. Not as easy as it sounds.

    Gethyn Ellis, one of SQLServerCentral regular bloggers, runs into the classic Full-Recovery-Mode-with-No-Transaction-Log-Backups.

    Mike Walsh tells us we need to listen. And tells us again. And again. Get the hint? We (IT people) don’t listen well.

    Erin Stellato uses quotes to remind us that teamwork and relationships are important.

    Kendra Little talks features. What people want, but what they then forget or don’t consider. This is a nice post to send to your business people, or perhaps paraphrase to try and help them understand your point of view.

    My good friend, Andy Leonard, talks ego, as a part of his series on interactions with business people. Check your ego. Good advice.

    Audrey Hammons has a top 5 list. These are the top five things the business has asked for. It’s a funny list.

    The very talented Wendy Pastrick from Chicago found some nice blog fodder with this month’s topic. Is your business like the blind men describing the elephant? If so, you might like Wendy’s post.

    Jason Bacani uses some great images to show why the business has trouble with us. They want everything yesterday and we want our procedures to get in the way.

    Richard Handloff tells us how to ask questions to get better information from the business.

    Meredith Ryan-Smith gives us a method that has worked well for groups in her cmpany.

    Cade Roux tells us that IT needs to be a strong partner with the rest of the business.

    John Welch has to convince the business of the value of data quality. I’ve fought this battle before as well and it’s a tough one.

    Amit Benjeree  writes his second T-SQL post and talks about the classic things you need: requirements and priorities.

    Allen Kinsel tells us to plan on changing. It’s a lesson I learned early, and not only do I plan for change, I design with the idea that my design will need to change for future requirements.

    Jes Borland learned about the business and learned just how valuable that knowledge can be.

    SQL Soldier, Robert Davis, warns us of the dangers when the DBA does not push back on business demands.

    Brian Kelley, longtime SQLServerCentral author and security expert, reminds us that technical people and business people start from different places. Neither knows what the other knows, and that can be a source of problems.

    Thomas Rushton left a job when the business didn’t appreciate the value of people. I’m sure he’s glad he’s not at that job anymore.

    Ryan Adams says that we need to be the interpreters. DBAs have to translate what is said into what is needed. Now if we just had a decent piece of software to help us, this would be easy. Ryan has a good example of a situation where things can get confused without someone acting as interpreter.

    Oscar Zamora talks about the importance of a strong business analyst in the software development process.

    Stacia Misner rounds out this months’ party with “a picture worth a thousand words” and gives us some insight into how she designs systems and works through projects. Rather than go left to right, she goes right to left. Read her post to understand more.

    This was a neat experience, but a lot of work. I’ll admit that I don’t always read that many T-SQL Tuesday posts. I tend to read 5-10 tops, and going through 40 and summarizing was hard.

    However I’m glad I did it, and I bet you will too. Contact Adam to host a month in 2011 and pick your topic.

  • T-SQL Tuesday #13– The Business and Disaster Recovery

    I’m hosting this month’s T-SQL Tuesday blog party, but that doesn’t mean I can’t participate. I posted the invitation last week, and hopefully we’ll have a large number of people participating today.

    The Cost of HADR

    I’ve dealt with a lot of business people over the years, and for a variety of reasons. At one job, I worked for a small company, literally about 50 people, and we had a number of systems critical to our business. We had a large fax server, receiving hundreds of faxes a day from overseas. We had a large imaging server/platform in which we stored our faxes along with quite a bit of other business correspondence that we needed a record of, and we had an inventory management/sales system running on SQL Server that people used every day to run the business.

    These were all important systems, and if they were down, business would slow to a crawl. The owner of the company wanted to take advantage of all our technology and mandated “no downtime” and “zero data loss” to my boss, the VP of sales for the company. He was, of course, happy to push this down to me as a goal for the next year.

    This was in 1997, and as I investigated our possibilities for achieving this, I realized very quickly that it wasn’t a good move for the company. Clustering in SQL Server v6.5 was shaky and hard to install and maintain. Marathon Technologies had a machine that was basically a “cluster-in-box” that would do a lot of what we wanted, but it would have been around a $30k server. Since our two SQL Servers cost about $7k together, this didn’t seem like a great solution to me.

    Instead I investigated what downtime cost us. What would it cost in terms of work and salary if we had to re-enter data that might be lost. We had been down for various periods of time due to different events, and the business had survived, even one day when we were shut completely down for 2 days.

    It quickly came to light that we’d have to be down for over a week to really cost the business more than $30k, and that was debatable. We could still conduct lots of work on paper and over the phone, so it didn’t make sense to aim for zero data loss and zero downtime. I explained it to my boss, laid out a simple case for him with a few numbers highlighted, and the provided an alternative for investment in a little redundancy, designed to minimize losses, but not prevent them.

    In the end the big boss agreed. He hadn’t really meant zero loss, after all, anyone in business knows that there are always going to be some losses. What he wanted was his business to keep running, cost effectively. That’s what we gave him at a much more reasonable price.