Tag: T-SQL Tuesday

  • T-SQL Tuesday #016 – Aggregation

    It’s that time of month again, time for Adam Machanic’s (Twitter | Blog) T-SQL Tuesday blog party. This month the party is hosted by Jes Borland (blog | @grrl_geek) and the topic is Aggregation.

    And I’m late.

    I planned on doing this yesterday, I really did, but I got busy and had to let it slide. A quick story, just to participate in the party.

    The Ordering Report

    Years ago I worked for a company that imported products from overseas. We imported wood products, and it was an interesting business. Mostly because the products we imported were not uniform. A sheet of 4’ x 8’ plywood, made of an Oak that is imported from South America, might not be the same as one imported from Africa. Even if it’s the same species, we had many customers that would only want one or the other.

    We even had customers that would want wood from one mill, but not another. Add to that the fact that your inventory is often sold in advance, while it’s being milled or shipped (literally on ships) from overseas. That meant that there were times when our any supply chain issues would cause our inventory to vary dramatically, with “available” dates sliding all over the place.

    Needless to say, it was a complicated process, but my boss wanted an ordering report to help the supply department decide how much of each product to order. It was a somewhat important process since the trees sometimes were not cut until a purchase order was sent from the US.

    I built the report, which used a rolling average of usage from the previous 3 months, along with factors from the previous few years to account for the seasonal nature of our customers and aggregated SUMs and AVGs across our product line. The report was deployed, and sure enough, I had my boss and the lead supply manager in my office complaining the next day.

    Apparently my report was “broken” since they both had run the same report, same parameters, and had gotten different numbers. I spent days checking the aggregations, sure that somewhere I had a calculation error that would come up with certain parameters. Eventually I realized that the constant inventory shifts were actually affecting the reporting. A simple explanation, but not a good reason for my boss.

    My solution? I decided that the daily inventory shifts, which were sometimes reversed by salesman just as quickly as they were made, weren’t helping matters. Instead I built a job that ran every night and pre-populated a table with all the aggregations needed for all products. Essentially simulating a simple cube, with limited measures for aggregation. The report ran against this table, and was the same for each day, all throughout the day, for all people that used it.

    After a few months of using the report, and comparing it’s recommendations to the previous guesses made by the supply department, they decided that it was good enough for them and the daily aggregation was close enough for them to use.

    Two added benefits to my overnight aggregation? First, the report would run quicker since all the aggregations were done overnight and not on the fly. Second? We could allow the people in the department to add an “adjustment” to the recommended order, raising or lowering the amount, and then store that value. Essentially making an interactive report that everyone in the department could see.

  • Wow, where did the month go?

    It’s T-SQL Tuesday again, and I’m not prepared (again). I missed all the tweets last week while I was at the MVP Summit, so now I need to try and find time today to knock out a post.

    It’s work, and effort, but I enjoy participating in this party, so I’ll be digging trough my mental inventory of stuff for this month later today.

  • T-SQL Tuesday Topics – Feb 2011

    I was looking for a static list of all the T-SQL Tuesday topics and couldn’t find one. The domain moves around to the current hosting blog, so when it was my turn to host it, I made a list just so I could reference what was out there.

    After a few more months, I decided to update my list with the latest topics. That will give me the chance to plan for my own future hosting of the party if Adam will have me again.

  • T-SQL Tuesday #15 – Automation

    TSQL2sDay150x150It’s that time of the month again, and this month Pat Wright and his SQL Asylum are hosting the T-SQL Tuesday #15 party.

    If you want to learn more about the party, read Adam Machanic’s intro post (Blog|Twitter). If you’re interested in hosting from your blog, contact Adam.

    Automation is Crucial

    I have worked in a number of different sized environments in my career. From single server, 3 person companies, to 1 many-hundred instance, 12,000 person organization. Surprisingly in most cases, the size of the DBA team hasn’t varied tremendously. While it’s often been just me at smaller organizations, at the largest company we only had 3 DBAs to manage hundreds of SQL Server instances.

    How did we do it? You probably know: automation.

    In every environment I’ve worked, my goal has been to be insurance for when things break. That means that all the routine tasks need to be handle by someone or something else. Preferably by SQL Server itself since it’s usually more reliable at the tedious routine tasks.

    In the past, mostly SQL Server 2000 and before, I heavily used custom scripts, an administrative database on each instance, SQL Mail, and SQL Server Agent to perform regular maintenance activities, monitor jobs and processes, and summarize the results for the DBAs. My goal in all automation is to log the things that did happen, and only alert the DBA for items that need his or her attention.

    More Complex and More Capable

    When SQL Server 2008 was being developer, and new features were being introduced, I was pleased to see that a number of changes were made that I thought would help make automation easier. Policy Based Management, and the Powershell links to SMO objects, along with the new XEvents, Change Data Capture, the Data Collector, and more, there are many more hooks and ways to use automation to manage and monitor your SQL Server.

    However there also is a tremendous amount of complexity in using these tools. PBM sounds easy, but once you start creating policies, you can easily end up with hundreds of policies that result in an environment that is harder to understand.

    Watch the Watcher

    The ultimate goal in automation is to handle tasks in a more efficient way, using code and scheduling applications. This should result in less work for the DBA, but it also means that the DBA might count on certain tasks being completed, which might not get done.

    No matter what method you choose for implementing automation, you need to ensure you have some independent scripts and checks running to check on your automated tasks. An independent set of automation checks should be available to ensure your tasks run when scheduled, and they perform the tasks that are needed. While you might have a job to defragment indexes, you might have a second job that checks the fragmentation levels and independently notifies the DBA if the levels get too high.

    You also need to distill information down to the essence of what is important for the DBA as well. Too much routine information dulls the DBA’s senses, and makes it harder for them to find out what is broken in their environment. While all actions need to be logged to analysis and auditing, only those actions truly requiring the DBA’s attention should be raised on a regular basis.

    This sounds like very general advice, without a lot of detail, but the reality is that the balance to strike here will change with each environment. Usually I have had to spend months slowly tuning my automation and monitoring until I eliminate notification of the routine items, and only get notified of the critical “fires”.

    Which means that I don’t need a notification every day. The lack of a “look at this” note is hard for many DBAs, myself included, to get used to.

    The last piece of advice for implementing automation is this. Don’t try to account for every situation in your system. It results in too many code branches, and too complex an automated system. Handle the 80% of routine tasks and let the 20% anomalies fall through to the DBAs. It means a simpler system to build, and maintain.