Voice of the DBA

Tag: sql server

  • T-SQL Tuesday #10 – Indexes

    TSQL2sDay150x150[1] It’s time for T-SQL Tuesday again, the brainchild of Adam Machanic (Blog|Twitter), and this month’s topic is Indexes. A thanks to Michael Swart for hosting this month’s event, and sending me a personal invite through email. That was pretty cool, especially since I don’t necessarily remember when the event is kicking off.

    The invitation is here, and time is running out to participate, but if you want to blog, you can do it today before this script (from Michael) tells you that you can’t

    IF GETUTCDATE() BETWEEN '20100914' AND '20100915'
    SELECT 'You Can Post'
    ELSE
    SELECT     'Not Time To Post'

    Remembering to Index

    Indexing seems to be one of those things that so many people forget to do. As a result, adding a few indexes is often the easiest win, the lowest hanging fruit, the most effective way of improving performance in many situations.

    I know in the past that I almost always create a single index on a new table, the PK, when I build it. If there are FKs, those are setup as well, and this usually helps, but when building a database, I sometimes forget to add other indexes since I don’t know what other queries I might be running when I start. And as I start to build those queries (to support new functions for the user), it’s easy to forget to add an index.

    sqldatagenerator[1] Since it’s hard to guess what you will query often, and the think/build/test/refactor model leads you down dead ends at times, I don’t typically build an index for a new query. I don’t notice performance issues since I’m usually working with test data that has dozens of rows, not hundreds. In the past, I’ve wished for something like Data Generator. Now that I have it, I typically don’t need it anymore.

    I try to go back and add in indexes as we get close to deployment, but I’ll admit that I typically have found in the production system that I’m missing indexes. This usually occurs over time, as data grows and performance decreases. I’ve wished that I had a better method for handling indexes, and apparently Microsoft was listening to wishes since we now have sys.dm_db_missing_index_details and other missing index DMVs.

    However I found someone that did have a solution. The company I worked for bought a third party piece of software to handle some specialized function for a department. I wasn’t involved, but one day they came to me with a performance issue. As I dug into the application, which had been working for months, I noticed something interesting.

    Every column was indexed.

    Not only was every column indexed, they had added other indexes that reversed columns in the index, so if I had a table like this one:

    CREATE TABLE Customers
    (
    CustomerID INT
    , FirstName VARCHAR(50)
    ,     LastName VARCHAR(50)
    ,     ADDRESS1 VARCHAR(50)
    ,     City VARCHAR(50)
    ,     StateID INT
    , Country VARCHAR(50)
    , Notes varchar(max)
    )

    I might have these indexes:

    • CI : CustomerID
    • NCI: FirstName
    • NCI: LastName
    • NCI: FirstName, Lastname
    • NCI: LastName, Firstname
    • NCI: Address1
    • NCI: City
    • NCI: StateID
    • NCI: Country
    • NCI: Notes
    • NCI: Address1, city, stateid, country

    It seemed like a little overkill, and after discussing this with the support engineer for awhile, he admitted that a number of these indexes weren’t needed, but the developer didn’t know what to index, so they indexed everything.

    We decided to run a trace on the application for a week, sort through queries, and come up with some more rational choices for indexes. We also removed a few of the duplicate indexes, and we improved performance for a number of functions.

    Indexing is important, but you can overdo it. Make sure that you index, but also think about what you index, and don’t index everything.

  • Common SQL Server Mistakes – Shrinking Databases

    I don’t like there being an easy command to shrink databases, and I especially don’t like seeing the shrink option as a part of the default maintenance plans.
    However it seems that this technique for managing sizes is used quite often, and even given as advice by some people. A few comments about this feature:
    First, don’t regularly shrink databases. Actually, don’t shrink databases at all if you don’t understand what it does. Paul Randal, who managed the storage engine team, wrote a blog about why not: Here’s a good reason not to run SHRINKDATABASE. The bottom line is that this fragments your indexes, which raises reads and decreases performance.
    If you are concerned about space usage, you have two choices: add less data or buy more space.
    SQL Server database files aren’t like a Word or Excel file. They don’t allocate space on disk as it’s needed. Well, they do if you have autogrow turned on, but really the files and server expect to have free space in the data files for data growth, change to data (and potential page splits/new extent allocations), and for maintenance.
    If you rebuild indexes regularly, and you ought to if they become fragmented, you need free space in your server. An index rebuild copies the entire index to a new, un-fragmented set of pages, and then drops the old index. So you need double your disk space for rebuilds.
    Managing space proactively is something you should do, and that means that you want to leave a pad inside your data files to allow for data growth. If you don’t have enough disk space, buy more. You need the space for data, and for performance.

    Transaction Log Files

    Now the transaction log files are a slightly different story. You still want to size them correctly, and some good reasons from Mr. Randal on this. You should set your log file size based on the frequency of your backups. The backups are scheduled based on your risk tolerance. Basically, more frequent backups, less transaction log space needed.
    However regularly shrinking your log files doesn’t introduce fragmentation, but it is dumb. Maybe not dumb, but it’s a waste of resources. Your server needs a t-log file size of xx to handle the regular activity on your server. Shrinking it at night and having it grow the next day to handle load is silly. And a waste of disk writes.
    Set your log file, manage it as needed, don’t shrink it.

    When to Shrink

    So should you never shrink? No, you can shrink, but the feature there is for emergencies or one-time events. If I get a load of 500GB on my 1TB data once a year, I might get crazy log growth. I might plan for that by expanding my log in advance, and then shrinking the log afterward, back to the size that I normally use.
    The same thing could occur in a database. Perhaps you move some data to a read only db and want to get the space down to data + largest index. Then you can shrink, rebuild indexes, and leave the log there. You can’t shrink to just data without fragmenting, so don’t try.
    When you shrink, use SHRINKFILE, and target specific files, for a specific reason. Not as part of regular maintenance.

  • Searching for Plans

    I was re-reading Conor Cunningham’s “Does Join Order Matter?” recently and found this quote in the post that stood out. As an aside, read this post. It’s fantastic.

    “…SQL Server’s QP is actually not set up to give you the best plan.  Instead, it is set up to give you a good enough plan quickly (where good enough is very close to “best”).”

    I get that, and it makes sense. Why spend 4 hours optimizing a query that runs in 2 minutes to get it down to 1:50 (Conor’s example) when the optimizer can spend 1 sec and have it execute in those two minutes. If the optimizer looked for the best plan, I’d be very angry as a customer. Especially if this happened with any frequency.

    Or would I?

    I’m not suggesting that we ought to optimize every query, or that I’d prefer the optimizer didn’t try to normalize queries, or that I want to see the internal tree and try to figure out if I can rewrite the query. I might, however, want to do that for some queries. There are queries that I might want to find the “best” plan for, either because they take a long time to run, or they are run often. I could then use that in something like a USE PLAN.

    Computers are cheap these days, as is disk. If I were to somehow able to copy my database onto an instance of SQL Server on a cheap PC, wouldn’t it be cool if I could set that optimizer loose on my query and let it try ALL possible plans? I’d be happy if it spent that 4 hours and gave me back an optimal plan that I could carry back over to my expensive, busy server, and shave that 10sec off a query that I run 10,000 times a day.

    I could see some value in a “test” instance of SQL Server that just looked for optimal plans. Now if I could just get Microsoft to build it.

    Steve Jones

  • Instrumentation

    In a discussion recently someone mentioned that they built in timing mechanisms into their application so they were aware of how long certain modules or functions were executing. Using this data, they could easily determine if the system was performing poorly by comparing the average of timings to the current performance.

    I thought that was a great idea, and it shows some proactive, forward thinking on the part of the developer. I haven’t often seen this in SQL Server, though it does seem that many good DBAs maintain some type of overall baseline for their database servers in order to help them response to performance issues.

    For a Friday poll, I was curious how many of you actually think about monitoring your systems. I decided to ask this in a general way to get an idea of what others might be doing inside SQL Server as well as in the applications that connect to SQL Server.

    Do you build instrumentation into your applications and code to measure the performance?

    I’m wondering if you actually collect, log, and even analyze data on how long processes take, or how much data you work with. Do you capture row counts from data loads or processing? Do you maintain an average time for code to execute?

    I haven’t done a lot of this in code. There was one application which watched for files and loaded them every day. We built code into SQL Server that tracked how long the load took, and how many rows were processed, mostly as a way to prepare for future hardware upgrades. As a production DBA, I have typically kept an average on size and time of backups for the same reason. It helps me capacity plan, and alerts me to growing loads for the server.

    If you do anything differently, or if you’ve found this to be valuable, or even a waste of time, let us know this Friday.

    Steve Jones