Voice of the DBA

Category: Editorial

  • Performance Tuning Through Panic

    We’ve all been there at one time or another: panicked by a performance issue. I once upgraded a FoxPro for DOS application to a VFP front end with a SQL Server back end. We had a much more powerful server, generations ahead of the previous file server share. When the first users logged in and searches ran slower, I was in a panic. My team worried about the size of our inventory data in memory and quickly upgraded our 4GB, 1996-era server to 8GB of RAM, hoping for improvement. We also hurriedly split out queries and separated some data in an effort at quick tuning. It wasn’t much above a knee-jerk reaction, but we had users, and management, yelling at us on an hourly basis.

    I’m sure many of you have had other performance problems that resulted in some panic during your career. I’ve had more since then, and will probably have more in the future. Many of us do, and many of us perform knee-jerk performance tuning. However you can, and should, avoid tuning in a panic, as pointed out by Paul Randal. Doing so usually doesn’t fix the problem, and when it reoccurs, it’s more embarrassing with people less likely to listen to your next great idea.

    Tuning a system is a science, but it’s a complex science. Without experience, without having knowledge about all the complex interactions, and a deep understanding of how all the parts fit together, it often looks like a hopeless task. Who can keep all that information in their head and relate all the various components of the application? The network, the OS, the SAN, clients, all can cause problems that appear to be in the database.  As Paul points out, failing to understand the actual root cause can lead you to make changes that will have no effect on the actual underlying issue. You’ll often realize this when the problem re-occurs in a short time.

    I think one of the best reason to hire consultants is that they see so many problems that they can recognize patterns in performance that many of us don’t see. What they do appears to be an art because they can seemingly understand what information is relevant, and what isn’t. They walk down a list of potential issues, gathering evidence in a logical way, but one that can appear random to someone with much less knowledge and experience.

    All of us can learn more, we can read about problems others have had and learn how they solved the issue. However we often face the same reaction that many consultants do: no one wants to implement the advice. Until clients and vendor are willing to tune their code, many problems will only be masked by hardware.

    For a time. Then they’ll reappear.

    Steve Jones

     

  • Top Talent Leaves

    Derek Jeter will likely play his entire career with the New York Yankees. LeBron James has already left one team, and might leave another soon. Both are Hall of Fame players that have had extreme success in their chosen professions. They are recognized as some of the most talented players in their sports. Why did one stay and one leave? Probably for a variety of reasons, but it seems more and more that organizations are realizing there is some value in the retention of their players and are trying to find ways to improve their success at retention.

    In the business world outside of sports, we have much more freedom to change employers whenever we want. Most of us are glad we have that choice. Sometimes we move on when we don’t want to, but in the majority of cases I think technology workers leave companies because they aren’t happy about something.

    There was an article about why the top talent leaves a company and I think it presents a number of issues that many employees experience within their own organizations. People aren’t engaged, respected, valued, challenged, and more. In short, they really don’t like their jobs. Many of the issues are often easy to fix, but only if you could fix two of the major items mentioned (#7 and #10). Those are leadership issues, often systemic ones that occur when upper management doesn’t care about culture or doesn’t make an effort to hire and train good managers.

    I don’t know if I’d prefer to see people in technology working only a few jobs during their careers, spending decades at each, or if we are better workers with the exposure to differing environments and businesses. All I know is that I am glad that I’ve had the ability to leave bad jobs in the past, usually because of bad managers.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.5MB) podcast or subscribe to the feed at iTunes and Mevio . feed

    The Voice of the DBA podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.

  • The Cloud

    I like the cloud. It’s very handy for a lot of the things I do. I like having backups restored for my iPhone upgrade. I appreciate having my documents synced through Evernote, Dropbox, and OneDrive (formerly SkyDrive) automatically moving to new machines. These services are incredibly convenient and handy. I expect that my mail will be accessible from multiple machines and my blog posts can be created, edited, and published from anywhere. Many of us work in a similar fashion, using the cloud as a service construct in much of our lives. In fact, many of us expect our own knowledge will be supplemented by the cloud. We use Google to search, getting us syntax,  solutions, or ideas from sources that we expect to just be there.

    Someone was talking to me about their concerns with the cloud recently. This person thought the cloud would be a fad that passes quickly and businesses will get back to owning, hosting, and managing their servers from the bare metal up to the network connection. I disagreed, and not because I use the cloud personally.

    I remember setting up a web server in 1996 for a company. I remember installing an email server in 1999. I would never do either of those things again for my own company, and I wouldn’t expect it from most companies. These days we’ve learned that email can easily be handled by a third party and more and more companies are using email services instead of managing their own systems. Web servers are mostly a shared service that we rent from any number of other companies, sometimes even stitching together a presence across multiple providers. Even many companies that want to manage their own host operating system are turning to third party companies to manage the hardware and networking. More and more we accept renting the things we need from others.

    We’ll get there with databases as well. Not all databases, but more and more of them will be hosted at third party companies. Whether this is in some type of VM environment or a service that hosts data, it will still be hosted at a third party. We will have concerns and we will have reasons to not do this, but I think for more and more of our data, we won’t care.

    Companies have lots of systems in place. They have legacy investments in infrastructure and facilities. In those cases, it’s hard to justify the cost of cloud computing. However as we grow into new areas, or with new companies, we’ll have to re-evaluate whether it is a good idea to continue to make those investments. As the management of software grows to allow the quick scale up and down of assets, I think we’ll be much more likely to consider moving applications into the cloud than ever before.

    We’ll never be 100% in the cloud, or out of it, in the future, but increasingly, we’ll be partners with the cloud.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.4MB) podcast or subscribe to the feed at iTunes and Mevio . feed

    The Voice of the DBA podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.

  • Zombie Data

    Most of us that work with data are concerned about losing any of the bits or bytes we are responsible for. Many of us practice restores when we can, we complain about the low disaster recovery budgets we’re allowed, and we regularly check our systems for corruption. Even those of us that aren’t extremely diligent in our daily data protection are often worried about losing our jobs if a disaster causes data loss. Many of us don’t even trust our users, preferring to implement logical deletes in applications rather than physically running a DELETE statement.

    However there are times that we do want to remove data from our systems. We may find data quality is poor and want to erase the results of an ETL load. We may find ourselves bound by regulations that require the removal of data from our systems. We may upgrade old software and end up with copies of obsolete databases whose contents have been copied and reformatted by a new version of an application.

    This article talks about the data-pocolypse, and in somewhat of a jesting way, but it has a few good points that we may want to consider when we do need to remove data permanently. We should understand that deletes are not always deletes, and if a permanent solution is needed, we should use a utility to wipe the drive or physically destroy the hardware. However there are other places we should worry about old copies of data. Backups should be deleted from remote storage that might not have cleanup jobs running anymore if the servers are decommissioned. We should be wary about taking databases offline, or detaching them without physically removing the files. Development machines, laptops, etc. should have data removed if we are sure we don’t need it again.

    We should be especially careful about security access to servers that we are not using. Users might easily have links or pointers to old servers, and mistakenly connect. Make sure you remove access when you decommission the instance. Be careful about keeping old file exports, such as reports or feeds that were generated in the past. It might not be practical to wipe backup tapes, but be sure you’ve changed documentation and surfaced the information to all administrators that copes of databases restored from tapes after xxx date are not useable in DR situations.

    Of course the first thing you should do is make sure you have a good backup of the most current version of your data before you start deleting older copies. Delete this last, after you’re absolutely sure that users are no longer going to request a restore. I wouldn’t even ask users about this, however, because they may not be sure themselves. Instead I’d set a reminder a few months in the future to go back and delete this one, last, most current backup.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.3MB) podcast or subscribe to the feed at iTunes and Mevio . feed

    The Voice of the DBA podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.