Voice of the DBA

Category: Editorial

  • Bring Solutions

    This editorial was originally published on Feb 13, 2013. It is being re-run as Steve is out of the office.

    What’s your job? Most of you will answer “DBA”, or “developer”, or “manager”, or something that corresponds to your title. That’s partially true, but for the most part if you are in Information Technology, I think your job is to solve problems. You should ensure that your systems, or your code, or whatever you work on, is running smoothly for the people that use them.

    That isn’t always possible. There will be times that things break, that bugs crop up or systems fail. When that happens, people don’t want to hear about the inherent problems, or even what went wrong. They might ask, and they might be curious, but really what they want to know is when will the system be fixed and what will it cost.

    I’ve never had a manager that was really interested in the problems that caused a failure when systems were down. I’ve rarely had managers interested in potential problems I’ve noticed in our organization. Managers might be curious about the issues, but they don’t want you to bring them problems. They want you to bring them solutions when you see a problem, multiple choices to fix things, and a recommendation for the option you feel is best. They want a solution that works well, doesn’t cost a lot, and can be implemented quickly.

    Competition for jobs gets more intense all the time and the employees that look for problems and provide solutions without being asked will be the most valuable. They will stand out from their peers and are more likely to not only keep their jobs during down times, but will get asked to solve the interesting and challenging problems.

    Steve Jones

     

  • What’s Your Backup Speed?

    I ran across a thread recently where someone was looking to estimate the time it too to perform a 600GB backup. I’ve seen other threads with similar questions. The general advice is that you need to run a backup and see how long it takes. After all, your hardware, your workload, the compression settings, and more will affect your backup.

    In researching speeds, I didn’t find much info out there, so I thought I’d start a poll. This might be fun for you, so I’m hoping quite a few of you will answer.

    How long does it take you to back up 1TB?

    I know some of you don’t have a 1TB database, and probably a few of you have much larger ones. However, look at your backup and extrapolate (or interpolate) the time it would take to complete a 1TB backup. Assume that your hardware would scale appropriately to allow you to run a 1TB backup if you actually had that much data.

    If you can share your setup, or your configuration, that would be very interesting. I know some of you use striped backups, some compress, some use third party tools to help, and maybe you even have another way to actually get data from a live system into a backup file.

    Over the years I’ve seen some amazing improvements in the backup and restore functionality that SQL Server provides. I think it’s one of the most solid subsystems in SQL Server, and I’m looking forward to seeing just how fast some of you can create backup files.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.5MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Server Hardware or Coffee?

    I actually chuckled out loud in my office recently. I was reading a piece about Expensify and how they learned some lessons from their database architecture over time. It’s a good piece, but at one point the author talks about two Dell servers. There’s a dual core 2.8Ghz CPU with 500GB of storage for US$700. This could be upgraded to an 8 core, 3.7GHz CPU with 64GB of RAM and 10TB of storage for around US$3,200. However, the author says don’t be cheap. Why?

    That’s less than the company spends on coffee each month.

    I thought that line was funny, and it proves a point. It can be easy to think about the cost of hardware as being high because it’s a large ticket item for most of us. When you view it in the context of the scale of business, then it’s cheap. If any of us spent $3000 a month on food, we might not think $3000 for a server is expensive. When you look at the cost of your labor that might need to babysit an underpowered system, a few more thousand dollars seem like a bargain for memory or CPU resources.

    The author also notes that while EC2 charges for managing your systems, it’s still a steal, and for many of us, capacity isn’t a big issue. We do find that one single system runs most of our workload. If it doesn’t, then we could buy more, or a second system, for a relatively low cost. At least, compared to the rest of our business.

    There certainly are some of you that have a large workload, or you have spent substantially more on server hardware. I think many of my systems have been tens of thousands of dollars, but there are good points to be made. Today’s computers are very powerful, and even though we have more data, a single server is likely able to handle many workloads.

    If we write good code.

    That’s a big caveat, but training your staff to code better, having them spend time learning to query hierarchies more efficiently or quickly splitting strings, can pay off with much happier servers and customers. While I am a fan of using hardware to avoid spending too much time tuning queries, I also think continuously improving the skills of your development staff is much more important and might be the best IT investment you can make. Assuming, of course, that you treat them well and they enjoy working for your management staff.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.9MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Generating Our Own Bad Data

    I’ve worked in a few different industries and for a number of companies throughout my life. In almost every position, employees were given goals to improve the performance of the company. Often these were revenue-related goals that were supposed to drive additional sales. The one thing I learned when these goals directly related to sales was that there would always be unintended consequences that might skew business in a way that management didn’t expect. This is especially true when the rewards for meeting goals lead to direct compensation for employees. In fact, additional compensation has always led some percentage of the staff to work towards improving their own pay, even with fraudulent or unethical activity. Hopefully this has usually been a small percentage of people, but perhaps that’s not always the case.

    What does this have to do with data and databases? I recently read about fraudulent activity at Wells Fargo, where a sales policy caused a number of problems. Employees used real customer data to open fake accounts in order to meet sales quotas, generate fees, and earn bonuses. While certainly criminal and immoral, this also shows what I’ve believed for a long time. While computers allow work to take place very quickly, computers also allow mistakes to be made at incredible rates. While these weren’t mistakes, the computing systems allowed this fraudulent data activity to take place at a large scale. One that resulted in not only fraudulent charges to customers, but fraudulent reporting to investors, unearned bonuses paid out, likely unearned promotions, and who knows what else.

    To be fair, this type of criminal behavior could have happened before the age of computers, and likely could have lasted just as long with employees forging paper documents to split accounts, open credit lines, or any other activity. However, I’d hope that computer systems would have detected this sooner. Too detect the bad data, customers really need to be tied into some sort of reporting system that can audit their accounts, that can examine credit bureaus, and warn them if anyone is making unauthorized changes.

    We have seen similar unintended consequences when companies have offered bug bounties to developers. More bugs appear, and not surprisingly, more bugs are fixed, with more payouts ensuing. I’d like to think that most people would be driven to be professional, but time and time again we find enough of a percentage of employees aren’t, and the programs are doomed to failure. We need to do better, though I’m not sure what would help.

    Protecting data from insiders and outsiders is our responsibility as data professionals. However, insiders can cause more problems, which is why I would hope that the auditing and transparency of data platforms would be a priority for vendors across the next decade. In my mind, not enough attention has been paid to these areas, and our systems are woefully immature in these areas.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.4MB) podcast or subscribe to the feed at iTunes and Libsyn.