Voice of the DBA

Tag: Performance

  • When Do We Worry About Scalability?

    “We need to get the code written for feature X. Can you finish this query today?”

    We’ve all heard some variation of that request. We have a request or demand, and we need to get it done. We need to get code out so our business can advance, sell more things, get more customers, etc. There’s always some reason to get new code pushed to production quickly.

    However, many technical people want to ensure their code works well. At least, I believe most do. While most people can write code that works and meets the requirement, some don’t know how to write code that performs well or don’t know how to test their code to check. Often there isn’t a large workload in dev or test environments to verify things.

    There may not be a large workload in production either, at least not at first.

    So, what do you worry about first: your code being used or performing well? That’s a similar question to this one: Worry about Scalability or Popularity First? While most of us don’t work for a startup and our organizations have some sort of financial stability, does popularity matter?

    I’d say that for any feature you build, whether a startup mobile app or a legacy ERP system, you’re still looking at this type of question. You want to know if it’s used, and how often. That might determine if you spend more time on this feature or area. Maybe you have some idea of popularity, or just plan old use of the feature. In that case, certainly make sure it will scale to not only meet your data size now, but plan for some level of growth across the next 6-12 months.

    If it’s a new area of functionality for your application, then maybe you have no idea. In that case, the DevOps approach is get something working, a minimally viable version of your code or query, and then tune it later if it becomes a problem. Many technical people approach the endless number of tickets and requests they get like this.

    The problem is management often doesn’t budget in time to clean up the technical debt (to care about scalability).

    My view for database code is that we should always be leveling up our database code knowledge. If we deploy bad code in production, and can’t fix it, then at least we can avoid adding to the problem by writing the same poorly performing code again. Learn a better way to write that type of query. Whether you’re splitting strings, finding islands and gaps, calculating running totals, or anything else. Learn what works well and write that code next time.

    That helps your team balance the scalability and popularity-chase by producing good code the first time. Or at least, the next time.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Invisible Downtime

    This article has a concept I’ve never heard about: invisible downtime. This is the idea that there are problems in your application that the customer sees. Your servers are running, but the application doesn’t work correctly or is pausing with a delay that impacts customers. From an IT perspective, the SLA is being met and there aren’t any problems. From a customer viewpoint, they’re ready to start looking at a competitor’s offering.

    Lots of developers and operations people know there are issues in our systems. We know networks go down or connectivity to some service is delayed. We also know the database gets slow, or at least, slower than we’d like. We know there are poor-performing code and under-sized hardware, running with storage that doesn’t produce as many IOPs as our workload demands. We would also like time to fix these issues, but often we aren’t given any resources.

    The current buzzword among executives and senior IT leaders is observability. It’s the goal of looking at how our entire system, application, database, and network, are linked and performing with an eye on improving performance. Not because they want to spend time or money here, but because customers are becoming more fickle and quick to move to another offering. Leaders know that degraded application performance (another phrase for invisible downtime) can have real bottom-line impacts on revenue.

    There are a lot of products in this space, application performance monitoring (APM), designed to look at lines of code and determine how well each is performing. They can help you spot issues in application code, but they lack insight into database and network details, at least at a level that the experts need. As a result, digging into performance issues and root cause analysis of problems usually means pulling data from multiple sources and correlating log entries.

    This is likely an area where AI/ML technologies can help, especially across large estates, though I think in many cases, what we need is just a pointer to poor-performing code. C#, Java, SQL, whatever. We need to know where the bad code is and then we need to train developers to write more efficient code. That might be the best way to improve application and database performance.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Be Careful with Missing Index Requests

    One of the things that has been interesting to watch over time is how the SQL Server platform has expanded the amount of information that we get back about the performance of the query optimizer and query processor. While it’s not perfect, and there is room for improvement, the advances made with intelligent query processing are helping many systems run faster. Not all queries, but some.

    As I’ve done a little work on other platforms, there are ways to look for potential missing indexes in PostgreSQL and MySQL, but these aren’t built into tools, nor are they easily accessible to developers or DBAs. There’s work to be done on many platforms, though I’m not sure if there is more work than required in SQL Server. On all these platforms, you need to dig into queries and understand why they are slow, though the tooling for SQL Server, with graphical plans in SSMS (or with Plan Explorer) can make the job easier.

    One thing SQL Server does is provide missing index recommendations in the query plan. You can find information on this in the docs, but you should make sure you read the limitations section. The recommendations returned should not just be run. I should repeat that for junior DBAs, accidental DBAs, and developers:

    DO NOT JUST RUN THE MISSING INDEX RECOMMENDATIONS WITHOUT TESTING LOTS OF QUERIES.

    I hate using all caps, but that is important. As an example of why, watch this short video from Erik Darling, where he shows that the simplistic view of the missing index is helpful, but not as helpful as it can be. In case that’s not enough, there are other issues that Brent Ozar, Phil Factor, and Aaron Bertrand share some of the problems they’ve found.

    There is a wealth of information that is available about queries in SQL Server and how they are processed. It will help you in your career to learn more about performance tuning and how to evaluate queries. We have articles here, there are more on Simple Talk, and Erik Darling produces information every week and also has training to help you learn to tune queries better. There are plenty of others that will help teach you as well.

    Maybe the best benefit of learning about tuning is that you can learn to write better queries the first time, which means no rework, no effort responding to complaints, and a cheaper bill if you move into the cloud. That might be something you point out to your boss and ask him or her to fund a little education to help you and your employer.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

  • The SPU

    Ever since I heard about the SQLOS and all the work that went into producing an operating system for SQL Server inside of the host OS, I’ve thought that perhaps there might be value in specializing the environment for a database platform. Given the importance of databases, especially relational ones, and the need to get every bit of performance out of hardware, I wondered if we wouldn’t see a version of SQL Server that is installed directly on the hardware, without a host OS.

    In some sense, I guess that’s what a PaaS database is in Azure, but I thought we might see that for the download-and-install version. I doubt that will ever happen now with the growth of the cloud.

    Instead, maybe there’s a better way to improve performance without requiring SQL Server (or Oracle, PostgreSQL, etc.) to implement some of the OS features they’d need. Perhaps we could change the hardware around and use a SQL Processing Unit (SPU). The SPU is another specialized chip. Like a GPU for graphics, or even some of the other chips that are made specifically for mobiles, storage, or networking, this one would be focused on database needs. Apparently, there are a few companies that are researching how they might build chips that focus specifically on the types of computations that data analysis requires.

    I don’t know how practical this is, or whether we’d see any major database vendor attempt to port to a new chip. Perhaps they’d add instructions that could use one of these chips inside a regular system, similar to how a GPU can offload work. That might seriously improve performance, something that a lot of our customers would appreciate. Especially if this happens without spending a lot of developer time rewriting old code.

    That’s not to say that developers shouldn’t learn to write better code. They ought to, mostly so that their initial attempt at producing reports or batches of data handles a wide variety of data values and workloads without stressing the hardware. More efficient code is always the best way to attack any problem. Even if you had a very efficient SPU, chances are that your workload will still grow to overwhelm the hardware at some point, especially as the number of users grows.

    Monitoring helps find problem queries, but it’s up to developers to change their habits, grow their knowledge, and produce better code sooner. Then we can use hardware to deal with the large numbers of users that need to access the system.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.