Voice of the DBA

Tag: monitoring

  • The Slow Growing Problems

    Both as a DBA and developer, I’ve had plenty of immediate, this-is-broken, fix-it-quickly issues. Usually, I, or someone else, wrote some bad code and somehow got it deployed. I mean, I do test things, and I would (probably) never change code after I’d tested it to fix that one little annoying thing, like the formatting. I’d (almost) never do that, and I’m sure you wouldn’t either.

    Yet somehow bugs slip in at times.

    Those are the acute issues, and they can be hard to fix at times, but often we can reproduce the problem in development and build a fix. Sometimes we even spot the issue quickly and just fix it in production. I’m sure you never do that, but I have had that experience myself a few times.

    However, in the database world, we can have other, slow-growing problems. I saw this post from Jacob Sebastian about production issues that don’t trigger alarms. There are just slowdowns that trickle across multiple systems and cause issues for clients. These aren’t things you instrument for, as a slowdown isn’t necessarily an issue. These things can resolve themselves, or they can develop into a major issue.

    I think about this like vehicle traffic. A minor fender bender on a highway might not be a problem, but it can become one. Cars don’t get out of traffic quickly enough, or traffic police don’t arrive soon and move the cars. Traffic starts to back up, which slows down the response, including that important tow truck that might keep things moving. Suddenly, it’s not a few people inconvenienced by an accident, but thousands.

    There are likely signals in your environment that would let you know about a potential issue coming soon. These are subtle and not always indicative of a problem individually, but taken together, they indicate a production issue is going to occur. To me, this is a place AI can eagerly be taught to look for these signals and then happily keep looking for them every day.

    The future of monitoring is the active examination of correlated data that precede an issue, hopefully giving humans, or other AIs, enough time to respond and prevent customers from experiencing a slowdown.

    I’d certainly welcome this in both traffic motorways and database systems.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Multiple Monitoring Tools

    Part of my Redgate work is with customers who need to monitor their database servers. With estates growing quickly, both in scale and types of database platforms used, keeping an eye on everything can be challenging. Add in the lack of staff growing as quickly are the number of servers, and I find many companies seeking out monitoring tools to better help them manage the entire estate..

    When someone evaluates a tool, one of the first questions from many people is about load. They are concerned about the load a tool puts on the system, which is always some amount. Most tools say they use less than 2% of total resources, some might hedge at 5%. Hopefully, there’s no more impact than 5%, though that might seem to high, especially if you have a busy database server already.

    I’ve seen several customers who have multiple monitoring systems. Often this is because each tool does something well, but lacks a feature or capability that another provides. Each of the tools needs its own data, which can result in more performance impact.

    Is it worth the overhead? If you had a second tool that provided more capabilities, would you ditch one of your tools? I know I work for a vendor that produces a monitoring tool (Redgate Monitor), but I’m genuinely interested in how many of you view the world.

    We often make trade-offs, but sometimes we aren’t willing to change. Perhaps you have a tool that works very well doing a certain task, and you don’t want to stop using it. I know I’ve been in that situation, and unless another tool adds that thing, or more things I desire, I’d likely continue to use two tools to accomplish all the things I need done.

    Specialized tools have a place, and it can be worth the hassle of using them when a more general tool just doesn’t get all the work done.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Scaling Up Monitoring

    Monitoring databases is important when it’s the systems that are in production. Operations departments know that catching issues early, being proactive, and having data to troubleshoot issues make their job easier. Not having these things makes their job much more stressful.

    Most of us work with data in some way and the availability of that is important. Certainly, security, integrity, and performance matter as well, but availability is key. Many organizations don’t have any monitoring systems set up. Instead, they troubleshoot problems when someone files a ticket or calls. I’m amazed at this, though I know building and managing a monitoring system is hard and purchasing third-party products can be outside of your budget. Still, having something in place makes everyone’s job easier.

    If you decide to build a system, then you can do it in many ways. I saw a description of how Amazon built a monitoring system for their Prime Video service. This is a more complex system than many of us deal with for databases, but I did find it interesting that they chose a distributed architecture that used multiple components. It didn’t scale, so they started to move from small functions, almost like microservices, to a bit more monolithic structure.

    I am not saying that microservices or functions or serverless are bad choices. They meet certain needs, and they can work very well. Azure SQL Database Serverless can work well in some situations. However, I do think that this was a case of engineers trying to be too clever and making assumptions about production loads from PoC-type experimentation.

    I would say that far too many software engineers think that their solution will scale without actually testing it. Too often their view is if it works here, it will work there, but the history of software has shown that working on my machine doesn’t mean working on another. That’s why we use Continuous Integration: for independent validation and verification. This is also a problem when databases are involved, as the level of data used for development and testing doesn’t do a good enough job of predicting how the system works under load. We need better test data management, which is becoming a whole new category of software practices and tools.

    We should ensure we include good instrumentation in our software for monitoring purposes, but we should also ensure that we start monitoring and evaluating how our system will perform in test and development environments, as that’s the idea of shift-left. Lastly, I think monitoring in production is important, but I wouldn’t build another system. I admit I’m biased, as I work for a company selling monitoring software. However, I also think the build v buy debate doesn’t make sense here unless your staff has a lot of spare time to spend maintaining a homegrown system. I’d like to think most of them have better things to do.

    Steve Jones

     

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • The Complexity of Metrics

    Monitoring your SQL Server instances is important to ensure you can meet your SLAs. Availability, performance, reliability, quality, whatever you care about, it’s important that whoever is responsible is looking at how the database is performing. At Redgate, we have multiple teams working on SQL Monitor to enhance and grow it to meet your needs.

    A short while ago there was an internal conversation recently about page life expectancy. We’ve had some customers ask about this and setting alerts to watch this value. Our developers and sales engineers asked for a few thoughts from Grant and others on how to respond. There are a variety of opinions, some saying monitor it, some saying don’t bother.

    I think both pieces of advice have merit, which is to say that this isn’t a metric that you can look at in isolation. There is no value of PLE that is good or bad, or that says x is wrong or y is right. There is both a subtlety and a complexity to understanding what PLE is telling you about your system. If PLE is growing, you have to look deeper. If it’s falling, same thing. If it suddenly drops, there are multiple possible causes, and you need to examine other things. However, in many cases, this isn’t an actionable metric, but one that provides context about what might be happening in the database when combined with other values you monitor.

    This certainly isn’t a metric that you want to set an alert on because it can rise or fall and many times the change isn’t indicative of an acute problem.

    This is just one metric of many that are available in SQL Server, and knowing which ones to monitor is something good administrators learn. They know that very few values they instrument have a good or bad value, and often the rate of change needs to be combined with the actual reading to determine if there is a problem. We also often want to know if a high (or low) reading appears for an extended period of time. Having 100% CPU being used for 3 minutes likely isn’t an issue. If it lasts for 3 hours, I might feel differently.

    Metrics have more complexity than just having a range in which we ignore them and a limit at which we alert people. They are intended to be combined with each other, with observations by clients, and with the experience of looking at past observations over time. Our systems often develop patterns, and we don’t get too concerned about any values when the pattern repeats. It’s when something new happens and someone complains that we dig in to determine if there is a problem or the start of a new pattern.

    We definitely need monitoring of our database metrics, but we also need to understand why values move and the implications of them doing so. That’s something which isn’t as simple as setting alert for each one based on some value we think should never be exceeded.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.