Voice of the DBA

Category: Editorial

  • How Many Days Can You Survive?

    I saw a great post from DCAC on disaster plans and using them after a fire in an LA data center. DCAC wasn’t affected, and I wouldn’t expect them to be. Denny, Joey, Monica, John, Kerry, and Meagan are all experts in running systems efficiently and effectively. They think about how to ensure that things keep working in the event of disasters and have delivered presentations all over the world helping you become better at managing your own servers.

    However, Denny brings up a great question in the post. How many days could you survive without IT systems? Days? Weeks? Have you ever been in that situation? I’m sure many of us have experienced a failure of some sort. An application crash, hardware dying, or most commonly, Internet access being cut. All of these are small disasters, which typically are fixed quickly. Hopefully, these aren’t issues you experience every week. If you do, maybe you need to call DCAC or someone else and fix more fundamental issues.

    While it’s unlikely that you lose all your systems, it could happen if you concentrate your resources in one data center, one region, one cloud provider, etc. Having a single point of failure is something we try to avoid in IT, and that is true not just for one application, but for your infrastructure design. Most of us depend on one authentication system, and a failure of our Active Directory could lock everyone out of a system. Those are rare, and hopefully, your administrators have enough redundancy (and backups) to recover from this type of disaster.

    I have experienced a few large failures at one enterprise. We had a few viruses, including the SQL Slammer worm in the early 2000s. Our network was shut down for a couple of days, when almost all systems from email to CRM to ticketing systems weren’t available. Everyone had to use whatever paper systems they could to keep business running. While we likely lost some revenue from these outages, we learned we could survive a few days without our network. We also learned that we needed better virus scanning and education for employees, as well as a few more resources for tech people. Before those events, everyone assumed an outage was bad, but had no idea how bad. I have no idea how much this cost us, but it didn’t appear in a 10-Q, so it must not have been too bad.

    I think there are lots of businesses that could find ways to continue to work if some systems were down. However, there can be costs, sometimes significant. Even if the company doesn’t go out of business, perhaps some people get terminated because of less revenue. That might not be the tech person in the short term, but how would you feel if your DR plan didn’t work (or you didn’t have one) and some co-workers were let go?

    The move to the cloud, and the move to more software-as-a-service systems, might help you better survive local disasters, but if you have too many systems concentrated in one place, it is worth preparing for some contingency. After all, even if this fire were to happen in an Azure or AWS data center, it’s possible that their process to move and restore all the systems from one data center to another could take time. Your systems might not even be their top priority as cloud vendors have some large customers. It probably won’t take months, but I wouldn’t want to bet my job on any cloud vendor getting everything moved in less than a week.

    If you’re not in the cloud, make sure you have a plan. If you don’t know how to do that, call DCAC or another consultant to help you.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Better Communication Through Less Messaging

    I have learned to really appreciate and enjoy Slack as a messaging tool. It’s something I use daily, and a place where many inside my company communicate about all sorts of issues. There certainly can be an overload of channels, but for me, I add and prune channels regularly and it’s a good way to segregate conversations.

    I didn’t feel that way when I started. At first, I resisted using it. Now I couldn’t imagine not having it, but not everyone feels the same way. There’s an article about a SaaS provider moving their company off Slack and instead using the Basecamp project management platform and keeping communications inside there.

    Why? They say they have less meetings, less interruptions, less direct messaging, and more productivity.

    Interesting conclusions. I certainly can see that some people might find Slack to be chatty, and there are definitely lots of channels devoted to non-work items. There can be a lot of unread channels from me, but I don’t know that we would have less meetings without Slack. We have various work tracking and productivity tools, like SalesForce, trello boards, etc. However, those aren’t the places I find it convenient to move communications.

    Personally, I don’t have a lot of DMs, and I like that I can see unread channels, ignoring them when I’m busy and looking at them when I have time. I can also just mark-as-read a channel when I want to declare thread bankruptcy.

    I usually don’t find Slack to be too intrusive to my day. I work fairly independently and asynchronously, as I’d expect many developers to work. I also don’t know that it causes more meetings, at least not for me. There is a fairly high bar to opening a meeting and I find many people don’t bother. Instead, we can have a discussion, in real time or across days, that deals with a topic.

    What do you think? If you use Slack (or Teams/etc.) is this a drain on productivity or an enhancer? I think the latter.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Scaling Up Monitoring

    Monitoring databases is important when it’s the systems that are in production. Operations departments know that catching issues early, being proactive, and having data to troubleshoot issues make their job easier. Not having these things makes their job much more stressful.

    Most of us work with data in some way and the availability of that is important. Certainly, security, integrity, and performance matter as well, but availability is key. Many organizations don’t have any monitoring systems set up. Instead, they troubleshoot problems when someone files a ticket or calls. I’m amazed at this, though I know building and managing a monitoring system is hard and purchasing third-party products can be outside of your budget. Still, having something in place makes everyone’s job easier.

    If you decide to build a system, then you can do it in many ways. I saw a description of how Amazon built a monitoring system for their Prime Video service. This is a more complex system than many of us deal with for databases, but I did find it interesting that they chose a distributed architecture that used multiple components. It didn’t scale, so they started to move from small functions, almost like microservices, to a bit more monolithic structure.

    I am not saying that microservices or functions or serverless are bad choices. They meet certain needs, and they can work very well. Azure SQL Database Serverless can work well in some situations. However, I do think that this was a case of engineers trying to be too clever and making assumptions about production loads from PoC-type experimentation.

    I would say that far too many software engineers think that their solution will scale without actually testing it. Too often their view is if it works here, it will work there, but the history of software has shown that working on my machine doesn’t mean working on another. That’s why we use Continuous Integration: for independent validation and verification. This is also a problem when databases are involved, as the level of data used for development and testing doesn’t do a good enough job of predicting how the system works under load. We need better test data management, which is becoming a whole new category of software practices and tools.

    We should ensure we include good instrumentation in our software for monitoring purposes, but we should also ensure that we start monitoring and evaluating how our system will perform in test and development environments, as that’s the idea of shift-left. Lastly, I think monitoring in production is important, but I wouldn’t build another system. I admit I’m biased, as I work for a company selling monitoring software. However, I also think the build v buy debate doesn’t make sense here unless your staff has a lot of spare time to spend maintaining a homegrown system. I’d like to think most of them have better things to do.

    Steve Jones

     

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Validating Password Expiration

    I would guess that the majority of instances I’ve had to manage in my career were those that I didn’t initially install and configure. I’ve inherited more instances than I would bother to count, and I often need to double-check what’s been done in the past. As noted in the series on new jobs from Tracy and Josephine, there are a lot of settings to check and adjust to meet your standards.

    While backups are often my first priority, security is second. I usually want to know who the sysadmins are and ensure systems are patched and configured to reduce the attack surface area. There is one other security check that I think I haven’t always been overly concerned about checking: password expiration.

    There was a post from Steve Stedman recently that mentioned the way to alter logins and ensure they have CHECK_EXPIRATION set ot on, which ensures that passwords expire and need to get changed. This is especially important for sysadmins. I try to ensure those accounts in that role are secured with AD, but there have been times when SQL accounts are used. Usually, I disable sa, but I’ve seen other accounts, especially those used by monitoring systems who seem to think sysadmin is required. It’s not.

    I don’t know that I’ve run queries to check the value in the is_expiration_checked column is appropriately set. If it’s not, then Steve’s post above will help you change those logins. That’s a handy script to have set up and use to ensure that all logins have this set. In fact, this is one of those areas where new logins could be created by junior administrators and not set the option. Perhaps this is something you want to run on a regular basis, perhaps weekly, to ensure that if any new SQL logins are created, they are done so with the password expiration set.

    Ideally, no one would ever create logins without expiration set, but sometimes things happen. I’ve seen monitoring systems set up with sysadmin privileges and passwords that never expire. A surefire way to dramatically increase the risk to your database systems. It would be better to have a known, consistent process for setting up accounts. Some companies have specific scripts, or snippets, that administrators use when tickets are filed. One customer of mine had even linked a script to a Slack command in a sysadmin channel. Only admins could use this channel, but they could use Slack to kick off scripts to create logins, add roles, and force password changes.

    No matter how you choose to handle security at a process level, it is important to include monitoring and remediation for issues. Mistakes will get made, settings altered, and exceptions approved. Sometimes we can fix things, sometimes we cannot, but knowing what our environment looks like and where we have potential issues is important not only for getting the work complete but getting the approvals to make changes that ensure better security. My recommendation is that you ensure you have a way to regularly check your systems, automatically fix issues where appropriate, and report on those that need additional approvals.

    Steve Jones