Voice of the DBA

Category: Editorial

  • Better Communication Through Less Messaging

    I have learned to really appreciate and enjoy Slack as a messaging tool. It’s something I use daily, and a place where many inside my company communicate about all sorts of issues. There certainly can be an overload of channels, but for me, I add and prune channels regularly and it’s a good way to segregate conversations.

    I didn’t feel that way when I started. At first, I resisted using it. Now I couldn’t imagine not having it, but not everyone feels the same way. There’s an article about a SaaS provider moving their company off Slack and instead using the Basecamp project management platform and keeping communications inside there.

    Why? They say they have less meetings, less interruptions, less direct messaging, and more productivity.

    Interesting conclusions. I certainly can see that some people might find Slack to be chatty, and there are definitely lots of channels devoted to non-work items. There can be a lot of unread channels from me, but I don’t know that we would have less meetings without Slack. We have various work tracking and productivity tools, like SalesForce, trello boards, etc. However, those aren’t the places I find it convenient to move communications.

    Personally, I don’t have a lot of DMs, and I like that I can see unread channels, ignoring them when I’m busy and looking at them when I have time. I can also just mark-as-read a channel when I want to declare thread bankruptcy.

    I usually don’t find Slack to be too intrusive to my day. I work fairly independently and asynchronously, as I’d expect many developers to work. I also don’t know that it causes more meetings, at least not for me. There is a fairly high bar to opening a meeting and I find many people don’t bother. Instead, we can have a discussion, in real time or across days, that deals with a topic.

    What do you think? If you use Slack (or Teams/etc.) is this a drain on productivity or an enhancer? I think the latter.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Scaling Up Monitoring

    Monitoring databases is important when it’s the systems that are in production. Operations departments know that catching issues early, being proactive, and having data to troubleshoot issues make their job easier. Not having these things makes their job much more stressful.

    Most of us work with data in some way and the availability of that is important. Certainly, security, integrity, and performance matter as well, but availability is key. Many organizations don’t have any monitoring systems set up. Instead, they troubleshoot problems when someone files a ticket or calls. I’m amazed at this, though I know building and managing a monitoring system is hard and purchasing third-party products can be outside of your budget. Still, having something in place makes everyone’s job easier.

    If you decide to build a system, then you can do it in many ways. I saw a description of how Amazon built a monitoring system for their Prime Video service. This is a more complex system than many of us deal with for databases, but I did find it interesting that they chose a distributed architecture that used multiple components. It didn’t scale, so they started to move from small functions, almost like microservices, to a bit more monolithic structure.

    I am not saying that microservices or functions or serverless are bad choices. They meet certain needs, and they can work very well. Azure SQL Database Serverless can work well in some situations. However, I do think that this was a case of engineers trying to be too clever and making assumptions about production loads from PoC-type experimentation.

    I would say that far too many software engineers think that their solution will scale without actually testing it. Too often their view is if it works here, it will work there, but the history of software has shown that working on my machine doesn’t mean working on another. That’s why we use Continuous Integration: for independent validation and verification. This is also a problem when databases are involved, as the level of data used for development and testing doesn’t do a good enough job of predicting how the system works under load. We need better test data management, which is becoming a whole new category of software practices and tools.

    We should ensure we include good instrumentation in our software for monitoring purposes, but we should also ensure that we start monitoring and evaluating how our system will perform in test and development environments, as that’s the idea of shift-left. Lastly, I think monitoring in production is important, but I wouldn’t build another system. I admit I’m biased, as I work for a company selling monitoring software. However, I also think the build v buy debate doesn’t make sense here unless your staff has a lot of spare time to spend maintaining a homegrown system. I’d like to think most of them have better things to do.

    Steve Jones

     

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Validating Password Expiration

    I would guess that the majority of instances I’ve had to manage in my career were those that I didn’t initially install and configure. I’ve inherited more instances than I would bother to count, and I often need to double-check what’s been done in the past. As noted in the series on new jobs from Tracy and Josephine, there are a lot of settings to check and adjust to meet your standards.

    While backups are often my first priority, security is second. I usually want to know who the sysadmins are and ensure systems are patched and configured to reduce the attack surface area. There is one other security check that I think I haven’t always been overly concerned about checking: password expiration.

    There was a post from Steve Stedman recently that mentioned the way to alter logins and ensure they have CHECK_EXPIRATION set ot on, which ensures that passwords expire and need to get changed. This is especially important for sysadmins. I try to ensure those accounts in that role are secured with AD, but there have been times when SQL accounts are used. Usually, I disable sa, but I’ve seen other accounts, especially those used by monitoring systems who seem to think sysadmin is required. It’s not.

    I don’t know that I’ve run queries to check the value in the is_expiration_checked column is appropriately set. If it’s not, then Steve’s post above will help you change those logins. That’s a handy script to have set up and use to ensure that all logins have this set. In fact, this is one of those areas where new logins could be created by junior administrators and not set the option. Perhaps this is something you want to run on a regular basis, perhaps weekly, to ensure that if any new SQL logins are created, they are done so with the password expiration set.

    Ideally, no one would ever create logins without expiration set, but sometimes things happen. I’ve seen monitoring systems set up with sysadmin privileges and passwords that never expire. A surefire way to dramatically increase the risk to your database systems. It would be better to have a known, consistent process for setting up accounts. Some companies have specific scripts, or snippets, that administrators use when tickets are filed. One customer of mine had even linked a script to a Slack command in a sysadmin channel. Only admins could use this channel, but they could use Slack to kick off scripts to create logins, add roles, and force password changes.

    No matter how you choose to handle security at a process level, it is important to include monitoring and remediation for issues. Mistakes will get made, settings altered, and exceptions approved. Sometimes we can fix things, sometimes we cannot, but knowing what our environment looks like and where we have potential issues is important not only for getting the work complete but getting the approvals to make changes that ensure better security. My recommendation is that you ensure you have a way to regularly check your systems, automatically fix issues where appropriate, and report on those that need additional approvals.

    Steve Jones

  • AI Thoughts on the Build Keynote

    If you haven’t seen the Build 2023 keynote, it’s, well, interesting. At a surface level, it’s focused on AI and delivers some demos that many of us might find to be useful and intriguing. I didn’t attend the event (or watch it live), but I did see it a bit later and I made some notes, pausing the 30-minute talk a few times to think about what I’d seen.

    The opening lightly glosses over some of the AI enhancements to development tools and the environments that can be created quickly in GitHub or Azure. Some of us will like those, and maybe they’ll grow on me, but I tend to prefer a development environment on my own hardware, where I have unlimited compute power at a fixed cost. The first big announcement is then showing Copilot technology, essentially some ChatGPT-like abilities, embedded into Windows 11. The demo shows asking Windows where settings are, with the response including buttons to take actions, like setting dark mode. Minorly useful, though I think Windows search works fine. I can type “env” and get the “edit environment variables” in the results. I still have to click through to change things, but this doesn’t seem like a better use of AI, especially if I need to type “set dark mode” instead of “dark”.

    To be fair, the demo has the user asking for ways to adjust the system to get more work done. The suggestions are for dark mode and a focus timer. I knew about the former, but not the latter. Perhaps being able to ask for general assistance with tasks is useful as there are likely lots of features I know nothing about and wouldn’t even think to look for. There is also the option to drop a document, like a PDF, in the chat and Windows asks if the user wants the system to “explain”, “rewrite”, or “summarize” the document. The user clicks summarize and gets a summary of the document.

    There is also a demo with plugins that developers can write for Bing, such as one that uses a legal package to make a change to a document. While lawyers might be worried about their practices (or paralegals about job prospects), I’m more worried about a fundamental problem that many of us data professionals have seen in the past: garbage in, garbage out.

    In this case, if the AI model isn’t well-trained, can I really trust it to summarize a PDF or change a legal document? How can I tell if it’s wrong, or slightly off? In some sense, this reminds me of a high school report. It might summarize some text at an A level, or a D level. It’s up to me to judge that, and I can’t assume the results are good or bad.

    The important thing to keep in mind, however, is that we aren’t in that place with AI. We can’t just trust the AI. We are in a time when AI is an assistant, where it can help us complete a task or get something done a little quicker. We are still responsible. We still have to verify and do some work, but if the Copilot can automatically launch Jira and navigate to a ticket, or attach a document and create a short message to our team, that saves us time. It saves us tedium. It can make our jobs easier. We are still needed, but we don’t do all the heavy lifting.

    I do worry about some of the opportunities for plugins that developers will write strictly to monetize their efforts. If I want a shopping list, I don’t want it to go to Instacart. I want a list I can use. I realize that doesn’t necessarily make Microsoft or a developer any money, but not all the tasks and advances are about profit. Or at least, I hope they all aren’t. I hope some are here to just make the world better. For a quick view of what that could be, watch the keynote closing video.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.