Voice of the DBA

Category: Editorial

  • Forgiveness

    We are frail, fragile creatures. I think this is one of the main reasons I push for and try to work in an agile, DevOps style of development. I know that I make mistakes, as do others. My goal is often to ensure our system, our process, our way of working can transcend those mistakes and we can still be effective. We can use automation and regular examination of what works and what doesn’t to ensure our system works well and isn’t overly dependent on one person.

    Part of dealing with mistakes and failures is learning from them. Not only do we need to learn ourselves, and avoid repeating mistakes, but we also need to learn to forgive others. I thought about this as I was reading Derik Hammer’s post on Active Forgiveness. In his post, he covers the situations where we acknowledge mistakes without a punishment. Instead, we forgive the act and go to work on finding ways to prevent the same situation from happening in the future.

    I think this is one of those concepts and practices that are very difficult for most leaders, and maybe more followers. We don’t want to excuse the behavior, and we certainly don’t want to tolerate this in the future. What we want is for this person, and others, to learn from the mistake and work to prevent it in the future. What often happens is that we blame the person, we’re glad we’re not the person, and we may not learn a lesson because we think that we wouldn’t make the same mistake.

    Throughout my career I’ve seen the same mistakes occur over and over. Sometimes the same person makes a mistake again, but often different people make the same mistake because there wasn’t a product retrospective where everyone learned for the future. Having a blameless meeting where discussions take place, mistakes aren’t minimized, and everyone learns is hard, but it is important if we are going to improve our teams performance over time.

    I’d encourage you to try to find ways to learn from failures and grow. Forgive others actively, but don’t minimize the mistake. Hopefully you can also help your management learn to do this.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.1MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Complex Constrained Security

    I was reading about Kerberos and authentication with SPNs recently. It’s a topic that seems to make sense and appears orderly, but when I’ve had issues with SPNs, it feels like voodoo and black magic sometimes might be needed to get things working. As I read through the document, trying to ensure I would learn a bit more about how impersonation and delegation work, I noticed this sentence:

    “As a security best practice, Microsoft recommends constrained delegation over unconstrained delegation.”

    That seems reasonable to me. We ought to limit where users can connect to specific systems to ensure good security. This makes perfect sense where we have systems like web servers or application servers and we should limit delegation to specific databases servers. This wouldn’t prevent all security breaches, but it would limit the scope of many.

    The complexity comes when we start to have multiple servers that might connect to multiple back ends, especially as we grow our architectures to include additional HA nodes with Availabilty Groups. Tightly linking security complicates the configuration and requires that our sysadmins setup new machines and properly add new delegation targets as machines change. DevOps and configuration as code can help here with ensuring that we always add the required security changes to the right machines.

    That still doesn’t make it easy to manage a tight security environment without lots of resources. As we rotate or retire machines, we need cleanup of the security settings that refer to these objects. If we rotate host machines, which is usually rare, we need to remember to update out configuration scripts to work with new machines and accounts. If we add nodes, we need additional lines in scripts. If we move to containers for database servers, this might require even more changes.

    None of these items is complex, but when you must repeat them for many systems, many accounts, and on a semi-rare basis, they add some overhead that is both tedious and difficult to keep up with for a staff. This is especially true as staff turns over. Do you want to let the new people know that they need to make all these updates while handling their “normal work”? I could see all these details becoming a chore because we’re human, we’re flawed, and we make mistakes.

    I like the idea of tighter security, but at a scale, at random times, in between all the other tasks we must complete, the tools and techniques we have don’t make this something that seems manageable. I don’t have solutions, but I think that we do need some better tools that ensure security can be both flexible and convenient, while enforcing the principle of least privilege. The management of systems at scale is helping (forcing?) companies rethink some security tools and features, but there is still work to be done to ensure our employees will correctly and consistently configure security.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.0MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • DR Planning

    Data professionals know that protecting and ensuring our data is safe is one of our primary jobs. Many of us worry that if we were unable to recover a database, our job might be jeopardy, which makes sense. If we lose data, an organization might make the decision that we aren’t trustworthy enough to manage a database system. This leads many of us to plan and prepare for different types of disasters, with procedures and systems in place to ensure that we can quickly get things running after an issue.

    Natural disasters, or other large scale disruptions of business sometimes exceed the plans we’ve put into place. Fortunately they’re rare, but I ran across an article that talks about a few items that we might not consider when making plans. Even if you can fail your database over with an Availability Group running in another location, there are still some points you might want to think about.

    Perhaps the biggest one for me is that testing is not optional. I’ve seen some amazing plans put into place, but never tested because no one wanted to disrupt ongoing business activities. Testing is really critical, perhaps because of the second more important item I see cause issues: failback. This isn’t always as simple as we’d like, even with some of the amazing work that Microsoft has done with SQL Server. We want to ensure that we do know that if something happens to our primary systems and we move them, we can come back. After all, these are the primary systems for a reason.

    While other items such as making DR planning something your organization cares about can matter, I think the viability of this depends on how likely these disasters are. If you’re in a seasonal hurricane path, or some other disaster, you likely need to have better plans than if you’re in a stable region. Not that you can avoid any plans, but most of the time a disaster is a rare event and it’s really IT’s job to ensure that systems are resilient.

    I do think it is worth noting that regulatory compliance isn’t optional. While your auditor might have sympathy if they are likewise affected by the disaster, that sympathy dissipates over time. A disaster next month might not factor into an audit review in 20 months. More laws are on the books and more are coming than most of us have dealt with in the past. Design your system and process to be compliant, even in a disaster.

    Lastly, the people side of business can’t be emphasized enough. While some of us might be expected to shoulder a greater workload during a disaster, keep in mind that we would still need support outside of work, perhaps even a replacement worker to handle some of our work tasks or even help with personal items. The longer our lives are disrupted, the less likely we are to work as hard, or remain focused. Part of any plan should be how can our organization help support those that are struggling themselves. After all, the effects of a localized disaster on business could get compounded if staff stops coming to work.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.5MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Why Change?

    I was reading Change is Hard from Jeff Hackert and I started to think about the ways in which I both change and resist change. I then started to think about the ways in which I ask people to change what they’re doing. I do this both at work and in life, with mixed levels of success. The one thing the article got me to think about was the idea that in asking someone to change, I’m implying they are doing something poorly, or wrong, already.

    In some cases, that’s true. When I coach volleyball, I am often telling someone they are not practicing a technique correctly, and they need to change. The trick here is showing them what they are actually doing, since many of us don’t realize when we are moving our body incorrectly. We think we’re doing what we were told, and often it’s only with video evidence do we realize what we’re doing wrong. Once we do, it becomes easier to make corrections.

    In technology, this is often more nebulous. There isn’t often a set way that we should write code. Requirements vary, our schemas and data differ, and we almost always have disparate (and often not enough) indexes. Asking someone to change could very well mean asking them to admit they built something incorrectly. Sometimes we may do that, but often we have confidence in our work. Asking someone else to change how they build or develop software can be hard.

    What can even be harder is getting management to change. We often feel pressure from management, who usually see our systems working well enough. While we may see sub-optimal performance, they see business getting completed. While we see stress and long hours, they see work getting done at the same cost. Changing priorities to improve our applications often becomes secondary when potential new features and functions exist. Telling management they’re wrong is not only difficult, it’s often fruitless.

    Change is hard. Most humans like the status quo to continue indefinitely, even when we know that things can improve with some alteration in our routine. Inspiring others to join in and invest for the future is a skill. It’s one that I’d encourage you to develop, both with peers and management.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.8MB) podcast or subscribe to the feed at iTunes and Libsyn.