Voice of the DBA

Category: Editorial

  • This is Why You Use Git for Scripts

    Git has become a fantastic tool for me, and many other technologists, over the last ten years. It’s almost ubiquitous in most of my clients, and so many people are comfortable with it. Many others aren’t, which is why I started a Git series for DBAs (and other Ops people) on my blog.

    Quite a few people asked me why I recommend git over a file share for storing code that a team of Ops people or DBAs might use. Why isn’t a global file share a better choice in an organization? I think I have a few good reasons, but if you disagree, let me know in the discussion for this piece.

    First, I’ve worked with a lot of file share tech over the years. Microsoft has done a good job of trying to ensure that a “file server” works well for organizations. I’ve used simple file shares on a particular server, I’ve had mapped drives to a file share appear on every desktop in the org, and I’ve even had to work with large orgs that use DFS (or something similar) to ensure we can replicate files across multiple servers for backup and speed.

    All of those work, but they have a complexity, and a dependency on the team managing the file share. They also require a network connection to the share in order to see the files, which isn’t always as reliable and solid as we would like, especially in this modern world of working at home, databases in the cloud, and a widely dispersed workforce that uses BYOD devices. Don’t get me wrong, those aren’t bad things, but they complicate the idea of having a single global location for stuff.

    Second, I want to version scripts because sometimes one team member alters a script for their piece of work, but it causes issues with another member’s use of the script. If we catch this quickly, we can usually CTRL+Z or revert the script, or we can copy the script to a new file and then try to edit it for a different purpose. The challenge is we don’t always catch it quickly. We also sometimes don’t know what was in the old script. While I wish most of you always logged the results of all executions, at the very least, save the script you ran. That’s not easy to do in a file share, and it’s something we don’t think about. Getting used to committing code helps us remember to do this, or at least, get the history of the files.

    Third, in the case above, the last writer wins. In a busy team, I’ve seen person A save changes to the script, and then run it, while person B saves changes to the same script and runs it. Person A might get a “reload” notification and re-save theirs. Person B might also do that, or they might reload Person A’s changes and lose theirs. It’s a nightmare. Imagine group work in Google Docs, trying to edit scripts with others while you try to execute them and complete some tasks. That would be frustrating and it’s a really good reason to use Git. Let’s not step on each other’s toes.

    Lastly, with a file share, I’m dependent on some other group for backup and recovery. If they have an issue, I am hoping they didn’t mess up my scripts. I have to wait on them if they are doing maintenance on the file share or patching the server.

    With Git, I can keep the scripts on my machine, while easily replicating and sharing with others. We each have a backup of the script in case of emergency. We can even protect the main branch, forcing everyone to make changes on another branch and then use a pull request so that someone on our team can approve merging those changes into the main folder that we all use. If we don’t like the script, or there are breaking changes, we haven’t edited a file that others are using because it’s in a branch.

    Using Git (or any VCS) is a change in habit for many people, but it has a lot of benefits. I think it’s worth the investment of time for teams.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

  • Un-Stable Teams

    I’ve always valued having a team of people I know and can work with. While I haven’t had a lot of long-term jobs before Redgate, I have had a few positions that lasted more than a year and appreciated working with the same group for a long time. We might gain or lose a person, but overall, the structure of the team was the same day after day. This was a comfortable atmosphere, and I liked knowing who I was working with each day.

    At Redgate, we have had some stable teams of people, but in our engineering area, we move teams around. There is an annual re-teaming each December/January where engineers can choose to leave their team and ask to transfer to another one. They get to put in their top three choices (or remain on their team), and we do a good job of trying to match up everyone’s preferences. The number and charter of teams do change a bit each year, so engineers get visibility into the structure we’re planning before they mark a preference. It seems from our internal reports that we match up 99% of engineers with their first or second choices (first choice is in the high 80s).

    Recently our head of engineering talked about the dangers of a long-term stable team vs. making changes. Chris Smith noted that 25-33% of engineers look to move, which is my impression. I get to know who works on different products (or parts of them), and while I often know year-to-year who can help with which issues, I’m also surprised by the changes at times.

    I’m also disappointed sometimes because some very talented engineers will leave a team to move to another. That can worry me as I know how much knowledge is leaving. At the same time, as Chris pointed out, sometimes very talented people will leave an organization because they feel stuck in one job. There also is value in cognitive diversity, which is something I look for. We need to get along, but I like working with people that think differently than I do.

    I haven’t seen this need to move as much in operations teams as in development teams. Maybe the nature of the person who supports systems is different than those who build them, but I do find long-term system administrators sometimes struggle with change and improvements. I’m not fond of change for change’s sake, and I want to justify new technologies/protocols/etc. However, I also think as technology changes, we ought to objectively evaluate if it’s better. Not all change is, but some certainly improve the environment.

    Do you prefer a stable team or do you like to change periodically? A lot of companies move people around, sometimes on a schedule, to ensure they can grow and bring new ideas and perspectives to other teams. Does your employer do this? Do you like the idea or would you prefer not to re-team periodically?

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

  • Do You Have a Jeff?

    In the Phoenix Project (worth a read), there is a character called Brent, who is to go-to person for everything in IT. I don’t know if this character was modeled after Brent Ozar, but I always picture him when I re-read the book, and I suspect he was that person in previous positions. I’ve been that person as well, and it’s both exciting, fulfilling, and very stressful. At Redgate, that person has been Robert C, who is my go-to person for many questions.

    In the DBA world, I think of Jeff Moden. He’s been a prolific and incredible author over the years on many things SQL-related and is a huge proponent of others learning to write better code and better utilize the database platform more efficiently. I suspect in his company, he is the go-to person for most database-related questions and problems. I also suspect he solves most of them very well and has the influence (or power) to effect change.

    However, are you Jeff in your organization? Do you have a Jeff? In many of the customers I work with, there is no Jeff. Sometimes there are very smart people, but they cannot effect change. Or they don’t know how to go about making change happen. In other companies, there isn’t anyone who is an expert on the database or related technologies. Too often, I think people aren’t often aware of what expert really is and they just do the best they can without knowing if it’s a great job or a poor one. That’s my opinion, but it’s based on decades of experience (and success) working with lots of others on database code.

    The good news is that most of you can learn to be Jeff. You can work to improve your skills, both technical and interpersonal (soft), to raise the quality bar at your organization. Maybe you can get more work done and get a huge sense of satisfaction from your job (and a raise). Maybe you just want to get things done more efficiently and get away from work more often.

    There are many ways to do improve the quality at work, but essentially the idea behind platform engineering is to produce tools that make developers and operations groups more efficient. The goal is to get more work done, easier, reducing the cognitive load on everyone involved with software. Not because they can’t do the work, but because we want them focused on their specialty. Whether that’s writing C# code for an application, writing zero downtime database changes, or making beautiful reports. The hassles of actually capturing and moving code should be something provided by a platform. A person can do some of this by sharing their knowledge, not just in conversations, but providing templates, models, standards, and other code elements that might help their co-workers become more productive.

    DevOps is about producing code quicker, reliably delivering that to the customer, and raising the quality bar (don’t forget this) and platform engineering is the next evolution here, where the tools, process, and flow are set to make the implementation of DevOps easier for most developers and DBAs. Even if your organization doesn’t want to produce a platform, the idea of building and deploying tools (samples, models, snippets, etc.) to make someone else’s job (or yours) easier and smoother is something we all can do. We can all aspire to be Jeff and get there with a little investment in our skill across time.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

  • Ransomware vs. AI

    Ransomware has been a growing and shrinking problem in the modern world. Every time I think that some new defenses and protections are preventing ransomware from being a problem, I see another issue. Recently, I saw Subway got hit with with an attack and a few friends have recently noted their companies were restoring systems after a portion of their network was locked down.

    With the advent of Rasnsomware-as-a-service, where criminals deploy software and then sell access to others, better detection and protection become more important. As with any software, criminal human operators will use the ransomware software in different ways. That means that we don’t necessarily have a simple threat that can be easily programmed against with anti-virus technology.

    Microsoft has been using AI technology to help them track and combat ransomware campaigns. Since there are similarities between how ransomware is used by different individuals and how it appears in systems, AI technology can be helpful here. There aren’t the same simple signatures on files that we’ve seen in the past with viruses, but rather more complex patterns. Humans might discover how ransomware appears in their environment with lots of knowledge on what their network ought to look like, but this pattern matching across many different networks and organizations is something that AI/ML might do quicker and at scale. Once successful, ransomware can be hard to recover from, so early detection is important.

    In the article, it seems that Microsoft is capturing lots of traffic and analyzing it for patterns, with multiple types of anomalous activity, and then aggregating this across devices to guess whether this is an attack or not. In some of their testing, they find the ability to stop an attack with only a few percent of assets getting encrypted. That’s not perfect, but better than finding 90% of your nodes are encrypted over morning coffee.

    I suspect this is just the latest escalation in cyber attacks and defenses. I’m sure that hackers will come up with new and novel ways to cause problems, but I do think that this is a place where AI, especially ML, technology can be useful to provide better security. I also think that database technology, especially graph queries, is particularly helpful here. I hope we learn more about how they are building protections as I think this is likely a great data analysis story.

    Once again, the database is critical to making software better because all of that data has to be stored and queried somewhere.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.