Voice of the DBA

Tag: software development

  • Scripting Makes Mistakes Easier Than Ever

    A number of you likely use Atlassian products like Jira, Confluence, Opsgenie, or something else. You might have been affected by a large outage they had (post incident blog, Company Q&A, TechRepublic report) recently which lasted at least 9 days. I don’t know if all customers have their data back and are working, but this was a surprisingly poorly handled incident according to a number of reports from customers. There’s a great write-up from the outside that you might want to read.

    The bottom line in this issue is that Atlassian looked to deactivate a legacy product with a script, but they apparently didn’t communicate well among their teams. The script ended up using the wrong customer IDs and also marked the sites for permanent removal, not temporary removal (soft delete). While they supposedly test their restore capabilities, they weren’t prepared for partial restores of subsites. I’m guessing this is likely a partial database restore, which many of us know is way more complex than a full database restore.

    Leave aside the issue of a software-as-a-service (SaaS) company failing their customers, and the lack of communication with customers. The more interesting thing for me is the challenge of poor coding and communication internally. Clearly, the project to deactivate their legacy app wasn’t well planned or tested and the code used was probably executed at too wide a scale initially.

    When we deploy code changes to a large number of items, we want to test them at a small number first. Whether we are deploying to multiple databases, against many customers, or different systems, a standard method of making changes at scale involves working in rings. Azure DevOps describes this in docs, and they actually use rings to change the platform. We used the same pattern 20 years ago for software and database updates to many systems. We would internally deploy to a few users to look for issues. Then a week later we would deploy to a small number of systems to check for unexpected issues. Then typically to most systems in the third ring with a fourth ring a week later to catch up stragglers that needed more time to prepare.

    I find many customers, especially those with sharded/federated databases or many systems unwilling to spread out deployments in this manner. Often they yield to pressure from business users to ensure everyone gets the same update at the same time. I would never recommend this approach as we need to ensure we are looking at scripts in a controlled environment, or even two, before we deploy things widely. I’d be even more cautious about one-off administrative scripts that might make a change similar to the one Atlassian attempted. Those are often not seriously tested enough.

    At the very least, any of us working with multiple customers in a single database or in multiple databases ought to ensure we can backup and restore a single customer, but more importantly, can you restore a group of customers. If you make a mistake like Atlassian, which scripting allows us to do extremely rapidly, can you recover a partial set of data? Many of us don’t test this, but that’s likely something we ought to consider when we work with scripts that are designed to only change some data. Most of us don’t experience complete failures, but partial ones, usually because of human error. We ought to know how to deal with these situations.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • The Backlog

    Software development is complex these days. Maybe it’s always been complex, but the shorter timelines and greater vision of what is currently happening in development teams mean that we can see a lot is going on at any particular time. Really, once you get beyond a handful of developers, the coding changes look like a chaotic ballet that may (or may not) be in sync across developers. Modern work tracking tools make this easier, and for any particular developer, it isn’t too hard to look at a Kanban board linked to a git repository and track the changes that have taken place.

    There is always more work to tackle than can be done at the current moment, so it is important to add new tasks to a backlog. Building those, and tracking what is needed, is a bit of an art. What do you write down, how much detail, and what level of effort is needed. Paul Andrew wrote a good post on what makes a good backlog story, using an example from his work. He looks at the need for infrastructure in this case, but that would be a common request these days when DevOps and the cloud require some blurring of the line between code and the resources it runs on.

    I like the completeness of this request, and I do think this level of detail is important, with the inclusion of criteria to measure the work and test that it is done. I just don’t know if this is required when we add the work to a backlog. The rapid changes in our software, as we adapt and work with customers’ feedback, mean that we might get many more ideas (or issues) than we ever build. In fact, some requests/suggestions/desires that get sent to developers or project managers might be no longer needed or forgotten by the time they reach the coding stage. I know I’ve certainly requested some features at SQL Server Central that I later decide aren’t important or needed.

    My view is that ideas ought to be captured, and triage ought to include enough detail to estimate how important and hard the work may be. Once we get to the point of actually performing work, then we need to flesh out details on what constitutes done and what tests can verify this. For very small levels of work, this might be just a few light tests, but even noting that a new column is in a table and associated stored procedures can help developers remember to check for dependencies and avoid simple mistakes.

    Building software and managing the process across a team and over time is hard. It is easy to lose control, and it can be hard to make decisions about what to build and what to defer. A backlog is critical, and managing that backlog with regular review and attention to detail is important. However, just as we don’t necessarily want to prematurely optimize code, I also don’t know that we need to build out the complete story when someone has a request. Grow your stories as you grow you code with an agile approach over time.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • The Right to Repair

    I’m a do-it-yourself (DIY) person in many ways. I work on equipment on the ranch, repair or improve things in the house, and often tackle electrical or mechanical tasks. I’ve replaced hydraulic hoses, changed oil, replaced brakes, changed outlets, wired up lights, and more. I keep a hand cleaning degreaser handy and use it regularly.

    I think many people who work in software development are the same way, and often I find they want to rebuild things themselves rather than adopt or improve what’s there. Many organizations have a fair amount of NIH Syndrome inside that results in money and resources spent on software that might be better used elsewhere. I think some of this is going away as current trends in software make use of APIs, OSS, and services to cobble together a workflow rather than just building something from scratch.

    However, that’s not always something we can do. In fact, we often can’t take apart commercial software and fix it, not even when it’s broken and in need of repair. That same philosophy is starting to pervade products in the real world, which is disappointing to me as someone that likes to be able to repair something I own.

    There is a good article on the right to repair over at Make magazine. In this case, the argument is that there are certain restrictions manufacturers put in place to limit the ability of individuals or third parties to repair products. In the physical world, I think this is a problem for actual devices, but it’s also becoming an issue as software pervades the operation of many physical devices. One of the higher-profile complaints has been with tractor service.

    I don’t know how we reconcile the right to repair and own something with the intellectual property rights of software. It is a thorny issue, and one that I am not sure of how I would want to frame the rights, but I do hope that we at least ensure that physical products people own can be repaired by individuals, including the ability to “reset” software as needed.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • The Test Mistake

    Last year a company was testing their software in a dev/test environment and accidentally sent out an “Integration email” to their customers.  They blamed this on an intern, though a little jokingly, and plenty of people shared their own stories about making mistakes.

    I did this once, and only once. I was doing some software testing on a copy of the production database and accidentally sent out thousands of emails to customers. I hadn’t scrubbed the email addresses in the copy, but since I also hadn’t scrubbed the data, customers got an “extra” email for their account and few complained. My boss, however, complained quite a bit and with good reason.

    These days using production data in dev/test environments isn’t really an acceptable practice. Plenty of organizations do it, and hopefully, they are taking precautions. Kendra Little wrote a little about what happened and what should be done, including scrubbing out sensitive data. Not enough people do this, though I am seeing more and more companies looking for solutions that prevent data release.

    This isn’t an intern’s issue. I appreciate someone making a bit of a joke here, and hopefully, there were engineers that taught the intern something. The bigger issue is that we make mistakes as humans and our protocols and processes should expect that. This is one reason why adopting automation in our software process is important. Whether that’s deploying code to the right production server or refreshing the non-production databases, we want to be sure that we limit the number of human mistakes.

    There are always places where we will use humans to perform a process. Whether that’s typing code or clicking something. It’s entirely possible someone clicks the wrong button or types the wrong item. As much as possible, we can use automation to provide a safety net to prevent these issues from reaching customers. We can’t prevent every mistake, but we can continue to improve our process but adjusting it over time. At the very least, we ought to prevent the same mistake from happening over and over.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.