Voice of the DBA

Category: Editorial

  • The Backlog

    Software development is complex these days. Maybe it’s always been complex, but the shorter timelines and greater vision of what is currently happening in development teams mean that we can see a lot is going on at any particular time. Really, once you get beyond a handful of developers, the coding changes look like a chaotic ballet that may (or may not) be in sync across developers. Modern work tracking tools make this easier, and for any particular developer, it isn’t too hard to look at a Kanban board linked to a git repository and track the changes that have taken place.

    There is always more work to tackle than can be done at the current moment, so it is important to add new tasks to a backlog. Building those, and tracking what is needed, is a bit of an art. What do you write down, how much detail, and what level of effort is needed. Paul Andrew wrote a good post on what makes a good backlog story, using an example from his work. He looks at the need for infrastructure in this case, but that would be a common request these days when DevOps and the cloud require some blurring of the line between code and the resources it runs on.

    I like the completeness of this request, and I do think this level of detail is important, with the inclusion of criteria to measure the work and test that it is done. I just don’t know if this is required when we add the work to a backlog. The rapid changes in our software, as we adapt and work with customers’ feedback, mean that we might get many more ideas (or issues) than we ever build. In fact, some requests/suggestions/desires that get sent to developers or project managers might be no longer needed or forgotten by the time they reach the coding stage. I know I’ve certainly requested some features at SQL Server Central that I later decide aren’t important or needed.

    My view is that ideas ought to be captured, and triage ought to include enough detail to estimate how important and hard the work may be. Once we get to the point of actually performing work, then we need to flesh out details on what constitutes done and what tests can verify this. For very small levels of work, this might be just a few light tests, but even noting that a new column is in a table and associated stored procedures can help developers remember to check for dependencies and avoid simple mistakes.

    Building software and managing the process across a team and over time is hard. It is easy to lose control, and it can be hard to make decisions about what to build and what to defer. A backlog is critical, and managing that backlog with regular review and attention to detail is important. However, just as we don’t necessarily want to prematurely optimize code, I also don’t know that we need to build out the complete story when someone has a request. Grow your stories as you grow you code with an agile approach over time.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Advanced Incident Response

    Early in my career, I worked at a few smaller companies where a problem with the computer systems meant I went into the office and stayed until things were working. As I went to a few larger companies, I wasn’t alone when a system went down, but the process was mostly the same. We figured out what was wrong and found a way to fix or replace things, occasionally with help from a vendor. Those were the (not necessarily) good days before our internal networks were connected to a public Internet.

    Companies developed formal incident response plans to deal with various issues, whether these were problems we caused ourselves or failures of an application. I had the fortune, or misfortune, to be involved in more than a few issues and learned a great deal in how to solve problems as well as how to manage the impact to a large number of employees.

    As email and Internet use grew, so did the attacks with viruses and other sorts of malware. Antivirus software helped a great deal, but these days ransomware seems to be a common problem that isn’t as preventable as I would have hoped. Quite a few friends have dealt with ransomware issues, most of which have not been widely reported in the news.

    I saw an article about a few things that you might want to consider adding to your incident response plan. While some of these items might be unique to the ransomware threat, the thing that struck me was that there is a need to react quickly, in real-time, in response to any detection of an issue. I can only assume this means that there needs to be some advanced monitoring of nodes to detect issues, and I’m not sure how many organizations would adopt this, but in today’s world where we want systems available 24/7, perhaps they will.

    Being on call is a part of working in many IT departments. Having a response plan, even the general outline of one, helps to coordinate resources and ensure that we use people effectively. Tools are important, especially in today’s complex world, and it is important that one of those tools is a simple backup, preferably air-gapped from your main systems. If you don’t have these things in place, you might suggest someone start assembling them. These days it seems it’s not if you will get attacked, but when.

    Lastly, I don’t often see this addressed in plans, but make sure you have spelled out some guidelines on rotating staff and getting rest. Far too many companies want “all hands on deck” and forget that normal business still needs to occur. Any incident could last longer than a day, and you want to ensure that some of your staff is fresh and ready to take over from those that do need rest. Don’t be afraid to send some people home, or better yet, don’t call them in the first place.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Executing Notebooks in a Stored Procedure

    Jupyter Notebooks are a popular way of consolidating a number of code batches together and them executing them as separate batches or all together. These are essentially a document with notes and code, all of which are stored together. Databricks can run these, as can a number of other services. They are also integrated into various tools, like Azure Data Studio.

    Just like many other technologies, we’d like to execute these in a way that suits our environment. While running these in a tool like ADS works fine, we might want to schedule these and check the results later. You can run notebooks with PowerShell, but keeping around results and managing the data is hard. Since many of us use SQL Server or Azure SQL Database, wouldn’t it be great if we could find a way to have these run inside SQL Server and keep the results around?

    I was poking around inside a database and discovered that I could do this. Only with notebooks that have T-SQL code, but that’s usually what we’d want to do inside a SQL Server (or Azure SQL) database. I need to write a more detailed article on this, but there are a few basics that might help some of you understand how this works. Essentially, there is an undocumented extended stored procedure, xp_notebookparse, that will read the JSON internals of a notebook and allow you to extract out there T-SQL batches. The @execute = 1 parameter will cause each of these batches to be executed.

    For results, a new notebook is created and added to a local temp file, #nbresults, as a BLOB value. You can explore the structure of this, but essentially, part of your stored procedure should take all the data from this table and add it to a permanent table in your database. If you use a stored procedure to wrap the extended stored procedure call and an INSERT..SELECT call together then you’ll actually get nothing to happen because today is April first. It’s April Fools and this is a joke. Hopefully, you aren’t too disappointed.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • My Favorite Data Platform Feature

    At SQL Bits a few weeks back there was a community keynote on Friday. Ben Weissman and Rob Sewell put this together as a fun way of involving others. They invited a number of people to each take 5 minutes and talk about their favorite part of the data platform, which encompasses a wide variety of technologies and products. There were a few people who chose Purview, Azure networking, and quite a few ways of using Power BI.

    Me? I choose the Create button in Azure. Really, Azure itself because of the amazing array of things that you can deploy in minutes. It’s stunning to me to think back to the mid-2000s, when I first saw the Azure platform at TechEd and a Microsoft employee demo’d a key-value table, literally a two-column table. They were trying to sell this as a great way to deploy lookup data to mobile sales applications at scale. I was less than impressed.

    This year I deployed a Synapse workspace in less than 10 minutes. Maybe less than 5. I know little about Synapse, and the idea of building a data warehouse and analytics platform, loading data, and running queries isn’t something I’d think I could do in 10 minutes, but I did it. I’ve found containers to be similarly useful in quickly getting something up and running, but even they require Docker or some other software installed and configured. In Azure, I can set things up in minutes.

    In the last year I’ve set up numerable resources, from databases to web apps to a Kubernetes install. All in minutes, which is quicker than I could do something on my local system except for creating databases. If I had prepped things or had more knowledge that might not be the case, but the lack of needing to prep or learn a lot to get something set up is amazing in and of itself. This is especially true when I want to experiment with a technology or a new solution. I can quickly build an array of systems with much less effort than on-premises.

    I don’t know how many of you use the cloud at work, but I find it amazing. It’s one part of the data platform from Microsoft that I really appreciate, especially as I’ve tried to work with MongoDB, Redis, and other technologies at home. The cloud makes things easier, even with databases.

    Today, I’m wondering what your favorite part of the data platform might be. Is it something in SQL Server? Maybe a related technology? Perhaps something else that helps you work with data these days. Let us know what data platform thing is exciting to your career.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.