Voice of the DBA

Tag: software development

  • Making Software Decisions

    I work for a company that builds software. At Redgate, we are constantly debating what to build and how, and more importantly, how to structure something that is easy for our customers to use. There’s a lot that goes into the process of building software, way more than I thought was important a decade ago. Today, even when I don’t like the decision or the final result in a product, I know our process often produces good results.

    Recently Netflix started a new series on how they make decisions for their software. It’s interesting in that they note that of all the ways they can make decisions, most of these involve few people. This leads towards their premise of using lots of A/B experimentation for helping to make decisions, and I look forward to future parts of the series that might explain the weight that they give to the results of A/B testing versus the opinions of developers and management.

    How do you decide what decision makes the most sense for your business? I often see technology professionals being very passionate about how or why something should be implemented. I also think that sometimes we lend more weight to the passion of the arguer, rather than the merits of the particular thing being examined. It’s easy to give in and go with the group, or not care, about many software decisions.

    For me, I’ve learned to try and give feedback, justify this with some rationale, and then trust the process. No matter how “right” I think I am, we are a team, and if the decision is made to go a different way, I support that. I’ve also learned that building software involves a lot of tradeoffs, especially with regard to the prioritization of work. There’s always more work to do than can be completed by our staff, no matter what the size of the team.

    I find the process of building commercial software to be fascinating and I’ve learned a lot in the last few years about how to make the process successful. I’ve also learned what Netflix talks about. A small group has a very limited viewpoint and it’s worth gathering more data whenever possible to ensure you make a decision that has a positive impact on your software.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • New Old Architectures

    Many years ago I was starting in a software development role using dBase and Clipper. I needed to upgrade a DOS based dBase program to Clipper to take advantage of some graphical libraries. Around this time, we were also moving many clients machines from DOS to Windows 3.1, giving them a GUI experience. Also, to the dismay of the management, access to Solitaire. One of our early jobs was to remove that game and Minesweeper from installs.

    In any case, I started to experiment with compilers that would allow Clipper programs to run in Windows as native apps. This meant moving to a more event-driven style of programming instead of the procedural way that DOS programs worked. This change was a paradigm shift for me and I spent many hours at home experimenting and trying to understand event-driven programming.

    It’s much easier to build a Windows app now, with many frameworks and libraries that handle the low-level events. However, the idea of using event-driven programming can be important in many distributed applications. When we work on the request-response model, it’s easy for a failure or problem to create a poor experience for the user, especially if there’s an intermediate layer between the client and the data. Certainly, there are retry and error handling techniques, but far too few developers implement them in a comprehensive way.

    I saw an article that introduces the idea of event-driven architectures as something you would want for microservices applications. It’s a good read that helps you gain an understanding of how this architecture would work. I’d recommend this to data professionals, as I expect more of us will encounter developers wanting to implement microservices as a way of magically making your software better. They might even want to implement a broker and queue system in SQL Server.

    There are many ways in which you might implement this type of system, and many technology choices, but an understanding of the overall concepts might help you guide developers to choosing an architecture that makes your life easier rather than harder. Event-driven architecture and microservices aren’t magically going to make your application run better. They could easily create bottlenecks that cause your phone to ring and ticket queue to grow.

    It’s helpful to learn about the things developers want to try and build. They get enamored by new technologies, some of which are amazing. Some aren’t, however, and some knowledge can help you point out potential problems and influence decisions. We all want better software, and as data professionals, we want to have intelligent conversations about how to get better software built.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Patterns and Potential Problems

    I saw a post recently from a developer that needed to refactor and rename a table in a live system. The post describes a pattern for doing so and gives the steps taken, though not the actual code. I like the pattern overall, and I think it can work well in many situations. It’s for a PostgreSQL table, so I don’t know what restrictions might be different from SQL Server, but this type of pattern can work for SQL Server as well.

    It also could be problematic. Using the famous “it depends”, there could be issues with this pattern, depending on your workload and how your application is structured. The triggers in use could also be an issue in some environments, as they create an additional load.

    The biggest concern I have with this pattern is the copying of the data. Likely this is something that always works on a developer’s machine with a few dozen or even hundreds of rows of data. If this is millions, or tens of millions, the copy could end up taking substantial time. There is also the issue of changing data, with data being added or changed in the table, separately from being copied out. Depending on your locking and concurrency schemes, you could miss data.

    Or you could just lock the table and cause issues for clients. Both things that might not show up in developer testing. The lesson here is that changes to big tables need to be tested in a big way.

    This isn’t to say the pattern is bad, but that you should be aware of the potential pitfalls and then develop mitigation strategies. One way to get around the data issue might be choosing a way of copying over new or changed data after the initial data movement, or maybe even a cleanup load later after the new table is online. There are many possible ways to mitigate issues, if you take some time to think about the potential issues and then come up with a solution.

    When we are looking to make changes to our system, patterns are important to help us and others adopt the processes that work well. As we find and develop patterns, we need to ensure that we understand the strengths and weaknesses, and choose what’s best for us, with mitigation strategies to get around the potential problems. There often isn’t a perfect solution, and ensuring you and your team understand the limits of your chosen approach, helps ensure that we deploy code that not only works, but is deployed without causing issues.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Eyes Wide Open

    Not many of us work in startup environments, but many of us do work with new databases that are created for new applications. These might be carefully designed, thrown together, or your database might be constructed by an ORM. In any case, I find many people make decisions and write database code for today, solving the problems that they see in front of them. they often do this with little data and a single system. They might have their eyes on using some new technology, and they decide on a data strategy without really considering what they will need later.

    That seems to be what happened with Expensify, which started in the financial industry. Their system had requirements for low response times, multiple locations, and detailed logging. This required a robust database architecture, which turned out to be helpful when the company pivoted to a new business model. Their CTO talks about some of the problems he sees with startups making database decisions. I think many of these lessons are helpful for all organizations that are trying to ensure their database can grow and meet their needs.

    I think one of the most common problems I see is that developers and leaders get enamored with new technology. There is a lot of promise in some of the platforms and designs that are being put forth. Some are even proving themselves in high-profile situations, but not all. For most of us, however, we aren’t going to be solving the same problems in the same way. As the article notes, we’re not Google, but we’re also no Uber, Facebook, or Spotify. Choosing to mimic their choices because of their success doesn’t necessarily map to our business model. I find no shortage of companies that struggle to adopt some new platform because they built a proof-of-concept and assumed the way the system works with small amounts of data. This becomes an issue later with the moderate or large amounts that they have over time.

    I also see companies creating complexity, with the chance that they will need to deal with many petabytes or exabytes of data at some point. Face it, most of us will barely deal with terabytes of data in any particular system. We ought to plan for a high-performing system at that scale, not worry about a future that will not likely come. At the same time, we aren’t going to be dealing with megabytes of data, so if your developers only test on MBs, they are going to miss problems.

    I like the advice to go into your decisions with your eyes wide open. Don’t copy others, and realistically think about what you will need. I believe that engaging a data professional early is helpful. Developers do some amazing things when they build software, but so often the majority of them don’t really think about the challenges of a database system. They don’t consider low response times or ensuring there are HA and DR (two separate things) strategies. They also forget about the challenges of aggregation and reporting lots of data. Most humans work with a few rows of data at a time, which is what developers do on their systems. When you need to aggregate things, or all your customers are generating a workload, that’s when a data professional can help ensure you’ve properly indexed entities and planned for a demanding workload.

    I do like the common sense advice that most startups won’t outgrow a relatively modest single database server. Many applications might not as well, but that doesn’t mean you can put all your eggs in that one server basket. Make sure if it dies that you have multiple people that can recover it and ensure your system is quickly running in another place. There are different ways to handle this, but engage someone that knows your platform and have them ensure you have some staff, operations or developers, that understand how HA and DR work in your environment.

    Lastly, be secure. I really like the idea of always using stored procedures. I know this becomes a pain for developers, who now write code in two places, but this really helps you ensure better security, and maybe more importantly, ensures you can tune one part of your code regularly, the database side, without impacting the other side.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.