Voice of the DBA

Tag: software development

  • Is the Database a Series of Microservices?

    Not too long ago I was explaining to a group of developers a few reasons why database development and deployment is more challenging than application work. As I was talking about the different objects and the dependencies between them, an analogy occurred to me.

    The database is like a series of micro-services.

    We have these objects that are compiled and built separately, but with dependencies between them. A view depends on tables. Stored procedures and functions might depend on tables or the reverse might be true. Each object essentially has an API, which is the structure of the table or other object. Each could be developed and deployed independently, and there could be conflicts that might be noticed at run time due to deferred name resolution.

    Just like micro-services. In the case of application micro-services, each has an API and exists independently of others. They can be deployed, upgraded, and altered separately. If there isn’t coordination with other micro-services, there could be problems with how the entire application works.

    Relational database development in teams is always challenging, precisely because each item can be independently build and compiled. Unlike a C# or Java compiler, all of your code isn’t evaluated at once. While you can use a technique such as schema binding to enforce some checks across objects, these are crude mechanisms that interfere with a lot of development changes, especially those where we are trying to approach zero downtime deployments.

    There is also power in this independence, allowing us to make small changes over time. If we structure a series of deployments well, we can achieve very close to zero downtime changes, though usually at the expense of extra space and processing for a period of time.

    As I think about each table, view, procedure, and function being a separate service, I can more easily understand the dependencies that exist between them. I realize there is a need for close communication with other teams. Developers know that they need to coordinate changes across teams when their service API changes, and they can apply the same idea to the database. They can also make some changes without worry, as we can ensure backwards compatibility in many cases.

    A lot of the challenges in database development feel less daunting when we re-frame them. Once we stop viewing the database as a single thing and realize that each object is independent and has a contract, implicit or explicit, with other objects, we can more easily plan on how to make changes in a way that minimizes the impact on other teams.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Making Software Decisions

    I work for a company that builds software. At Redgate, we are constantly debating what to build and how, and more importantly, how to structure something that is easy for our customers to use. There’s a lot that goes into the process of building software, way more than I thought was important a decade ago. Today, even when I don’t like the decision or the final result in a product, I know our process often produces good results.

    Recently Netflix started a new series on how they make decisions for their software. It’s interesting in that they note that of all the ways they can make decisions, most of these involve few people. This leads towards their premise of using lots of A/B experimentation for helping to make decisions, and I look forward to future parts of the series that might explain the weight that they give to the results of A/B testing versus the opinions of developers and management.

    How do you decide what decision makes the most sense for your business? I often see technology professionals being very passionate about how or why something should be implemented. I also think that sometimes we lend more weight to the passion of the arguer, rather than the merits of the particular thing being examined. It’s easy to give in and go with the group, or not care, about many software decisions.

    For me, I’ve learned to try and give feedback, justify this with some rationale, and then trust the process. No matter how “right” I think I am, we are a team, and if the decision is made to go a different way, I support that. I’ve also learned that building software involves a lot of tradeoffs, especially with regard to the prioritization of work. There’s always more work to do than can be completed by our staff, no matter what the size of the team.

    I find the process of building commercial software to be fascinating and I’ve learned a lot in the last few years about how to make the process successful. I’ve also learned what Netflix talks about. A small group has a very limited viewpoint and it’s worth gathering more data whenever possible to ensure you make a decision that has a positive impact on your software.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • New Old Architectures

    Many years ago I was starting in a software development role using dBase and Clipper. I needed to upgrade a DOS based dBase program to Clipper to take advantage of some graphical libraries. Around this time, we were also moving many clients machines from DOS to Windows 3.1, giving them a GUI experience. Also, to the dismay of the management, access to Solitaire. One of our early jobs was to remove that game and Minesweeper from installs.

    In any case, I started to experiment with compilers that would allow Clipper programs to run in Windows as native apps. This meant moving to a more event-driven style of programming instead of the procedural way that DOS programs worked. This change was a paradigm shift for me and I spent many hours at home experimenting and trying to understand event-driven programming.

    It’s much easier to build a Windows app now, with many frameworks and libraries that handle the low-level events. However, the idea of using event-driven programming can be important in many distributed applications. When we work on the request-response model, it’s easy for a failure or problem to create a poor experience for the user, especially if there’s an intermediate layer between the client and the data. Certainly, there are retry and error handling techniques, but far too few developers implement them in a comprehensive way.

    I saw an article that introduces the idea of event-driven architectures as something you would want for microservices applications. It’s a good read that helps you gain an understanding of how this architecture would work. I’d recommend this to data professionals, as I expect more of us will encounter developers wanting to implement microservices as a way of magically making your software better. They might even want to implement a broker and queue system in SQL Server.

    There are many ways in which you might implement this type of system, and many technology choices, but an understanding of the overall concepts might help you guide developers to choosing an architecture that makes your life easier rather than harder. Event-driven architecture and microservices aren’t magically going to make your application run better. They could easily create bottlenecks that cause your phone to ring and ticket queue to grow.

    It’s helpful to learn about the things developers want to try and build. They get enamored by new technologies, some of which are amazing. Some aren’t, however, and some knowledge can help you point out potential problems and influence decisions. We all want better software, and as data professionals, we want to have intelligent conversations about how to get better software built.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Patterns and Potential Problems

    I saw a post recently from a developer that needed to refactor and rename a table in a live system. The post describes a pattern for doing so and gives the steps taken, though not the actual code. I like the pattern overall, and I think it can work well in many situations. It’s for a PostgreSQL table, so I don’t know what restrictions might be different from SQL Server, but this type of pattern can work for SQL Server as well.

    It also could be problematic. Using the famous “it depends”, there could be issues with this pattern, depending on your workload and how your application is structured. The triggers in use could also be an issue in some environments, as they create an additional load.

    The biggest concern I have with this pattern is the copying of the data. Likely this is something that always works on a developer’s machine with a few dozen or even hundreds of rows of data. If this is millions, or tens of millions, the copy could end up taking substantial time. There is also the issue of changing data, with data being added or changed in the table, separately from being copied out. Depending on your locking and concurrency schemes, you could miss data.

    Or you could just lock the table and cause issues for clients. Both things that might not show up in developer testing. The lesson here is that changes to big tables need to be tested in a big way.

    This isn’t to say the pattern is bad, but that you should be aware of the potential pitfalls and then develop mitigation strategies. One way to get around the data issue might be choosing a way of copying over new or changed data after the initial data movement, or maybe even a cleanup load later after the new table is online. There are many possible ways to mitigate issues, if you take some time to think about the potential issues and then come up with a solution.

    When we are looking to make changes to our system, patterns are important to help us and others adopt the processes that work well. As we find and develop patterns, we need to ensure that we understand the strengths and weaknesses, and choose what’s best for us, with mitigation strategies to get around the potential problems. There often isn’t a perfect solution, and ensuring you and your team understand the limits of your chosen approach, helps ensure that we deploy code that not only works, but is deployed without causing issues.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.