Voice of the DBA

Tag: software development

  • Distributed Monoliths

    I was watching a video called Microservices are Technical Debt. In it, the person being interviewed said that a lot of people really have a distributed monolith. That caught my eye since I’ve worked with a number of customers who are trying to adopt microservice architectures for their applications. I think this is less a performance/scaling choice than a reworking of their software development teams, and I’m not sure they will end up with a better system.

    What is a distributed monolith? I am not an expert, but this appears to be a place where all the services still depend on each other. For example, I might have a service getting user profile info that an app calls, with another service getting previous orders, and a third service that returns inventory. In a monolith, if any of these are down, the others don’t work. In a distributed monolith, these might be built independently, but perhaps the core app/web page still requires all these to be working to show the user something.

    That’s how my gym app seems to work sometimes. It can’t get my profile and the list of classes and the things I’ve registered for, so it hangs. This is very annoying when all I really need is my ID QR code, which never changes, and is validated by another system.

    In a microservices architecture, if any of these were down, the others would still be working, and the app would continue functioning. If inventory is down, I can still update my profile. If previous orders are working, I can view them even if the current inventory doesn’t work.

    Things are loosely coupled for the end user and for the development teams. Not for just one of those groups.

    I’m not sold on microservices, especially when they include databases. There is a lot of power, and efficiency, from being able to pull together different data sets in a database efficiently rahter than doing so in an application, which might require different API calls, separate db connections, and the delays that come with returning more data than needed to let the client join it together. At the same time, I can see how separate services could potentially scale much higher over time.

    The thing that I tend to see is that most of us never need to really scale as high as the Google/Facebook/Spotify/Netflix/etc. organizations that espouse these microservice architectures. They have different problem domains and different needs than most of us. Even Uber has realized that one size doesn’t fit all and is moving some things from microservices to macro ones. I’m not against microservices, but they need to be used in appropriate situations, not all of them.

    What most of us need is a little more help from Brent or Erik or Paul or someone else in the form of a little consulting that tunes our code to run more efficiently. And if we use them, we should learn a few things from them so that we write better code the next time.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Separate Reads

    Recently I was watching a presentation on how to scale performance in your SQL Server environment and one of the suggestions was setting up Availability Groups (AGs) and having read-intent connections that would query the secondary and not the primary. It’s not a bad idea, and the SQL Native Client (and other drivers) support this and make it easy to implement.

    The pattern of using multiple connections in an application, one for reads and one for writes, has been suggested often. However, in practice, I’ve rarely seen this work. Apparently having a connection variable, named dbConn, for writes and a second one, named dbConnReadOnly, for reads is too complex for most developers or teams.

    Or maybe the idea of having to pick the right access point is a human problem? I’ve seen no shortage of problems in restaurants when we have specific “in” and “out” doors. Lots of people go through the wrong one and we end up with plates and food on the floor. Even broken noses or fingers at times. Perhaps I shouldn’t pick on software developers too much.

    How many of you use two connections from apps? Meaning, do you think about reads and writes in separate connections. Even if you read and write from the same database, this can be a nice practice that future proofs code. It’s a small change, but it gives you room to grow if you get a read replica for analytics or reporting.

    Of course, you could take it too far with different connections for different “services”, aiming for a microservice-style architecture. We could have dbUser for user stuff and dbOrders for the business side, and other connections for other services. I wouldn’t do that, as I think many of us will get confused, and we’ll often be doing two different type of service things in the same code. If I need something from a customer to write an order, do I have two connections in my method? I could, but I bet lots of developers would try to re-use a single one.

    Plus, if developers get into trouble with two connections, then what will they do with 5 or more? There are lots of ORMs that might even support this, or if they do, not make this easy to code.

    I’ve always liked the idea of separating reads and writes, but maybe the better solution is using one connection whenever we have simple CRUD work and another one for any sort of complex querying or reporting. That would make more sense as I suspect many of us will eventually offload reporting or analytics in some way to another system. A Delta Lakehouse of some sort seems likely if the current trend takes hold in more organizations.

    Let me know today if you used (or have tried) different connections for reads and writes.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • What Metrics Do You Collect?

    One of the hot terms in software these days is observability. There are a few definitions (Splunk, RadixWeb), but essentially this is the insight into how your software runs and performs using metrics, logs, traces, etc. In DevOps, we do this with an eye toward improving performance and identifying the root cause of issues. The focus is slightly different from monitoring, where we often focus more on resources and health. We need both, but often in trying to improve software and the behavior for users, developers need observability. Infrastructure people responding to acute issues and looking to ensure we have the capacity, availability, and other x-bilities, that need monitoring.

    Today I’m wondering if you collect a variety of types of metrics for your software that might tell you how your system is running. What things are important to you in order to better serve your clients? If you’re a DBA/sysadmin, what is important to you? If you are a developer, are there different types of data you want?

    Certainly, you might collect various resource measures (CPU, IO, reads, etc.), but there are many more things. There are logs, which could include the SQL Server error log, but I’d hope that you had a more in-depth way of measuring the activity on your system. Do you have custom xEvent traces running? Can you collect application logs easily when you’re looking at issues? Do you spend time trying to solve chronic issues? Do you look for potential future problems?

    Most software applications should include some sort of basic logging of major functions, but I would hope that there are various levels available. If problems are reported or noticed, can you increase the detail of logging? Can you correlate this with logs from different systems, such as the database? Can I get execution data that matches calls and trace down the potential issues that users are experiencing?

    I know that we have problems in applications. Some (many?) of these are data-related, which might be easy or hard to trace down. That often depends if the data changes too quickly or is static enough for someone to investigate a report. Some errors are logical code errors, which might indicate a lack of testing early in the software process, but we ought to be able to determine this quickly from logs. If there are performance issues, and we have a lot of these, how easily can we verify a problem?

    Let us know what types of metrics help you solve issues. I certainly think that you should have some sort of monitoring and observability system in place that helps you dive deep into the database, especially concerning execution plans at the time of the issue. The situation can change quickly inside a database, so capturing data regularly is important. If there is something you wish you had, let us know as well. Maybe someone else will have a neat solution for you.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Polyglot Persistence

    I’ve seen the term polyglot persistence floating around Redgate a bit recently in the marketing department. I haven’t really seen this term anywhere, and I wonder if you have. If you have, drop a comment.

    The definition is based on the polyglot programming, where you write an app using multiple programming languages. I don’t know a lot of people who do that, and if you do, let me know. I guess some do, since I’ll see things like HTML+C#, or maybe something like Blazor + C#.

    In any case, the idea seems to be that you use multiple databases to satisfy your requirements. A few examples might be:

    • SQL Server for most data, Redis to cache shopping carts
    • Oracle for most data, ElasticSearch for full-text searching
    • PostgreSQL for profiles and leaderboards, MongoDB for real-time game actions

    I’ve tended to showcase NoSQL + Relational here, but I have seen a few people using two relational stores or two NoSQL stores because it suits their needs. The really common additions (for me) are some type of NoSQL store for performance with more stable, long term data in relational.

    I don’t think this covers the relational for OLTP and something else for OLAP/warehousing, as that seems to be an ETL/ELT/transfer of data over time for a different purpose rather than for the same application, but I guess it could cover that.

    I think Polyglot Persistence is a good idea, especially when you have complex requirements that aren’t easily solved with one database, even when you have in-memory or search/graph features in your database. Often those don’t perform as well as specialized systems, though there is a level of complexity and the challenge of ELT/ETL’ing the data to/from the second system.

    Evaluate carefully, PoC, and test at scale before you decide this is worth the trade.