Voice of the DBA

Tag: databases

  • There Are a Lot of Databases

    I was reading Andy Pavlo’s end-of-year review of the database world. He’s done this for a number of years, and there are links to previous recaps in the piece. He is an associate computer science professor at Carnegie Mellon University, working on quite a few database-related projects. In the review, he tends to track the database world from the perspective of business success and money. There are certainly parts of it that discuss technical changes, but my overall impression is more about the business and usage success than it is about the way database systems work.

    The main thing that struck me after reading the review was how many database systems there are in the world. I hadn’t heard of any of these: RaptorDB, TigerData, Tembo, StormDB, Translattice, FerretDB, DocDB, SpiralDB, Tantivy, SkySQL, HeavyDB, and more. I’m sure I missed listing some I didn’t recognize, and quite a few of these are PostgreSQL-based systems, but still, that’s a lot of database systems that exist and are having success.

    Last year, I ran into someone who worked at a company that had implemented ArangoDB for the software their company sold. This system had something to do with tracking parts and managing schematics for machines, which is a great place to use a graph database. I asked them why they didn’t pick a more well-known and used graph database like Neo4j. He answered that cost was a big reason, but if Arango failed to wrok, they felt could port their data over to another platform. He did mention that training new people was a challenge, which I believe is a good reason to stick with more mainstream systems. However, I understand that people placing bets on less well-known technologies is how the popularity of those platforms grows.

    As a side note, I keep confusing ArangoDB with AvacadoDB. Maybe because I like guacamole.

    If I look at DB-Engines, I see lots of platforms I recognize and a few I don’t, but overall this is a long list. Some you could argue aren’t really database platforms, but these are platforms people report they are using. There are 429 ranked, which is quite a few. I’m not sure there are that many different models of cars being produced in the US each year.

    Many of these are specialized platforms and might be suitable or even preferred in certain situations. I wonder if any of you reading this are running Hazelcast or Presto. Or anything else unusual. If you are, why? What’s better about one of these systems than the top 5-10 in any category?

    As I look around I realize there are so many databases available to choose from. Perhaps it’s just me, but I prefer choosing from a small list rather than a huge one. Do you feel the same way?

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • T-SQL Tuesday #182: Personal Integrity

    It’s time for the first T-SQL Tuesday blog of 2025, with an invite from the first non-founder to host a party, Rob Farley. I reached out to Rob and he graciously agreed to host. His invitation this month is on integrity, leaning towards data integrity.

    I still manage the T-SQL Tuesday list, and I’m always looking for hosts. I have a few scheduled for 2025, but I can use more. If you’re interested in hosting, hit me up at one of these places:

    While data integrity is important, I think personal integrity really matters as well.

    Integrity at Work

    I used to work for a large corporation, which I won’t name here. This was a 10,000+ person organization and I helped manage part of the data group for the company. I had a number of reports, 10 or 11, that handled different aspects of production operations across a variety of database platforms.

    It was a busy job, and our environment was far from stable. As is the case, things get cobbled together, become popular, and then limp along for years. This happens in small companies and large ones, but in large ones, I think it continues because when there are problems, we can throw people at the problem. We could throw resources and fix chronic issues, but that wasn’t the case at this company.

    At the time I worked at this company, we had a stacked ranking system for reviews.  Microsoft used to do this (and stopped), but think about ranking everyone in your team from 1-5, 1 being underperforming and 5 being outstanding. The “stacked” part comes from the need to have a certain number of 2s, 3s, and 4s. 1s and 5s were rare (5s more rare). Essentially a 1 meant you were on a performance plan and on the way to termination.

    I had to rank my staff, who worked hard and kept our systems running, despite lots of incidents. We performed some large migrations and upgrades of systems where my staff worked multiple weekends, with no comp time, just meals covered.

    My first integrity stand was that even though I wasn’t officially allowed to give comp time, I did, working through my staff with extra days off that were unrecorded anywhere. This might be more of a violation, and if my boss’s boss had caught wind of this, I would have been terminated. However,  I felt my staff deserved something, as a few of these long weekends weren’t adding business value; they were things executives wanted to do for optics.

    Near the end of the year, we had to rank our workers for annual reviews. I had most of my staff at 4s, with a couple 3s. I put in one person as a 5 for some great work they did. Justifying a 5 is hard and I thought I had a good case.

    In a meeting of our IT department, the VP of Operations told us that we had too many 4s and that each director (1 level above me) would get an allocation of rankings and that each manager had to work to fit their staff inside that allocation. Anyone ranked a 1 would be outside the allocation.

    My second integrity stand was to argue with my boss on why my staff deserved their rankings. According to their job descriptions and performance, I’d ranked then well, even though his quota meant he wanted me to move a couple 4s to 3s and 1-2 people to a 2 ranking. I refused, arguing with other managers whose staff I had seen underperform through the year. I told a few others they needed to absorb the 2s and 3s and not the data team.

    I lost that battle and had some uncomfortable reviews (and the accompanying bonus/salary numbers), but I acted like an adult and told people I had ranked them a level higher and upper management lowered their ranking, not because of performance, but because of a quota.

    As we went through the next year, management decided they needed to cut costs. They had managers compile a lot of numbers that were sent to outsourcing companies in an RFP to essentially remove IT from the company’s books. In a meeting, our CTO tried to spin this as good for workers as the winning company would hire our IT staff. The framing was that this outsourcing would save the company money.

    My third integrity stand was when I questioned the way this worked in a large meeting of IT management and eventually got the CTO to admit that:

    • a) everyone would have to re-apply for their job
    • b) they might not get the same salary (higher or lower, you decide which is more likely)
    • c) not everyone would be hired

    This might work out for some people, but likely those who got jobs at a new company were risking their salary and workload in a new situation. I questioned this as being good for the company overall as we would likely lose lots of knowledge.

    I left shortly thereafter, voluntarily, but I’m sure I would have been let go at a layoff that occurred near the end of that year.

    I hated that job in many ways and was glad to eventually leave. I decided I would never work for a stack ranking company again.

  • Polyglot Persistence

    I’ve seen the term polyglot persistence floating around Redgate a bit recently in the marketing department. I haven’t really seen this term anywhere, and I wonder if you have. If you have, drop a comment.

    The definition is based on the polyglot programming, where you write an app using multiple programming languages. I don’t know a lot of people who do that, and if you do, let me know. I guess some do, since I’ll see things like HTML+C#, or maybe something like Blazor + C#.

    In any case, the idea seems to be that you use multiple databases to satisfy your requirements. A few examples might be:

    • SQL Server for most data, Redis to cache shopping carts
    • Oracle for most data, ElasticSearch for full-text searching
    • PostgreSQL for profiles and leaderboards, MongoDB for real-time game actions

    I’ve tended to showcase NoSQL + Relational here, but I have seen a few people using two relational stores or two NoSQL stores because it suits their needs. The really common additions (for me) are some type of NoSQL store for performance with more stable, long term data in relational.

    I don’t think this covers the relational for OLTP and something else for OLAP/warehousing, as that seems to be an ETL/ELT/transfer of data over time for a different purpose rather than for the same application, but I guess it could cover that.

    I think Polyglot Persistence is a good idea, especially when you have complex requirements that aren’t easily solved with one database, even when you have in-memory or search/graph features in your database. Often those don’t perform as well as specialized systems, though there is a level of complexity and the challenge of ELT/ETL’ing the data to/from the second system.

    Evaluate carefully, PoC, and test at scale before you decide this is worth the trade.

  • Navigating the Database Landscape

    The title of our keynote session at the Redgate Summit in Atlanta is Navigating the Database Landscape, and I’ll be delivering part of the talk, along with Grant Fritchey and Kathi Kellenberger today, Mar 13. This is based on the State of Database Landscape Survey results, as well as our experience working with customers and implementing DevOps solutions over the last decade. The talk was mostly written by others, but as I rehearsed the session, I found myself wondering about how I’d approach my job if we returned to being a DBA or developer.

    When working in technology today, there are many challenges outside of actually learning about any of the particular products, languages, platforms, etc. We have the politics of working with others, ongoing work, emergency requests outside of channels, random questions asked by others, code reviews, and probably a few other things I’m forgetting, all outside of learning any new skills. While I consider myself a lifelong learner, I know that finding time (and energy) to acquire the basics of any new technology is challenging.

    At the same time, while working in any size estate, it seems that someone always wants to add a new tool, platform, language, service, or database to the environment. It’s great we have choices, but it seems like sometimes every technologist wants to just use something new rather than work within the areas we have experience. Early in my career, it was rare to find more than 1 or 2 database platforms in a company. Now we have lots, often seemingly just added because one person watched a talk or video and thinks it would solve all our problems in this particular instance.

    Working in an enterprise of any size likely means there are multiple database platforms in use. While you might only be in charge of 1 or 2 today, who knows when someone will call you as the “database expert” and expect you to configure Redis or troubleshoot ElasticSearch. I’ve had friends in this position, and I’ve had people come ask me to figure out MySQL, DB2, BTrieve, Lucerne, and more in my career. It’s a challenge, and it’s also stressful because I want to do a good job, even if I don’t know what I’m doing. Fortunately Internet search, contacts among friends, and more have helped me usually solve an issue.

    The modern database landscape is likely to be more complex than ever, and with the advent of cloud services, we find there are lots of options that anyone in an organization might choose to use in production, and then ask you to support them. Our jobs are increasingly complex, both from the depth of things we need to know about a database to the breadth of different products and services that might enter our realm of responsibility. Navigating all this is a challenge, but if you rise to the occasion, there can be a lot of rewards.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.