Voice of the DBA

Category: Editorial

  • How Do You Experiment?

    One of the things that DevOps asks software developers to do is experiment. Try new ideas out, get feedback quickly, and then choose how to grow or stop your experiment. This is great for features, and it works well for application software.

    The general flow for this is to talk to customers, and then decide what to build. In some sense, this can work, but as I heard at the DOES Summit recently, if Henry Ford had asked his early on customers what to build, they’d have asked for a faster horse.

    Customers are limited by their current experience. This includes not only end users, but for us database pros, the developers that build software. When they want to experiment, they often need some backing from the database to store information and query it.

    If we want to help enable experiments, and allow our software to evolve, there are two things we need to deal with in experiments. One is schema changes, either through new data buckets in tables, or programmable objects, such as views, functions, and procedures Adding these, or removing them when experiments aren’t useful, can be cumbersome and difficult. It’s amazing how quickly we create dependencies and how slow we are to remove them.

    The other area is in ensuring that we properly or appropriately, handle resource usage. Do we go back and tune queries, or restructure the way that we’ve indexed items to ensure that our system works optimally? Some tuning can be done early, and should be, but some requires some feedback to understand query patterns or data loads.

    Today, I’m wondering how, or if, you experiment in database work. What works for you, or what doesn’t? Or do you hate the idea of experiments in the database world and want more specification up front? Let me know with a comment.

    Steve Jones

    Note: Podcasts are suspended for a week as I deal with the PASS Summit.

  • Extreme SQL

    I have made a career of working with SQL and databases. Usually I’ve looked for interesting companies and people, but I’ve avoided extreme situations. For me, that often is very large, or very real time environments. I once declined a job for a 13TB database on SQL Server 6.5. My suspicion is that job would have taken me away from my young children and wife far too often.

    Facebook has a lot of users, and a lot of queries they run. With over 1billion daily users and hundreds of TBs of daily uploads, they really need strong databases. While they have multiple databases, and that includes SQL ones, they have struggled with analytic queries in the past. They started using Presto as a solution, an open source query engine for running analytic queries against data in different storage locations like RDBMSes or in something like Hive/HDFS. This sounds like what Polybase does for SQL Server.

    The problem with any engine at Facebook’s scale is the load. While they like Presto, they needed to make it work better. They initially built a caching layer that required users to build ETL jobs to load data into SSDs attached to the Presto cluster. However, they outgrew this and ended up turning to a distributed file system called Alluxio.

    The article linked above talks a bit about how this works, and allows users to query petabytes of data. Most of us have users that often don’t qualify their queries completely, so we expect that some queries that might need to scan 100GB end up reading much more until the users tune them appropriately.

    The thing I found interesting in here is that some queries were taking up to 10s, which users found unacceptable. The move to Alluxio gave them a 30-50% boost, which doesn’t sound like a lot. 5-7s over 10 isn’t a great savings to me. The reduction in reads, is impressive, which is good, but I wonder to what expect there is some management and tuning needed here to ensure the cache works well.

    I have no desire to work on these extreme systems, but I am glad someone does. The lessons and tricks learned here often trickle down to improve the daily performance many of us see in our smaller systems. I think that the Hyperscale work Microsoft is doing, and the Big Data Clusters, are fascinating ways of organizing SQL Server based servers, and some of that tech will likely trickle down and help us continue to improve our smaller systems’ performance over time.

    Steve Jones

    Note: Podcasts are suspended for a week as I deal with the PASS Summit.

  • The Pace of Data Platform Change

    I was watching Vicky Harp in the 2020 October GroupBy conference recently talking about her challenges of working with the SQL Server platform as it’s grown. It’s a good keynote, with Vicky noting that she started with SQL Server 2000 and the journey to today is incredible. There’s a great view of her world at the 10:00 mark (after the start).

    She had this quote, which I found really thoughtful. This is something I think is true and something that many of us don’t like.

    “Usually growth comes at the expense of the previous comfort of safety.” – Josh Waitzkin, The Art of Learning.

    What’s your response? Run, fight? Embrace change? Resist doing anything different? I think many of us want to think that we easily embrace change, but think about the last time someone wanted to change something at work? Reorg, new protocol, etc. Did you resist and think it was silly or go along and give it a chance? For many people, it’s the former.

    To be fair, it is for me as well, and I think many people that have some success in their career often want to stick with the things they are experienced in. That’s not necessarily a problem, but it is worth investigating and embracing some new things, just to see if they might be better.

    The data platform is certainly one of those technologies in my life that has changed dramatically, and the pace is sometimes overwhelming. At this point, it’s hard to keep up, and hard to understand sometimes if new tech is better or worse. I’ve started to try and assume there is some good reason why Microsoft makes some changes, and then experiment, test, and evaluate the tech.

    Not just once when something is released or I encounter it, but by also watching what others do and then learning where they’ve had success or failure. ADF is one of those areas where I initially dismissed it as a poor port of SSIS, but I’ve come to appreciate some of the ways in which this is an improvement to many flows, especially with hybrid workflows.

    The data platform is an exciting place to work these days, and I hope you embrace some of the changes and see where a new technology might improve your environment. Of course, lots of traditional features out there work very well, and it’s not worth changing just to change. Make sure there is value and an improvement in some way, beyond you just enjoying working with something shiny and new.

    Steve Jones

    Note: Podcasts are suspended for a week as I deal with the PASS Summit.

  • All In to the Cloud

    I was listening to the fall 2020 GroupBy conference recently and heard Gethyn Ellis note that he wasn’t aware of any companies that were over some age (a decade?), and that had made a 100% move to the cloud. The comment caught my eye, not because I haven’t felt the same way, but because I’d just heard from a company that surprised me.

    During the DevOps Enterprise Summit recently, one of the keynotes was from Capital One. They are a 26 year old company that is focused on financial services, and they said that this year they are 100% in the cloud. They recently shut down their last data center. For a highly regulated company, and one that’s not a startup, that is impressive.

    Almost all customers that I talk with are looking at the cloud in some sense. They might not be looking to move their biggest or most important applications, or even a majority, but they are often looking to hedge their bets, while they continue depreciating or using their on-premises assets for some period of time.

    Not so with Capital One. They have heavily embraced DevOps, flexibility and rapid change in a way that had them open their 8th data center in 2014, only to pivot and look to eliminate that part of their infrastructure. They believe there is sufficient capability, reliability, and perhaps most importantly, security, in the cloud providers. They’ve embraced cloud-first, after an initial lift-and-shift approach for their thousands of systems.

    I don’t know that most companies need to move in this direction, but this does show it is possible, and no matter what your concerns are, there are models of enterprises that have found ways to take advantage of the cloud and that you can look to emulate..

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.