Voice of the DBA

Category: Editorial

  • The Virtual Experience

    I just completed the 2020 DevOps Enterprise Summit a few weeks ago, and it was enlightening to me to see that so many people liked this experience. They appreciated not traveling, being able to drop in and out of sessions while near their home and family. I found that lots of people also liked the MVP Summit last spring, which was also virtual.

    The PASS Summit went virtual recently, and I went through an all day pre-con as well as a few sessions during the week. As many of you, I’m a bit screen and Zoom fatigued, and while I enjoy learning and seeing people passionate about a technology, I’m tired of seeing it inside these four walls.

    I worry that I’m in the minority. I recognize the value of the inclusion of those that don’t have the time or money to attend a conference live. I appreciate that the ability to watch when and where it fits into my schedule. I love that others can pause, rewind, get captions, and more that spread knowledge out.

    However, I think we’ve lost something, and certainly lost some of the collaborative, discussions and debates that we can have in live events. While I can reach out and discuss with anyone, it’s hard to discuss with just someone. Others can join in, and semi-private channels, and especially the times during breaks when I can get up and walk while talking, are things I miss in the virtual experience.

    If virtual events were spread out more, and not everything in my work world were virtual, perhaps I would feel differently, but for now, I don’t love the virtual experience, and feel it’s a pale shadow of a SQL Saturday, user group, or live conference. I hope that this isn’t something I do too often, but I suspect this will be a permanent part of our lives forever.

    Steve Jones

  • How Do You Experiment?

    One of the things that DevOps asks software developers to do is experiment. Try new ideas out, get feedback quickly, and then choose how to grow or stop your experiment. This is great for features, and it works well for application software.

    The general flow for this is to talk to customers, and then decide what to build. In some sense, this can work, but as I heard at the DOES Summit recently, if Henry Ford had asked his early on customers what to build, they’d have asked for a faster horse.

    Customers are limited by their current experience. This includes not only end users, but for us database pros, the developers that build software. When they want to experiment, they often need some backing from the database to store information and query it.

    If we want to help enable experiments, and allow our software to evolve, there are two things we need to deal with in experiments. One is schema changes, either through new data buckets in tables, or programmable objects, such as views, functions, and procedures Adding these, or removing them when experiments aren’t useful, can be cumbersome and difficult. It’s amazing how quickly we create dependencies and how slow we are to remove them.

    The other area is in ensuring that we properly or appropriately, handle resource usage. Do we go back and tune queries, or restructure the way that we’ve indexed items to ensure that our system works optimally? Some tuning can be done early, and should be, but some requires some feedback to understand query patterns or data loads.

    Today, I’m wondering how, or if, you experiment in database work. What works for you, or what doesn’t? Or do you hate the idea of experiments in the database world and want more specification up front? Let me know with a comment.

    Steve Jones

    Note: Podcasts are suspended for a week as I deal with the PASS Summit.

  • Extreme SQL

    I have made a career of working with SQL and databases. Usually I’ve looked for interesting companies and people, but I’ve avoided extreme situations. For me, that often is very large, or very real time environments. I once declined a job for a 13TB database on SQL Server 6.5. My suspicion is that job would have taken me away from my young children and wife far too often.

    Facebook has a lot of users, and a lot of queries they run. With over 1billion daily users and hundreds of TBs of daily uploads, they really need strong databases. While they have multiple databases, and that includes SQL ones, they have struggled with analytic queries in the past. They started using Presto as a solution, an open source query engine for running analytic queries against data in different storage locations like RDBMSes or in something like Hive/HDFS. This sounds like what Polybase does for SQL Server.

    The problem with any engine at Facebook’s scale is the load. While they like Presto, they needed to make it work better. They initially built a caching layer that required users to build ETL jobs to load data into SSDs attached to the Presto cluster. However, they outgrew this and ended up turning to a distributed file system called Alluxio.

    The article linked above talks a bit about how this works, and allows users to query petabytes of data. Most of us have users that often don’t qualify their queries completely, so we expect that some queries that might need to scan 100GB end up reading much more until the users tune them appropriately.

    The thing I found interesting in here is that some queries were taking up to 10s, which users found unacceptable. The move to Alluxio gave them a 30-50% boost, which doesn’t sound like a lot. 5-7s over 10 isn’t a great savings to me. The reduction in reads, is impressive, which is good, but I wonder to what expect there is some management and tuning needed here to ensure the cache works well.

    I have no desire to work on these extreme systems, but I am glad someone does. The lessons and tricks learned here often trickle down to improve the daily performance many of us see in our smaller systems. I think that the Hyperscale work Microsoft is doing, and the Big Data Clusters, are fascinating ways of organizing SQL Server based servers, and some of that tech will likely trickle down and help us continue to improve our smaller systems’ performance over time.

    Steve Jones

    Note: Podcasts are suspended for a week as I deal with the PASS Summit.

  • The Pace of Data Platform Change

    I was watching Vicky Harp in the 2020 October GroupBy conference recently talking about her challenges of working with the SQL Server platform as it’s grown. It’s a good keynote, with Vicky noting that she started with SQL Server 2000 and the journey to today is incredible. There’s a great view of her world at the 10:00 mark (after the start).

    She had this quote, which I found really thoughtful. This is something I think is true and something that many of us don’t like.

    “Usually growth comes at the expense of the previous comfort of safety.” – Josh Waitzkin, The Art of Learning.

    What’s your response? Run, fight? Embrace change? Resist doing anything different? I think many of us want to think that we easily embrace change, but think about the last time someone wanted to change something at work? Reorg, new protocol, etc. Did you resist and think it was silly or go along and give it a chance? For many people, it’s the former.

    To be fair, it is for me as well, and I think many people that have some success in their career often want to stick with the things they are experienced in. That’s not necessarily a problem, but it is worth investigating and embracing some new things, just to see if they might be better.

    The data platform is certainly one of those technologies in my life that has changed dramatically, and the pace is sometimes overwhelming. At this point, it’s hard to keep up, and hard to understand sometimes if new tech is better or worse. I’ve started to try and assume there is some good reason why Microsoft makes some changes, and then experiment, test, and evaluate the tech.

    Not just once when something is released or I encounter it, but by also watching what others do and then learning where they’ve had success or failure. ADF is one of those areas where I initially dismissed it as a poor port of SSIS, but I’ve come to appreciate some of the ways in which this is an improvement to many flows, especially with hybrid workflows.

    The data platform is an exciting place to work these days, and I hope you embrace some of the changes and see where a new technology might improve your environment. Of course, lots of traditional features out there work very well, and it’s not worth changing just to change. Make sure there is value and an improvement in some way, beyond you just enjoying working with something shiny and new.

    Steve Jones

    Note: Podcasts are suspended for a week as I deal with the PASS Summit.