Voice of the DBA

Category: Editorial

  • Big Data or Small Data

    I went to San Francisco for Small Data SF, a conference sponsored by Mother Duck. The premise of the event was that smaller sets of data are both very useful and prevalent. The manifesto speaks to me, as I am a big fan of smaller sets of data for sure. I also think that most of the time we can use less data than we think we need, especially when it’s recent data. That often is more relevant and we end up with contorted queries that try to weight new or old data differently to reflect this. Maybe the best line for me is this one:

    Bigger data has an opportunity cost: Time.

    I think time is a very valuable commodity and large sets of data can slow you down. There’s also the chance that looking at too much data starts to blur the lines of understanding. We may start to miss information in our dataset, or we may find people arguing about different things the data means, because we have so much data that we can find support for any position somewhere in the vast sea of numbers, strings, and dates.

    Big data also has a real cost in resources, often money. One of the examples was from the organizer, who once gave a demo on stage, querying a PB of data.  That’s impressive, and lots of us would want to be able to query our very-large-but-less-than-PB-sized data in minutes. However, the thing that wasn’t disclosed in the demo was the query cost over USD$5k.

    I’ve heard from a number of customers and speakers that most people don’t have big data. Most of us have 100s-of-GB-sized working sets of data, sometimes with TB-sized archives in the same database that slow everything down. If we could easily extract out the useful data, we could query those hundreds of GB more efficiently.

    This is especially true in the era of small devices that can handle something close to a TB of data in a small form factor. With some of the columnar systems that compress data, a TB of raw data might be substantially compressed in Parquet files or an analysis system like DuckDB. In that case, we might realistically search and analyze 1TB of data on a laptop.

    I know that big data is relative, but many of us face challenges with data sizes and query performance. I know lots of you embrace the challenge and see working with TB (or larger) systems as a badge of honor. I also know the reality is that most of us struggle to separate our archive data from current working data in our systems. However, if we could, would most of you want to work with smaller data sets or do you enjoy large ones? I know which way I lean.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Learning to Grind

    When I was younger, I had a variety of jobs, but in most of the positions I had to work hard for stretches. Really hard, as in more than 8 hours a day or 40 hours a week. Often as I was starting a new position, it took some time for me to develop some understanding, some skill, and some muscle memory. In some jobs, especially in restaurants, I also had to build the physical skills to be on my feet for many hours.

    In technology, I’ve often found myself unsure of how to approach a new position, aware I had knowledge gaps about how things worked, and often, I was naïve or ignorant of some piece of technology my employer used. Even at jobs where I started as a developer or DBA on a known platform (ASP or SQL Server), I sometimes encountered some aspects of the technology that I hadn’t used in the past (like clustering).

    In those situations, something I learned from my parents and a few youth coaches came to mind. I needed to bust my butt to be successful. The lessons I learned weren’t expressed so politely, but they boiled down to putting in extra time and focus, and continuing on that path until I was competent in the eyes of someone else, usually my boss.

    I’ve encountered many people in the last decade that have much to learn. I’ve met far too many that didn’t understand their environments as well as I’d expect them to as a manager. I have encountered far too many people who wish they could be more skilled in some way, but they haven’t made a commitment do the work to further that wish. I’ve met far too few people who are working to improve themselves on a regular basis.

    How do we teach people to grind away at something to improve themselves?

    I don’t know. I’ve tried to motivate people, I try to give them examples, I’ve tried to provide suggestions. It seems that many people have lost the drive to invest in themselves to prepare for the future. Too many want their boss to train them and then re-train them when they don’t use a skill and forget it. Or they want their time in a position to count as experience. Or they want their boss to give them time out of their 40 hours, without having to make their own investment of time at night or on weekends.

    Skill and experience don’t magically appear. They take work. They take grinding away, making mistakes, achieving small successes, taking a step backward, then driving forward in new ways. It’s effort, and it’s time. Read any story about a person who’s achieved success and you’ll find tales of study, work, practice on their own time.

    If you want something different in your career, or in life, you have to work at that thing. Make a plan, but then work at it. Give up some leisure time. Not all, but some. Give up something fun to achieve something else later. Learn to sharpen your saw, polish your craft, grow your marketability, whatever you want to call it.

    Just start doing it.

    If you want to read a few examples, I have a short series of posts on grinding away at life.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • The AI/Human Spectrum

    I was asked this question recently: is it more likely that AI will replace humans or assist them in their work?

    It’s a good question. If you think about the way AI is being hyped in 2024, many people think AI is, or will soon be, replacing people and we need less of them in work. I guess the simplified view is that AI can do the jobs of many people, but I’m not sure the world is that simple. What I think is more likely is that AI becomes a lever that assists a few people in getting more work done and potentially replacing other, less knowledgable humans.

    Maybe it’s the ultimate, do-more-with-less pressure that the management in many organizations places on workers that has people looking to AI to help. Get more done this year, but we’re not adding staff, and potentially we’re removing a few staffers. Maybe AI can do your job?

    AI can be a lever, and I do think that there are tedious tasks that we might have AI take on for us. We’ll need more of an agent/proxy approach to AI systems, as now I can ask AI to do some things that relate to text/images, but not actually do work for me. I want an AI to actually set my Out-of-Office for me, not give me instructions on how to do it or write the message for respondents.

    The current Generative AI/LLM systems aren’t really smart or intelligent, but they do process vast amounts of data and mimic the responses that other humans might give. If you work in an area that can benefit from that type of interaction, maybe an AI works well as a lever and lets you get more done on any given day. For some jobs.

    However, getting more done might not be enough. If I pair program with someone else and they write a bunch of poorly performing code, it takes me time to judge the quality of what they’ve written and then additional time to fix it. Getting more done in that situation can be a burden because the additional code produced requires additional rework. I might get less work done in some cases if the code is low-quality and I use a lot of time to rewrite or improve the code.

    However, if I am tackling simple tedious tasks, perhaps basic CRUD work in an application, maybe an AI can generate enough SQL, web, C#, etc. code that the job is done quicker. Maybe not at the most efficient level, but how many of you think the code for your internal applications is amazing? Is it good enough? Can an AI do “good-enough” work?

    As with a lot of dramatic changes to technology, I find myself going back and forth on the value produced. Some days I think the tech is amazing and some days I think it’s akin to the stuff I shovel out of horse stalls. AI is in the same boat with me, and while I think there is potential, I know that there are also downfalls and potential detractions from its widespread use.

    Certainly, the need to evaluate and judge quality is a challenge with AI, which leans me towards the lever that assists talented humans and replaces less talented ones. The other issue is cost. The LLMs are expensive, and use a lot of compute power. I’ve seen some smaller models, perhaps tailored with RAG or other methods of refining (and limiting) their use will overcome that, but who knows. The current models, however, cost something to run and someone is going to have to pay for that. Is there enough ROI to do that?

    Lastly, trust. Can we really trust an AI to give us accurate responses, or even perform work on our behalf? We have that problem with other humans now, but they work slowly compared to a computer. Can you imagine the problems that a rogue computer system could create with access to change things in the real world?

    Answer my question today. Are AIs more likely to assist or replace people?

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Continuous Learning

    It seems there’s quite a dichotomy in the technology workforce. On one hand I hear about the Great Resignation where many employees are leaving their jobs because of RTO (return to office) mandates or some other dissatisfaction with their job. On the other, I’ve seen quite a few people who were laid off and are struggling to find new positions. On the third hand, there is no shortage of companies who report they are struggling to find and hire talented people for some positions.

    I don’t quite know what to think, but I do know that employment seems to be harder to come by. If you are looking for a job, I believe you need your own sustainable learning plan. If your organization can’t find qualified talented staff, then the organization definitely needs a continuous learning plan.

    I saw a piece on creating a continuous learning culture, which seems more suited for managers than technology workers, but there are a few good pieces of advice. First, set some goals. I write about this often, and I’ve seen others recommend this as well. Think about where you could improve your skills, and where you see demand for skills, and then start working backward on a plan to gain those skills. Think about where you want to be in 6 months or a year and build a plan to get there.

    Another good piece of advice is to learn within your flow of work, which often means spending a little more time deepening or broadening your skills in an area where you are already working. If you’re writing a query, analyze the plan and reads and see if you could tune it to run more efficiently. Read about those skills and practice them, which deepens your skills. If you don’t know much about query tuning, then broaden your experience and learn about a new operator, new index, new hint, and experiment. Then you have a wider view of what possibilities there are in query tuning.

    More importantly, build the habit of curiosity and experimentation, which is what will help you drive continuous learning throughout your day. It’s good to look at new things, but don’t forget to sharpen your existing skills, practice being better at the work you already do, and find a balance that ensures you are growing while still meeting your other commitments, both at work and away from work.

    There’s an old joke that if you train people they might leave, but if you don’t train them, they’ll stay. For organizations struggling with talent, embrace some investment in current employees, at least those who have potential. Grow them and you might find that the person you’ve been struggling to hire is already working for you. They just needed a little boost.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.