Voice of the DBA

Category: Editorial

  • Can You Become a More Productive Engineer?

    The short answer is of course, most of us can learn and improve our skills to become better developers, engineers, DBAs, etc. While we might not be able to become the 10x engineer that many aspire to be, we can certainly become a better employee inside of an organization.

    There’s a piece on becoming a more effective engineer, which is actually titled know how your org works. It a piece from an engineer that started with a tweet: The text was:You can either complain and pontificate on Twitter on how the tech industry *should* ideally work, or you can learn how your org *really* works and what’s rewarded, and optimize for that. Or quit and find another job. This might sound cynical – but it’s what it is.

    That sounds a little harsh, but the reality of how your org works or is structured or interacts is a reality. We all have hindsight to look back and wish someone (including us) had written code better. We might be sure if we could change one thing, or add/remove someone else, or make some other change, then things would be better. We might feel that there is a simple solution. Those things might be true, but they aren’t the reality of the situation.

    Learning to be more effective an engineer in your situation does involve learning to work within the reality of your position in your organization. You need to learn to work well with others, to understand what needs to be completed from the perspective of you and your group, and what the organization places importance on getting done. That last one might not be what the organization expresses, as there can be a big disconnect between words and actions.

    You also need to learn how to deal with uncertainty. Often when we build applications, we don’t have all the information, and we can’t get it. We can’t necessarily count on everyone we work with to have the same shared understanding or work at the same pace. We have to learn what levels of trust we can extend, work within ambiguity and be prepared to change our plans in response to how things work.

    In other words, most organizations have a lot of chaos and to become a better engineer you must learn to navigate the chaos and still get the things done that others notice.

    The piece has a section that soft skills are hard skills, which is true. When I see people who are very effective engineers, they can be the brilliant, 10x engineer, but that’s rare. Usually it’s the engineers that have strong soft skills in both working with others and understanding the power hierarchy in the organization that succeed.

    Of course, some organizations are full of bureaucracy, favoritism, and disfunction that it’s hard for anyone to become effective. It’s amazing these places continue to exist, but they do. I’ve been in a few and I’ve left as soon as possible, which is my advice to you. Don’t work in a dysfunctional environment. If you can avoid it. If you can’t, then work to leave.

    Effectiveness is part being better at coding (or tuning or admin or whatever), but it’s also part learning to work well with the rest of the organization.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • The AI Budget Assistant

    I saw this article about AI helping IT leaders plan their budgets, and I had visions of certain departments getting cut, especially data-related ones. My concern is that a lot of what a DBA does can be acting as insurance or supporting others and not necessarily doing things which are easily visible or that might show up in reports, praise in Slack/Teams, or in other places an AI might be prompted to look. If an IT leader has to decide what’s important for the AI to look at, is data included?

    Of course, it’s entirely possible the AI would look at all the performance problems and want to spend more budget on the data side. Fingers crossed for that interpretation.

    The idea of an AI looking at a large amount of current data in various spreadsheets, reports, goals, requirements, usage, and more to synthesize a budget makes sense. This is the type of thing many AI-type technologies do well, finding patterns and summarizing the data in a way that assists a human in producing a final report of where to spend money.

    Just like any report produced by a human, I would hope that someone reviews the results and verifies the analysis makes sense. However, in reality, I suspect a lot of human reports are passed along as-is, with only a cursory glance. The world of AI might make this worse as we can produce larger and more detailed reports.

    The funny thing might be that whoever has to approve the budget might let their GenAI model summarize the budget for accuracy. In that case, do we need humans involved? I can see some managers trying to reduce the staff that might have internal knowledge and replacing them with GenAIs. Will we just have AIs producing things other AIs read and summarize?

    Most of us know that garbage in/garbage out is a constant problem in the world of data and reports. Judging data quality and ensuring the accurate (and complete) data is fed into any system is something that humans need to be a part of. An AI model might assist, but humans have context and knowledge that can be hard to craft into a prompt. After all, this might be the training data for your model, and I’m not sure general models will learn well without handholding from humans, especially in specialized areas like budgets.

    I’ve had to build budgets, and while I don’t think it’s hard, it’s time-consuming. There is a lot of drudgery, and I think an AI can help here. However, I’d want to audit and check the data and ensure that my helper is summarizing the data in a way that makes sense to me. Even then, what I’d most like to do is let the AI give me a few suggestions of future costs and let me fine-tune those before I present them to anyone else. After all, if we make large errors, I suspect I’ll feel the pain of failure more than my AI assistant.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • The Types of Changes

    I would guess that most of you reading this piece are database professionals in some sense. Either you work on databases as a developer of some sort (app and/or database) or manage them as a DBA or sysadmin. In working with databases, I’m sure that you need to alter object structures. Maybe you perform a lot of changes, or maybe you just alter the schema occasionally, In either case, you need to make decisions on which changes to make.

    I would guess that the most common changes are altering stored procs/views/functions. Often these are to handle new or updated business rules. In those cases, how do you ensure you haven’t added regressions? I would hope you had a set of test scripts to verify that your results are correct. If I were very optimistic, I’d hope that you had a set of automated tests, such as those you could write with tsqlt or the Microsoft Unit Test Framework.

    I’m sure many of you add new columns to tables. What are these types of changes? Capturing new data? Splitting existing columns or de-normalizing data? In these cases, are you re-examining the data model to ensure the changes make sense?

    Maybe even more importantly, if you are capturing new data, do you consider vertical partitions for the table? This can be a good way to separate out data in columns that isn’t queried as often. Despite most recommendations, lots of developers will still SELECT * from tables, which can flood the buffer cache with data that isn’t often needed by an application. Vertical partitions can reduce the impact of these types of queries.

    For those of you that add new data, do you consider using different data types for your new data, or do you have a standard string/numeric/date type that you use? SQL Server (and other platforms) often have a variety of different data types that can save space, while still accommodating the requirements for a change. As an example, many people use datetime as a standard type when they just need a date. In this case, we could save 5 bytes per row by choosing a more appropriate type.

    If you add status values or flags, have you considered bit storage for your values? Whether you use the bit datatype or bits in an int, you can dramatically save space in each row. Newer versions of SQL Server include T-SQL functions to help you work with bits, though I would hope most developers can handle simple AND/OR logic against an integer to test if a particular bit is set.

    Most software under active development has a need to continually alter and modify the database to meet new requirements. While lots of us have experience in making these changes, and do so often, once we make a change it often becomes enshrined in our production system for years. I would hope that we feel confident in refactoring our code as needed, but the reality of what I often see is that people are loathe to change the database for fear of breaking our software. Even when the refactoring actually improves the performance of the application.

    I’m curious what types of changes you often make, and how you go about deciding on the specifics of the change. Let us know about your approach in the discussion. I’m especially interested in how you might handle the various options for changes. Do you assume defaults or are you explicit in how you structure your ALTER code?

    You might have a novel way of examining requests and implementing changes that can help others in learning to ensure they make better changes in their own system.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • The Managed Cloud Database Options

    There are many, many choices for cloud database services these days. I would hope everyone is aware of the various IaaS options in public clouds with EC2, Azure VMs, GCP Compute Engine, and others. These are often the easiest way to move your workload, but you’ve really just moved a VM from one place to another (likely more expensive) place.

    For managed databases, there are lots of choices, but you might not be aware of your options. I ran across an article that discusses the various flavors of managed databases in the big three public clouds for SQL Server. In the piece, there is a section that talks about when a managed database makes sense. I like that it discloses the development on a managed service is expensive.

    The problem I have is that I know lots of companies that struggle when they don’t have a development environment that matches product. Invariably developers will use something in their local SQL Server/MySQL/Oracle/PostgreSQL database that doesn’t work in a managed service and causes no shortage of pain. Containers can help, but they’re not always available.

    I won’t delve into details, but the article lightly looks at RDS (Amazon), Azure SQL DB and Managed Instance (Microsoft), and Cloud SQL (Google). There are pros and cons for all of them, but to me, the Google option is interesting. It gives you an instance (with SSIS/SSRS) and a proxy, but it does have limit you to instance restores. Same for RDS, which is a limitation I’d be worried about. Often I have an issue with a database, really often with just a table. Restoring a whole instance instead of a db might be a delay I wouldn’t want to deal with. I hope this changes over time, as not recovering a database from an instance is a big hole to me.

    The comparison of hardware resources and performance seems to show Azure has the highest capacities, but as the article notes, if you approach the maximums, the cost is very high. I tend to agree there, and I think this is one of the things that gets lost when companies consider the cloud for workloads. They look at the maximums with an eye towards growth if they need it. Rarely, however, do they compute the increased cost if they’ve under-provisioned resources. There’s also the concern that trying to match CPU/RAM/disk from on-premises to the cloud isn’t easy. This often isn’t the same type of comparison that most of are used to with local machines.

    The main thing I find too many people forgetting is that moving to the cloud is a new architecture. You likely need to rework/refactor/rewrite your application to work better. Part of that work is reducing the amount and frequency of data queries. Without changing those things, you likely will end up spending a lot more money than you expect. Possibly without getting the workload response you expect.

    The cloud can work well for your application and database, but it’s not easy and it’s not quick.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.