Voice of the DBA

Category: Editorial

  • Is the DBA Title Dying?

    Not is the DBA joib dying, though we could argue about that as well. Instead, I’m asking if the title of Database Administrator is going away. Are there going to be people that really want to send out resumes and apply to be Database Administrators beyond the next few years? As I look at my resume, I wonder if that’s a job or title I want to keep around.

    Think about that for a few minutes. We’re in the age of complex systems with cloud platforms, automated backup software, PowerShell and other scripting, Chef, Puppet, containers, self-service clones, automatic indexing, query store plan fixing, and more monitoring options than you would want to spend time evaluating. We have plenty of tools to assist, or takeover, much of the daily administration of database platforms. Even security can be mostly outsourced to AD, AAD, other directory services. While there is some level of what we might consider administration, is that the core of many people’s jobs? Will it still be in five years?

    I read a few of the posts from T-SQL Tuesday #100, including Adam Machanic and Brent Ozar. They both think the DBA is dying, and I tend to agree. I know that inertia slows change in many large organizations, and I’d bet there will be a delay just because of HR. After all, if you want to be a data reliability engineer or data professional, most organizations don’t list that job title and would have no idea how to hire you. Even if they loved your resume, they’d end up hiring you as a DBA or developer (developer III maybe) just because they like those nicely labeled buckets for the HR systems. They might not even know how to pay someone in some new role.

    However, the more I talk with people, the less they seem to be doing administration. They still tune queries, but often they’re helping do more database development or even database architecture than administration. They may to more reporting or ETL work, which can easily fall under development as well. Security is still a part of their jobs, but that doesn’t seem to change as often as it used to. Many people have moved to database roles and AD integration for authentication. Then security becomes more of a set-it-and-let-someone-else-manage-it. I even see Slack bots or other tools that let people self-service requests, and others quickly (or automatically) approve them.

    We still do some work, as things like HA/DR still aren’t quite a smooth as they could be, but that’s changing. Cloud services, and the evolving Azure Stack may mean that more and more of our work will be done by templates and patterns. We may have to create the template, but that feels more like architecture and less like basic administration. I expect at some point we’ll just link a grid of machines, on premises or in the cloud, and let people self-service their requests for systems. The “template” will let a service deploy HA databases across the machines as it sees fit, providing an address to connect to, implementing monitoring, and even solving many simple problems without human intervention. I expect that security and auditing features and capabilities will also grow rapidly, becoming more automated, or at least automatic, and request less administration.

    I used to joke I was a data janitor in many jobs. Just dealing with whatever situations people couldn’t, or wouldn’t, bother working on with databases. I often cleaned up messes made by developers that weren’t sure how to build a database that scaled beyond “their machine”. These days I think I’m no less of a data janitor, but I call us data professionals. It’s a better catch-all term, and certainly sounds more appropriate. We do our best to ensure the safety, accuracy, and availability of data in whatever way works best in our environment. We’ll run your T-SQL, R, python, machine learning, SSIS, SSRS, or whatever code for you, and audit the actions.

    I think that as we do that, we’ll do more development, cleansing, and analysis than actual administration.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 5.2MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • ETL or ELT

    I was reading a piece on ETL patterns and found it interesting that the author noted that one way to reduce the challenge of ETL processes keeping up with changing schemas is to move to an ELT process. That’s an interesting perspective, and I wonder what those who work with the relational model think. IS this a pattern you like?

    In a traditional ETL pattern, we get data from a source (Extract), manipulate it to suit the target schema (Transform) and then Load it into the target. The hassles here, and the reason this is hard, is that the transformations are often complex. We deal with ragged rows, various delimiters and terminators, and more. Lots of consultants have made quite a bit of money by building these transformational applications and processes and maintaining them as requirements change.

    The flip of the last two parts, moving from TL to LT, looks to defer some of the transform work until the data is in the destination. This means that the transformation could be a schema-on-read, as the article proposes. The idea here is that all data is ingested into the target system as is. This means that structured CSVs reside alongside spatial data and Word documents.  Then, once you have the data, you can transform it as needed, to suit whatever requirements exist at that time.

    Certainly this sounds appealing, but for many of us in organiztions, the challenge is still determining how to transform the data. This is often dependent on what the data is and how it can be used. We still need to do the work to understand how this data is structured and change something to make data available to the end user. Whether we do this in the middle or after the data is in a target system doesn’t matter. I know some products, like the MarkLogic one in the post and Azure Data Lake want to reduce must of the ETL effort, but I’m not sure that will work for really important data. For some, sure, we can collect it and sort out the transformations later. However, for useful data, data that the business wants regularly and quickly, most of us need to just get a flow working that will move data on a regular basis.

    I’ll admit that I often prefer to transform data inside of a SQL Server target, OLTP or DW database. The reason is that mass changes are often easier with T-SQL, working sets of data rather than row by row. I recognize that doesn’t always work, but I know that for most data that business people do find useful, I’d rather have an ETL process build sooner rather than later.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.8MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • A New New Microsoft

    Microsoft is changing again. Microsoft is reorganizing and the head of Windows, Terry Myerson, is leaving. The reorganization isn’t that surprising, as like most large companies, Microsoft usually does something each year. What is interesting is that they are moving away from their major divisions focusing on products. We’ve had divisions for Windows, Servers, Devices, etc. Now we move to Experiences and Devices as one major group. The other will be Cloud + AI Platform, which seems more product oriented, but still is rather amorphous. Gaming is still separate and there’s still a large Research engineering group.

    Why are they doing this? There’s some thoughts at ZDNet and Geekwire. It seems that the cloud is becoming the most profitable part of Microsoft, and I’d expect that we will continue to see more push for cloud and subscription type software. Along with the advances in AI, Microsoft seems to be hoping that more of us will start to do work using their infrastructure. That makes sense for some, but not for others.

    For those of us using the data platform, I do think that we ought to be thinking cloud first unless we already have substantial infrastructure and automation capabilities to quickly stand up new instances. As development and other departments look to get work done, especially around data analysis, we need to meet their needs or many people will start to use the easy to provision and use cloud services, especially those for machine learning and AI. Companies are entranced by new technology, even if we aren’t, and the media hype around ML and AI will put pressure on us to do better. We don’t need to learn R or Python, but we should be able to run those scripts against data if our business demands the capability.

    The GDPR and other security initiatives might seem to slow cloud deployment, but in many cases the cloud isn’t more or less secure than our own systems. Certainly we may have issues with privacy with the new US CLOUD Act, but for most of our businesses, that won’t really matter. The conflicts there are more around illegal activities, and if you are in that space, US based companies might not be your choice for hosting data.

    For many of us, we do need to understand how to build and enforce more secure applications, and I’d argue that Azure makes this as easy any any other system. Our recent move at SQLServerCentral to AWS was essentially the same as if we’d have moved to a VM at Azure, Google, or even back to the Redgate office. The systems look the same, it’s the cost, the firewall and networking, and things beyond the data platform that are just different. And that’s the key, they’re not necessarily better or worse, but different.

    As Microsoft changes, I think their code quality improves, though certainly their rapid pace means that people getting updates too quickly are on the bleeding edge. Their DevOps deployment, however, means that the bleeding edge usually doesn’t last tool long if the issues are severe.

    Steve Jones

     

  • dbatools is Growing Up

    The dbatools.io project is growing up rapidly. I went to check something the other day and I was amazed at the command list. This used to fit on one page. Now it’s not close. Here’s the first page:

    In fact, it doesn’t fit on two pages. Here’s the part below the fold.

    It’s crazy how quickly this project has grown, and how many cmdlets are available to help you not only migrate objects and settings between instances, but perform useful operations on your systems and test or change their configuration. I’m starting to think that this needs to be required for most people that manage systems. The ways that you can easily manage your systems in a programmatic manner has dramatically changed with this module.

    I was surprised the other day when I saw a blog post for a cmdlet I didn’t know existed. I went to try it on my system and it wasn’t available. I’d updated the module the previous week, but apparently there are new changes on a regular basis. I might need to update my module more often, and there’s a cmdlet to help me do that!

    Those of us that have a lot of experience with SQL Server might find some of these tasks easier in T-SQL. I certainly spend more time inside the platform than out of it, but there are definitely advantages to using PowerShell to enforce consistency and easily work with multiple machines is nice. These utilities are often like scripts or procedures I’ve written to handle simple tasks in a wrapper-like fashion. dbatools do that from PoSh, and allow me to easily string together disparate comands. I won’t give up T-SQL anytime soon, but I’ll do more with dbatools than I might do with PoSh otherwise.

    If you haven’t tried this project, you ought to. This will make it easier to learn PoSh and work with SQL Server. Maybe you’ll enjoy it enough to built your own module and contribute to the project. There are still holes, especially in the SSIS area, though I expect new cmdlets to appear any day.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.8MB) podcast or subscribe to the feed at iTunes and Libsyn.