A Memorial and Grief fund has been set up, if you’d like to contribute.
Author: way0utwest
-
Is the DBA Title Dying?
Not is the DBA joib dying, though we could argue about that as well. Instead, I’m asking if the title of Database Administrator is going away. Are there going to be people that really want to send out resumes and apply to be Database Administrators beyond the next few years? As I look at my resume, I wonder if that’s a job or title I want to keep around.
Think about that for a few minutes. We’re in the age of complex systems with cloud platforms, automated backup software, PowerShell and other scripting, Chef, Puppet, containers, self-service clones, automatic indexing, query store plan fixing, and more monitoring options than you would want to spend time evaluating. We have plenty of tools to assist, or takeover, much of the daily administration of database platforms. Even security can be mostly outsourced to AD, AAD, other directory services. While there is some level of what we might consider administration, is that the core of many people’s jobs? Will it still be in five years?
I read a few of the posts from T-SQL Tuesday #100, including Adam Machanic and Brent Ozar. They both think the DBA is dying, and I tend to agree. I know that inertia slows change in many large organizations, and I’d bet there will be a delay just because of HR. After all, if you want to be a data reliability engineer or data professional, most organizations don’t list that job title and would have no idea how to hire you. Even if they loved your resume, they’d end up hiring you as a DBA or developer (developer III maybe) just because they like those nicely labeled buckets for the HR systems. They might not even know how to pay someone in some new role.
However, the more I talk with people, the less they seem to be doing administration. They still tune queries, but often they’re helping do more database development or even database architecture than administration. They may to more reporting or ETL work, which can easily fall under development as well. Security is still a part of their jobs, but that doesn’t seem to change as often as it used to. Many people have moved to database roles and AD integration for authentication. Then security becomes more of a set-it-and-let-someone-else-manage-it. I even see Slack bots or other tools that let people self-service requests, and others quickly (or automatically) approve them.
We still do some work, as things like HA/DR still aren’t quite a smooth as they could be, but that’s changing. Cloud services, and the evolving Azure Stack may mean that more and more of our work will be done by templates and patterns. We may have to create the template, but that feels more like architecture and less like basic administration. I expect at some point we’ll just link a grid of machines, on premises or in the cloud, and let people self-service their requests for systems. The “template” will let a service deploy HA databases across the machines as it sees fit, providing an address to connect to, implementing monitoring, and even solving many simple problems without human intervention. I expect that security and auditing features and capabilities will also grow rapidly, becoming more automated, or at least automatic, and request less administration.
I used to joke I was a data janitor in many jobs. Just dealing with whatever situations people couldn’t, or wouldn’t, bother working on with databases. I often cleaned up messes made by developers that weren’t sure how to build a database that scaled beyond “their machine”. These days I think I’m no less of a data janitor, but I call us data professionals. It’s a better catch-all term, and certainly sounds more appropriate. We do our best to ensure the safety, accuracy, and availability of data in whatever way works best in our environment. We’ll run your T-SQL, R, python, machine learning, SSIS, SSRS, or whatever code for you, and audit the actions.
I think that as we do that, we’ll do more development, cleansing, and analysis than actual administration.
Steve Jones
The Voice of the DBA Podcast
Listen to the MP3 Audio ( 5.2MB) podcast or subscribe to the feed at iTunes and Libsyn.
-
Date Fun with SQL Server
I saw a people discussing date formats on Twitter after PASS put out an advertisement for a webinar with a mm-dd-yyyy format. Plenty of people were annoyed and wanted to be sure that they realized that much of the world might misinterpret 3-2-2018 as Feb 3, as opposed to the US Mar 2 view.
That’s fair, and while some people noted that yyyy-mm-dd is the best format, I saw this tweet from Mladen Prajdic. It looks at the DATEFORMAT setting can cause issues.
Here are a few reproductions. On my US centric system, I ran this:
SET LANGUAGE ‘us_english’
GO
SET DATEFORMAT YMD
GO
DECLARE @dt DATETIME = ‘2016-03-02’;
SELECT yyyy = YEAR(@dt) ,
mm = MONTH(@dt) ,
dd = DAY(@dt);
GO
SET LANGUAGE ‘French’;
DECLARE @dt DATETIME = ‘2016-03-02’;
SELECT yyyy = YEAR(@dt) ,
mm = MONTH(@dt) ,
dd = DAY(@dt);
GO
SET LANGUAGE ‘French’;
DECLARE @dt DATETIME = ‘20160302’;
SELECT yyyy = YEAR(@dt) ,
mm = MONTH(@dt) ,
dd = DAY(@dt);
GO
SET LANGUAGE ‘us_english’
GO
SET DATEFORMAT YMDMy results:
Now, let’s change a few things. I’ll move to datetime2. If I do this, all three queries return 2016, 3, 2 for year, month, day. The same thing occurs with the DATE datatype.
If you examine Mladen’s test, you’ll see that without dashes things work fine. However, with dashes, the datetime datatype has issues. These can manifest themselves with both SET LANGUAGE and SET DATEFORMAT settings.
One thing to keep in mind is that SET LANGUAGE will automatically change the date formats to match that language, and you can override those with SET DATEFORMAT if needed.
The takeaway? First, use modern datatypes. Not worth using datetime and avoiding the “2” if there is potential for incorrect dates.
Second, avoid the dashes for now.
-
ETL or ELT
I was reading a piece on ETL patterns and found it interesting that the author noted that one way to reduce the challenge of ETL processes keeping up with changing schemas is to move to an ELT process. That’s an interesting perspective, and I wonder what those who work with the relational model think. IS this a pattern you like?
In a traditional ETL pattern, we get data from a source (Extract), manipulate it to suit the target schema (Transform) and then Load it into the target. The hassles here, and the reason this is hard, is that the transformations are often complex. We deal with ragged rows, various delimiters and terminators, and more. Lots of consultants have made quite a bit of money by building these transformational applications and processes and maintaining them as requirements change.
The flip of the last two parts, moving from TL to LT, looks to defer some of the transform work until the data is in the destination. This means that the transformation could be a schema-on-read, as the article proposes. The idea here is that all data is ingested into the target system as is. This means that structured CSVs reside alongside spatial data and Word documents. Then, once you have the data, you can transform it as needed, to suit whatever requirements exist at that time.
Certainly this sounds appealing, but for many of us in organiztions, the challenge is still determining how to transform the data. This is often dependent on what the data is and how it can be used. We still need to do the work to understand how this data is structured and change something to make data available to the end user. Whether we do this in the middle or after the data is in a target system doesn’t matter. I know some products, like the MarkLogic one in the post and Azure Data Lake want to reduce must of the ETL effort, but I’m not sure that will work for really important data. For some, sure, we can collect it and sort out the transformations later. However, for useful data, data that the business wants regularly and quickly, most of us need to just get a flow working that will move data on a regular basis.
I’ll admit that I often prefer to transform data inside of a SQL Server target, OLTP or DW database. The reason is that mass changes are often easier with T-SQL, working sets of data rather than row by row. I recognize that doesn’t always work, but I know that for most data that business people do find useful, I’d rather have an ETL process build sooner rather than later.
Steve Jones
The Voice of the DBA Podcast
Listen to the MP3 Audio ( 3.8MB) podcast or subscribe to the feed at iTunes and Libsyn.
