Voice of the DBA

Category: Editorial

  • Brush Up on Your ETL Skills

    Many of us that work with data will find requests and demands to import or export data at some point. Plenty of us have regular processes that perform these actions, and we may regularly troubleshoot or enhance these activities. In fact, I know some people have a full time, or nearly full time, position just dealing with ETL operations.

    Working with data in disparate formats and the myriad of inconsistencies even when formats are known is a challenge. Integration Services is a useful tool, but many us find that we need to pre or post process data separate from a simple import or export. Some of us may prefer using T-SQL or other languages, such as R or Python, to process data rather than programming SSIS. It seems that I often find that every client wants a slightly different format or change to their data that a simple query export won’t handle.

    These days, as we add in Machine Learning and other downstream processing activities, it seems that there is more and more of a need to process data beyond imports and exports. After all, it seems that the majority of the time in any ML project is spent preparing and transforming data. In addition, in Article 15 of the GDPR, there is language that notes a data subject has the right to request a copy of the data relating to them when it is being processed by an organization. I don’t know how often someone will want to get data about themselves or their organization, but I’m sure it will happen more than it happens today.

    I think this means I’ll need to brush up on ETL skills, perhaps to ensure I can easily extract out a copy of an individual’s data. In fact, I probably should compile some scripts now to ensure I can let someone know what we information keep at SQLServerCentral that would fall under GDPR. I think it’s just email addresses, but I could be wrong.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.9MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Virtual Data

    Yesterday I republished an editorial from 2014 for the holiday. The topic was production subsets of data, which has been something that many data professionals have struggled with for years. Many of us have built scripts to delete, change, obfuscate, or alter production restores as a way of providing useful, but manageable development database sets. Or maybe it’s just some of us. I’m sure more than a few of us have given up on this task and just restored production databases in entirety to test and development systems.

    I changed over my career to become a fan of additively building a known dataset rather than deleting extra data. I advocate adding rows from production (properly masked/obfuscated) and maintaining this set over time as requirements change. However, this isn’t without it’s own administrative headaches. I think it’s easier, but this does require commitment from everyone to keep going over time. It’s certainly better than each developer adding their own 10 rows of data to a table for testing.

    A year ago, Redgate released SQL Clone, designed to solve some of these issues. Once an image is created, new databases for test and development and be provisioned in seconds. I found this to be an amazing product that really changes how I develop against databases, though it does require me to stop getting caught up trying to undo changes or manage a single database. Instead, I need to ensure I am saving code to version control and then build the habit to drop and rebuild a baseline database.

    As we’ve worked on SQL Clone, I’ve found that there are lots of companies that offer similar ways of virtualizing your data, giving you access to large, production scale systems in seconds. Data masking, obfuscation, and more are features, with some vendors requiring specific hardware. Others, like Red Gate, have software add-ons (Data Masker).  All of these products cost money, which can be an issue for many organizations, but I’m glad that this technology is growing and advancing. With GDPR and other draft legislation, many of us need to take better care of our data and build more secure architectures.

    Containers are another interesting way to virtualiza data, though they don’t solve the scale issues. If you can work with a smaller data set, and maintain that, then containers might provide a fantastic way for you to learn to build, teardown, and rebuild databases in seconds.

    The world of databases hasn’t changed a lot in some ways across my career, but in others, I’m amazed. Data virtualization is one of these areas, and if you haven’t trialed the technology, maybe you should give it a whirl this year.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.6MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Template Configuration

    One of my goals this year is to really spend some time learning more about Extended Events (XE). I’m somewhat embarassed that almost a decade after their introduction in SQL Server 2008, I have a fairly rudimentary grasp of the system. I’ve watched talks from Jonathan Kehayias and Erin Stellato and others. I recognize the value in a lightweight system, but for some reason I haven’t done much with XE.

    As part of my learning, I’m going back to the basics with videos, articles, and documentation. In one section, I ran across the templates and how they can be used to greatly simplify the creation of new sessions. I’m a bit fan of templates, and I love the ones I use for T-SQL in SQL Prompt. I know templates are incredibly useful in plenty of other areas.

    XE is more complex system. The verbiage, the complexity of filtering, and the variety of targets make this a very flexible system, but also one that can overwhelm people. When do you use the histogram target v the ring buffer? What events make the most sense to solve or diagnose your system?

    Today I wonder if some of you out there can share some knowledge. Are there built in templates that you find useful for certain situations? Have you created your own that speed up your analysis of an issue? If so, it would be great if you could share some code and explain why you find certain session settings useful. If you find some templates to be problematic, perhaps share that.

    As I’m learning, I think that it’s likely I’ll have my own set of events and settings that I lean on heavily. Since I can save these on an instance, and not just a workstation, that’s a huge improvement over trace. These sessions can be shared with other DBAs in my organization, which is helpful and handy. Maybe one of you will give me a new template that I can add to my toolbox.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.7MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Muddle Through

    I used to work in the restaurant business. I got started after my first year at college, landing a job waiting tables in a new hotel. It was eye opening for me to see just how stressful and difficult that job can be. However, I wanted to earn money and jumped at opportunities. When we needed room service waiters to pick up shifts and elevators didn’t work, I carried trays up staircases, taking advantage of the opportunity. When a bartender didn’t show up, I volunteered to take the lunch shift. I had no idea where things were in the bar, I was 19 and didn’t know how to make drinks, and I’d make less money, but it was an opportunity that I knew would pay off at college.

    It did, and at the next three jobs, I wound up me getting hired, starting work, and having either no one to train me that day or a very busy shift where I was mostly on my own to survive. I had to muddle through and learn on the fly. The ability to do that, without panicking  being overwhelmed, or giving up has served me in quite a few positions since then.

    When I started working for various companies as a developer or DBA, I found myself in similar situations. Problems would arise, often the day or week I started, and I’d have to solve them. Usually with DBA positions,  I was the only one there, so I couldn’t depend on anyone else. It was  a good thing as I often found that sysadmins or developers were not managing or configuring databases in an efficient way. As I gained experience, I could make more and more of a difference earlier on at each organization.

    Kevin Feasel wrote a bit about his experiences with Lucerne (near the bottom), muddling through the need to write queries. I’ve seen similar stories from other friends working with SSIS, SSRS, Redis, Azure, and more. They don’t know a lot, but they dig in and learn, making mistakes, but getting tasks done for their employer.

    The ability to work through adversity, have some confidence, learn quickly, and be effective are valuable skills. Those data professionals that can do so often find more opportunities, challenges, they grow their skills, and get more compensation. Those that find reasons to avoid learning, that lack confidence in their ability to find a way to solve a problem, or are unwilling to tackle challenges often stagnate a bit. I’d like to think there are more of the former than latter in this business, but I constantly seem to find people that just look to repeat the same work they’ve done over and over for a long time.

    It can be mentally difficult to start a project using technology with which you have little familiarity. It can be disconcerting to have someone ask you a question that you can’t answer because you’ve barely begun to learn. That ability to muddle through, to keep learning, accepting that you don’t have answers but can find them, knowing that some of your answers will be wrong and you’ll need to backtrack. That ability to muddle through will serve you well.

    I’d urge you to develop it.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.0MB) podcast or subscribe to the feed at iTunes and Libsyn.