Voice of the DBA

Tag: career

  • Batch, Streaming, and Relational Data

    This is part of a series on my preparation for the DP-900 exam. This is the Microsoft Azure Data Fundamentals, part of a number of certification paths. You can read various posts I’ve created as part of this learning experience.

    The first part of the DP-900 skills document has these items:

    • describe batch data
    • describe streaming data
    • describe the difference between batch and streaming data
    • describe the characteristics of relational data

    These are concepts that are important to this exam. I lightly blew these off when I started studying, but every other person with guides and the practices tests has lots of focus here. I’m glad I spent time here.

    This post covers these concepts a bit. Note, these are more ETL/analytic concepts, not really

    Batch Data

    Most of my career deals with batch data, meaning a bunch of data that arrives at once and is imported into a system. This is different than a connection and query submitted to an OLTP system. The general idea is:

    • Lots of data
    • Processed periodically
    • Latency doesn’t matter.

    Think these key words:

    • Not real-time
    • periodic
    • large/big/lots

    There is an MS Docs article on this. The general idea is that you want to think about a scheduled (or some periodic) processing of lots of data for a purpose.

    Examples of where batch is used.

    • Total up all hours worked last week for employees
    • Load and transform log files from all web servers each day
    • Import files from regional offices into a main database server

    In the analytics space, you’d be using Azure Data Factory (ADF), HD Insight (U-SQL, Hiuve, Pig, Spark), Azure Data Lake (ADLS).

    Streaming Data

    There is a course on this topic. When you think of streaming, think of these key words:

    • real-time
    • stream
    • data processed as soon as created
    • IoT
    • few transactions
    • monitoring or instant decision making

    Streaming is really about time series, about tumbling windows, about data like a stock ticker that you need to constantly and/or quickly process.

    Differences

    These items helped me:

    • Lots of data – Batch
    • Low latency – Stream
    • Long latency, latency doesn’t matter, periodic work – Batch
    • Small, constant sets of data – Stream

    Relational Data

    The workload here is that you are handling regular changes to data, lots of insert/update/deletes, for a business process. Really this means you are thinking some sort of CRUD application the gets and sends data to users in real time, but not with low latency issues. We are thinking a web server, a data entry business app, something that operates on time scales for humans, seconds. Not real time, IoT millisecond work.

  • Making Video Chat Better

    I ran across an article on Zoom fatigue and new tech that was written in July 2020. That month seems like ages ago, and at the same time, not too long ago as the world seems to have still been in some sort of suspended animation for much of 2020 and 2021. However, one thing that has remained since that time is a regular stream of Zoom and Teams meetings that I attend.

    I’ve been dealing with video meetings for over a decade. For most of the time I’ve worked for Redgate, I’ve had regular meetings every week with various people that are in Cambridge, UK. I’m in Denver,  US, and while I enjoy going to England and visiting the office, that’s a few times a year. Across the years, we’ve tried a number of platforms, and various different configurations, including a dedicated conference room with extra mics and sound-absorbing padding on the walls. None of this made for a great experience, though I have to say our adoption of Zoom across the last 2-3 years has been the best experience overall.

    Still, when I have multiple meetings in a day, I feel fatigued. Many people feel similarly, and that has not gone unnoticed. While we do try to cut down on some of the meetings as a company, they aren’t going away entirely for us, or likely, for your company. Some companies are going back to the office, at least part-time, but many employees are resisting, which means we need better tech. Something I hope is coming.

    The article I linked above talks about some of the changes taking place, with some newer tech being tried. I don’t know I’ll ever like video chat without some sort of VR-style view of a room or holograms, but perhaps something will come along that makes these meetings better. I’ve certainly found that having larger views of speakers is good, but when they rapidly switch from one person to another it feels like a 5-year old directing a live TV show. I want to turn off my video and just listen. Or end the call.

    One thing that I’ve found helps is I try hard to get more done through Slack and email, avoiding meetings that aren’t needed. The best way to combat something that doesn’t work well for me is to do less of it, so I work to minimize video, type more, and hope for more live face-to-face meetings a few times a year.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • DP-900 Preparation Guide

    This is part of a series on my preparation for the DP-900 exam. This is the Microsoft Azure Data Fundamentals, part of a number of certification paths. You can read various posts I’ve created as part of this learning experience.

    I was asked to pursue a few Azure exams as part of Redgate’s partnership with Microsoft. I’ve been slowly studying some concepts and working towards exams in spare time, but getting more focused in January. I made this a part of my goals for 2022, and it’s been an interesting experience to work towards this exam.

    I’m providing an overview of my resources, and I’ll add new posts for various detailed items I found interesting with links as well.

    Overview of Preparation

    The first thing I did was look over the exam page. This lets the reader know what is being covered as well as the purpose of the exam. There is a note that some things have changed as of Oct 2021, and there is a new PDF to look at with the various areas being tested.

    Download the PDF.

    That’s first, and then check the scope of the exam. There is a lot here, including, but not limited to:

    • data workloads
    • relational Azure services (including PostgreSQL and MySQL/MariaDB)
    • Synapse
    • security (firewall, auth)
    • deployment options
    • query tools
    • DDL and DML
    • non-relational data
    • Azure Storage, Azure Files, Azure Tables, Azure queues
    • Azure CosmosDB
    • analytic workloads, including HD Insight, Databricks, ADS, and Synapse
    • Power BI

    That’s daunting.

    I also looked over some guides from others:

    This seems like a lot, but since the exam can only cover so much, I didn’t expect to be too detailed in each area. However, you do need to know quite a bit about a number of different areas.

    I looked over these, trying to determine what I knew and didn’t know. These gave me hints about which places to spend a little time. I didn’t watch all of the Andrew Brown video, but I did look over the others.

    Prep Process

    I started looking for an overview of each area. Microsoft Learning has some free reading courses, and sandboxes for practice, that I used. I went through these learning paths:

    These each are tens of minutes, but there is a lot of information to absorb about each area. Quite a few of these areas I hadn’t spent much time working in, so I found myself dropping into the MS Docs to get a little more clarification of the area.

    I also took the MS sample question test to get an idea of what I actually knew. I learned that I had quite a few holes in small areas, though overall I knew most of what was being covered.

    I haven’t taken the exam yet, as I’ve been spending an hour every 2-3 days going over some concept and trying to learn a solid overview of what it consists of and how it might be used for data work. Lots to learn here, and the thing I do like is that this forces me to actually dig into topics at a slightly deeper level and helps me to understand a bit more about the options in Azure.

    It also helps me to have better conversations with customers, which is important for my job. I’m not an expert, but I have an idea how of what Azure Table Storage is v. CosmosDB and can understand why someone might choose one of the other in a conversation. At least I can talk basics and know where to look for more detailed information.

  • Are You a Good Engineer?

    I would guess that many of us think we’re good at our jobs, or at least that we do a good job. Whether you’re a system administrator or software engineer, I’m sure you think you earn the compensation you get.  I wonder if others in your company would see you this way? If they know what you do as a job, and understand the role, then you should hope they see you a good engineer/admin. Perhaps you should even ask a few people if they do.

    There is a good thread about software engineers, talking about some of the things that make a good developer. It’s from a longtime engineer, and the thread talks about the things that you might consider important in a good software developer.  Actually, I think most of these items would apply to admins and even other business people.

    The first item in the thread really caught my eye because I see many people taking requirements as facts and then working to meet them exactly, or complaining and trying to avoid meeting them exactly. It isn’t often that I find people really digging in and trying to be a partner with their clients. Too often we’re too accepting or too argumentative, without trying to find a solution that makes the entire system better.

    The second item about not being a jerk is also important. It took me some time to learn that, and I still find too many people that think being right, or more talented, makes up for poor behavior. I’m sure it does in some places, but none where I’d want to work. Instead, follow the other advice and build relationships with others. That’s important, especially over time in your career. I know plenty of people who get poor recommendations from peers, and that causes them issues when looking for a job in their local area.

    There are some other items, but the one I find especially important is work ethic. Often hires need to be taught about our environment, even if they are the 10x engineer. I’d rather have someone willing to learn, someone I like working with, and someone that isn’t going to be afraid to work hard and help improve things. I’ll take those skills over past “expertise” in most situations.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.