Voice of the DBA

Author: way0utwest

  • What Happened to Hadoop?

    Six or Seven years back, Hadoop was the big thing. It was going to solve our big data analytic needs, it would provide cheap storage and query power with commodity servers. More and more companies were going to be using it. Microsoft invested in HDInsight, SQL Server got Polybase to query data directly from HDFS. I was seeing the Hadoop elephant everywhere. I still remember popping into a few sessions at SQLBits to try and learn a bit more about how Hadoop worked.

    In the last couple years, Hadoop has somewhat dropped off the radar as “the thing” that most companies need to become data driven and deal with large amounts of unstructured data. I found this analysis that talks about why Hadoop hasn’t taken over the world.

    The short answer? Real time needs, cloud computing, and containers. I think really, though, the complexity of Hadoop became a problem. It was too hard for most companies to deal with, and with too few willing to invest in the large infrastructure needed and skills required to manage a system. I’d say the same thing about Kubernetes, but it’s evolving rapidly to become easier, and it’s dirt simple in the cloud. I suspect we’ll see more Kubernetes cloud deployments than on premises.

    The other issue is the Hadoop batches jobs, which isn’t what many organizations want. They already deal with, and complain about, plenty of relational batch jobs. Whether this is ETL to a warehouse, cube processing, or some other delay. They want queries on data in place, which is becoming more commonplace all the time.

    Of course, one other important point from the piece is something I believe. The relational database, or data warehouse, is not going away. It’s still important to many organizations, and it’s useful to handle lots of reporting. With the growth of the SQL Server platform, you might even do more AI/ML analysis on your data in place, without the need to move it to an HDFS platform.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • Setting 2020 Learning Goals

    I deliberately didn’t set any goals at the start of 2020 since I was taking six weeks off for a sabbatical. That’s done, and it’s time to look forward.

    I know I have a few things that will be important for work. Oracle, PostgreSQL, and MySQL will be on the list as we expand and try to sell more database development tools. We’re also working hard on improving SQL Monitor, so finding a way to improve me query tuning, analysis, and monitoring skills will exist.

    The goals I want to set here will be more personal goals that may relate to my career, but they’ll not necessarily be things I do at work. I’ll carve out some time elsewhere to get through these.

    Reading

    I like reading, and do a lot of us to escape and relax. I completed around 120 books last year, but almost all were fictional or non-technology books. I had hoped to get through 6 tech books, but that didn’t work.

    This year I’m going to scale back and tackle this in two ways. First, I want to get through at least 3 tech books that relate to specific coding or other skills. I’ve still got a couple from last year, including a Power BI one, so that’s one goal.

    The other one will be 2 career related tech books. I started Making Work Visible, and I need to finish that, along with another one.

    Goal:

    • 3 technical books
    • 2 non-tech, related books

    Projects

    I do a lot of demo stuff and build lots of PoC type things, but I need a couple projects. Specifically, I want to get projects done that will help at work, but will be useful in some way to me, or perhaps even others. With that in mind, and dovetailing slightly with those items above, I want to tackle these three projects and get them in a somewhat useable state:

    • A Power BI report that updates from a database
    • A mobile app reading data from somewhere
    • A website that showcases changes and data from a database.

    Not big projects, but getting something organized and relearning some tech about reporting and coding is important to me. I have a few rough ideas for which topics to use for these, but I want to try some basic PoC and see how things go.

    Checking In

    I think I need to check in monthly with these goals and start to see how I’m making progress. I’ll set reminders weekly to spend time here and then monthly to update some stats on where I am.

    I think this feels slightly ambitious, but we’ll see how 2020 goes. Hopefully better than 2019.

  • Automatic Refaction of PII

    Data privacy and protection has become a hot topic. Across the last few years, as I’ve worked with customers that deal with the GDPR or other laws, they’ve become more concerned and careful with their use of data. I find less and less resistance from developers about using sensitive production data in development environments, but still too much.

    Using data that’s stored in databases or other text files is one thing. What about data in less structured forms? I’ve dealt with a few customers over the years that recorded customer interactions for various purposes. Call center or financial organizations commonly do this, and sometimes deal with sensitive information. I know I’ve given my date of birth, credit card number, bank account number, and other data to representatives at various times.

    Those calls are often recorded, and the IT staffs have often had to ensure extra security is applied to these files. Not everyone has access to listen for various reasons, but certainly when there is sensitive information inside the audio (or video), this data needs the same protection we’d apply to data in other forms. Providing protection, or redacting the information, isn’t an easy task.

    I saw recently that the Amazon Transcribe service will now redact some PII information automatically. This service can be configured to automatically remove the information in text, which is fantastic. This is a great way to start to use technology in a safe way to ensure that we have less data  leakage when we re-use data. Certainly people might look at this data to better train reps, but it’s also likely someone wold look through transcripts to determine why customers are calling in and use that information to better design applications. In either case, there isn’t any need to expose PII data to them.

    This doesn’t protect against the data inside the audio, but perhaps companies can delete and remove those recordings sooner with transcripts available and more quickly reduce their potential attack surfaces. We’ll always have some liability, but reducing that and not unnecessarily creating issues is part of what we want to do when protecting data.

  • The Sabbatical Part Deux Review

    I just completed my second sabbatical break this week. Redgate Software is an amazing employer and we give employees a six week sabbatical every five years. I was overdo, and finally took mine in Jan/Feb of this year, with the third one coming up in 3 1/2 years. In any case, all the posts tagged Sabbatical and named “Part Deux” documented this break.

    I had written about my plan before starting the break. This time I wasn’t as structured as I was for the first one, but overall, I think things worked out well. I dropped a few thoughts in an editorial, but this post looks back at how I did with my initial plans.

    Guitar

    I think I did best here. I played most days, often for an hour or more. I watched a lot of YouTube videos, and even worked through parts of a few courses. I’d subscribed to GuitarJamz late last year and started building up some strength and calluses in my fingers. I worked through some exercises and learned some songs. I’ve followed Marty Schwartz on YouTube for years, and this was a good time to focus.

    I also discovered Six String Fingerpicking and started working through some of his all time top songs. I managed to work through the top ten, though I just barely know them. I’m not that good with all of them, but my wife did enjoy hearing me play. She thinks I improved quite a bit.

    This was a random assortment of songs, and I added a few more in there. What I think I did well was learn these things at a competent, not performing, level:

    • Blackbird
    • True Colors
    • Everybody Hurts
    • Wonderful Tonight
    • Ain’t No Sunshine
    • Wherever You Will Go
    • Every Breath You Take
    • 12 Bar Blues progression

    I lightly knew these songs, but I improved quite a bit and learned a few things I had been playing wrong/poorly:

    • More Than Words
    • Fast Car
    • Landslide
    • House of the Rising Sun
    • Good Riddance
    • Iris
    • Hey There Delilah
    • Hallelujah
    • Slow Dancing in a Burning Room
    • I Walk the Line
    • Strong Enough
    • All My Loving
    • With a Little Help From My Friends

    Lots of improvement here, and a good hobby I’ve enjoyed. Looking forward to taking this further in the future.

    The Basement

    I had hoped to not only clean the basement, but rework some framing and perhaps drywall a bit. The reality is that I underestimated the amount of junk and how quickly I’d fill our dumpster. It’s only emptied every other week and with the weather being poor, it wasn’t a good time to try and get a larger container.

    I did clean out about half of it and organized things. I also managed to donate three or four carloads of things to Goodwill and the library. A vast improvement, but not what I wanted. The existing framing was in worse shape than I thought, so I didn’t tackle any of that. Ultimately, I pivoted a bit on this.

    The Garage

    The same thing here as the basement, though this was more weather related. I had hoped to clean out and reorganize some of my workshop areas, but this was a big project and I didn’t have good space to work when the weather was bad. Between cars, my son’s motorcycle and a riding mower, I couldn’t get going. When I was ready, it was freezing in there, so I abandoned most of this.

    A failure, a bit, in my mind.

    Barn Projects

    One of the things I didn’t plan was barn/ranch projects for my wife. However, I did get a few things done here. I repaired a few of the feeders that I’ve built over the years and I managed to cover a building in metal.

    Perhaps the biggest project was getting an electric winch into the riding arena to raise and lower the door. We’ve done this manually for years, with a hand crank. Every one of us has climbed onto a barrel to lower this slightly, then go unhook the door, then come back and open the door. When we’re done, we reverse that, resulting in four barrel climbs every day or two.

    I replaced this with an electric winch and a remote, so there’s no need to climb, but also we can hook/unhook the door at the door without walking back and forth. My wife’s employees love this.

    I had to engineer a way to attach this winch to the frame, which was custom metalwork for me. I then had to wire this to a battery and add a solar panel to keep it charged.

    Attach7512_20200221_134709

    Across the last month, this has worked well and everyone appreciates it. I even pleased myself with my work at the end of the sabbatical, trying to get hay to horses. This was one thing I’m quite proud of.

    I also identified a few other repairs I need to make across the next couple months. These are things I’ve let go and not checked on. I wasn’t aware how bad things had gotten with all my travel, but now I know I need to focus on some side projects around work.

    Health and Illness

    I wanted to take care of myself during the break, hitting the gym, getting massages, and snowboarding. I ended up with these stats in six weeks:

    • 5 massages
    • 7 days snowboarding
    • 28 gym workouts

    Good progress. In 42 days, I had 35 workouts. Not fantastic, but good. The limiting factor in most of this was my health. I fought a cold for about 2 1/2 weeks, just struggling to get healthy. It only knocked my down in bed for a couple days, but I was moving slow.

    Overall, this wasn’t much better than I might have taken care of myself while at work, other than the massages. Those are hard to fit in. Not great progress, but a lot less stress.

    That might be the best part of this break. My stress and concern with getting things done, and having deadlines, was low. I still had coaching duties, and I kept fairly busy, but I could let things slide a bit if I needed to, like the garage.

    This was a fantastic break. After my first sabbatical, I wasn’t sure what to do the second time. Now I know I don’t necessarily need a plan, but the break will do me good.