Voice of the DBA

Category: Blog

  • Investigating Azure Databricks

    At SQL Saturday #884 – Pensacola, I dropped into Rodney Ladrum’s session on Azure Databricks (ADB) and the Traditional DBA. I had heard a bit about Databricks, and read a little, but I didn’t really know much more than a rough overview. I’ve heard quite a bit from Microsoft about running Databricks notebooks, but I didn’t necessarily know what that meant.

    I was hoping to learn a bit, and I did. This wasn’t really a Databricks session, but a look at how you might need to manage and work with Databricks as a DBA. This assumes that there are data scientists or other data engineers that need to work with and process data, choosing to do so with R, Python, or Scala, but wanting to use Databricks notebooks.

    I need to learn more, and stumbled on 30 days of Databricks. I can’t get to that right now, but I added a subscription to remind me to come back. Between Rodney’s session showing me the basics of working with this in Azure and the intro video, I know a tiny bit about the technology.

    Databricks is a company and a way of managing and running Spark analysis. Apache Spark is an open source project that implements an analytics engine. However, it’s complex to run, so Datarbricks (the platform) is used to make this easier. Azure Databricks is an implementation of what Databricks sells, under license I assume.

    I know a little more, as there are notebooks that can be executed under the Databricks engine, and the code in here can be bash, R, Python, Scala, probably something else.

    That’s my start. I’ll learn more over time. If I find time.

  • Learning more Kubernetes

    I’ve been slowly working my way through the 50 days of Kubernetes (K8s). As you might have guessed if you remember my first post, this has been more than 30 days. Life and work get in the way, but I’m working through the series of posts and videos.

    A couple of interesting things I’ve watched lately:

    Serverless Kubernetes and Serverless on Kubernetes – I worry about the idea of containers needing to spin up to meet serverless workloads, but maybe not. It’s a few ms for these containers to spin up, especially for small functional environments like Python or .NET. I suppose if there are issues, you can schedule more pods to be available or more replica sets to spread the load. This makes sense to me, and containers are often a perfect use for this on Kubernetes since you want to have lots of small items, but one large endpoint for people to hit things. I suspect this is how the FaaS implementations work on Azure and AWS.

    This video also talks about the need for a virtual kublet to allow the API to get things ready, without having underlying VMs, a prerequisite for scheduling items on the cloud. This is the serverless kubernetes concept. I’m not sure I completely understand this, but I get the idea here. We don’t have hardware provisioned, we’re running Kubernetes, and we want to push some of our load into the cloud. Since we don’t have a node assigned to us, we want to schedule on a virtual node that the cloud provider will actually spin up when we hit it.

    Fascinating and not something I’d have thought of before I saw this video.

    How the Kubernetes Scheduler Works –  This is interesting. Scheduling workloads and pods in different places is important. Flexibility while meeting demands matters. The idea of hard (required) and soft (optional or preferences) constraints for choosing where to run pods is fascinating. Another video by Brendon Burns. Most of us might care about these items since we may want some spreading assurances for instances or minimum resource requirements for our instances.

    For example, we might have a hard constraint that our pod (container) needs 128GB of RAM. This might limit the nodes that we want to run this pod on, and the scheduler takes this hard constraint into its decision making process.

    For soft constraints, we might prefer that a reporting instance not run on the same node as a OLTP instance, but if there are no nodes available, perhaps we’d live with this. That’s a soft constraint, and the scheduler tries to honor these, but it isn’t bound to prevent scheduling on those nodes.

  • Great Access to Data with Google Fi

    I’ve been a subscriber to the Google Fi (formerly Project Fi) network for a couple years now. Like many of you, I depend on my mobile device for lots of data items, usually email, but I’ve also been able to get work done at times, and certainly Slack and Skype are a regular part of my work.

    I made the switch because I often travel to the UK and a few other countries. Getting access from the US carriers is a pain. I’ve switched SIM cards before (never like doing that), carried a second phone that I can put a SIM in, and even tried to live on Wi-Fi. These are all compromises and a hassle.

    I used to have Verizon and the had great coverage, but at $10/day, that adds up quickly. T-Mobile had free data, but low speed. My kids have Sprint and get the same, low speed connection that’s almost useless in today’s very chatty applications.

    The last couple years have had me use the Google network in my travels. I’ve gotten phone and text calls in many places, tethered my computer, and been able to keep in touch in same way I do every day, regardless of where I am. I’ve been in these countries with Google Fi:

    • United States
    • United Kingdom (GB, Scotland, and N Ireland)
    • Ireland
    • France
    • Switzerland
    • Greece
    • Norway
    • Denmark
    • Canada
    • Australia
    • New Zealand
    • Hong Kong

    In all of these countries, my phone connects and gives me 4G speeds, with my same data plan. I pay a bit for international calls, but texts are included. What’s even better is that my data cost is capped at $60. I pay $10/GB, but if I go over $60 (And I did while in Australia), there’s no extra charge. If I use less, I get charged on a pro-rated basis.

    If you decide to move, I’ve got a referral link. You’ll save some money and I’ll get a little credit. Referral: https://g.co/fi/r/KAF848

  • Containers make easy upgrades

    I’ve been working with containers and writing a bit about them for awhile. I find them to be fascinating and useful as a technology, and I’ve come to really believe these will change how we work with the SQL Server data platform.

    In any case, I wanted to get the latest SQL Server 2019 version recently, so I decided to download the container image. I’ve had a few on my machine, and so upgrading to the latest was a question of running this:

    docker image pull mcr.microsoft.com/mssql/server:2019-CTP3.1-ubuntu

    The main SQL Server 2019 image exists in a few flavors on my machine, so as you can see, I don’t even download the entire image.

    2019-07-03 19_15_29-cmd

    There were only 4 layers I needed to pull since I already had the other 5 layers. This took about 10 minutes on my nice, but not great, 15MBps download speed. That’s much quicker than I could download an iso to install, plus I’d need to uninstall the previous CTP.

    With containers, I can just run any CTP by picking the image from my list.

    2019-07-03 19_18_01-cmd

    If you haven’t tried working with containers, I’d urge you to start learning a bit. This is amazing technology that your developers already use, or want to use, and I expect data professionals to follow suit in the next few years.