Voice of the DBA

Category: Blog

More Kubernetes Concepts
This post continues looking at my process of learning more about Kubernetes. I’ve been working through the 50 days of Kubernetes (K8s). Now that I had a break from some travel, I went through a few videos in a short period of days. This post combines a few looks at Scenarios, storage, and stateful sets

Scenarios

This was a good look at how one of the founders see the Kubernetes being used. The initial idea was microservices, where many services each live in a container and communicate. Managing lots of containers, and lots of instances of each, is where Kubernetes excels.

The second scenario is the diurnal or cyclical scenario where a load goes up and down each day/night, week/weekend, etc. Having Kubernetes manage the increase and decrease of workload is valuable, and if you’re in the cloud, then you’re not investing heavily in hardware that is unused.

The third is with complex workflows, where one thing, then another, then another happens. I’m less sold here, but I do see that spin up/spin down, and having these workloads/processes in containers rather than on certain machines makes sense.

Lastly, AI/ML places where all parts of your flow can access the Kubernetes API and spin things up and down or move workloads to better hardware, like GPU/FPU concentrated systems, that can help keep things moving.

Good to see the different ways that this might be useful, though I was wondering if I’d see anything database related. I didn’t.

Volumes and Storage

The important thing for database workloads is storage. This is a constant question when I talk containers. What about the disks and storage and how does this work.

This is a basic video, just noting that volumes are objects in K8s. They are attached to pods, but can be mounted (or not) inside containers in the pod. There is temp storage (/emptydir, need to look this one up), but we can also abstract away other storage with Persistent Volume Claims. These are the aspirational storage requirements that the container/pod needs.

I’m not completely clear on these, and the explanation is a little light, so I’ll have to learn more later.

Stateful Applications

The last set of learning for this post. More relevant because SQL is a stateful application. Or so I thought.

This really looks at stateful sets in Kubernetes, which are the ways in which we want orderly copies in a replica set. This means hostnames and DNS entries in stateful sets are ordered, and 0 based indexed. If I have a host of SalesSQL with 3 sets, I’ll get
- SalesSQL-0
- SalesSQL-1
- SalesSQL-2
Likewise, I may have salessql.mydomain.com, but I’ll also have the –0, –1, –2 DNS entries as well. The Kubernetes platform also will scale up and down from the end, adding or removing the next ordered element container.

There can also be volumes associated here, or not, as your stateful application may have the need to use the same volume (like a clustered SQL Server) or separate volumes, like AGs. In any case, it’s a light explanation that I need to know more of.

One thing I didn’t really like is that the data stores were associated as “cloud native storage applications”, not data stored or data platforms. Application developers I think sometimes forget there’s more to data than storage.
July 29, 2019
The Data Request

Spotify implemented a way to get your data from your profile. This looks at the request and initial data download.

The Request

When you go there, you see a “Manage Your Data” section.

Below this is a “Download your data” section, with three steps. The first is a request to verify your identity. Clicking this will send an email to your registration email. I like the idea of verifying your identity somehow rather than just letting you get the data. Minor security hurdle, but still one.

After clicking on the email, step two lights up and shows you that it can take up to 30 days to get data. Similar to a GDPR request, I guess this sets some expectations if they get overloaded, but for now it’s annoying. We’ll see when the data comes.

Step 3 is the download link, that you can resend.

Overall, a good process to get what might be private data. It will be interesting to see what they keep on me and what songs and artists appear, both in popularity and unpopularity.

Getting the Link

About nine (9) days after sending my request, I woke up and found this in my email:

I clicked the link and went to the same manage data page shown in the first section, but before I could get the download, I saw this. Nice to see they are verifying access to get data.

When I verified my password, a file downloaded immediately. As expected, it’s zipped.

What’s in here? A number of JSON files, of various sizes. There are some items that are less interesting, like the plan and payments.

If I open the playlist.json file, I see this:

One cool thing here, is that this first playlist is one I created yesterday, the day before I received this link. Meaning, I think the notification was someone processing my request, not the time to get the data. The data was assembled with a process, that used data as of yesterday. I could be wrong, but I have a list of songs in a playlist.

Going on to StreamingHistory, which is where I thought I might see interesting data, I see this:

This looks like I only have a bit of history, not a lot. The first row is April 2019, the last row is July 2019, when I got the download. Disappointing, as I’d assume I’ll have a much different set of history this year than last year. I’ll have to load this into a database and try to decode it. That’s for another post.

It is nice to get some of this data, and I wonder what it looks like if I download it again in a week or two. Especially after another trip or two.

If I lose the data, I can go back and get the file again, at least for 30 days.

I’m glad that companies are starting to give us some data, at least for us data people. Certainly most people might just like a visualization of analysis of data, but allowing users to get their data means that *anyone*, or more *anyplace* can do this analysis for you. Opportunities that your data gatherer might not create.

If you use Spotify, I’d be curious what you think of your data. Feel free to write your post and leave me a comment on how you loaded and examined this data. I’ll do some follow-ups on this later.

July 26, 2019
Redis University

As if I don’t have enough to do, I started a data structures class at Redis University. Someone recommended this to me as another way a company has structured learning, so I decided to give it a try. There were some things they liked that we might use as Redgate University.

I decided to start with the 101 course, since I have a rough idea of what Redis is, but now how it works. The class has a number of videos, with transcripts alongside that teach various concepts. One thing I like is that if I get lost, I can glance at the transcript, click it, and the video jumps back to that point. That’s handy, as life sometimes distracts me.

The quizzes every video or two are nice, but I wish there was a bit more chance to practice things in the middle. I could certainly grab a Redis VM or container, but I have no real idea of how to access it, and I’d have another project. I would actually suggest some “launch lab” points after each set of videos with some practice tutorials or exercises.

The class has two weeks, with a few chapters, each of which is about 6-7 videos. I don’t know about you, but that’s a lot in a week. If I could abandon some other things for an hour a night, maybe, but that can be tough to do consecutively. I’d really rather this was a 4 week class, with each of the items listed as a set of learning and practice for one week.

Overall I learned some interesting things about how Redis works. With the power and speed of this memory based key-value system, I can see why some of our customers are using it. If you are thinking about it, or your app uses it and you want to know more, maybe check out their university.

July 24, 2019
Kubernetes Pipelines
This post continues looking at some of the Kubernetes concepts I’ve been learning with the 50 days of Kubernetes (K8s). Specifically in this post, I’m writing and thinking about the flow of containers from code to a Kubernetes cluster.

The video talks about pipelines, which is the way in which you get code running in a Kubernetes cluster. The start looks at the idea that we don’t want arbitrary containers running in the cluster, so we limit access to the cluster to somewhere like a container registry, like Azure Container Registry. This ensures that only those containers loaded into your space get onto the cluster. No pulling from Docker Hub or some arbitrary location.

However, then we worry about developers loading containers into the registry that haven’t been checked. So, we don’t give them permission, we give it to some CI/CD pipeline, which means any code written by a developer that builds a container has to be checked in.

Now, we have a pipeline flow that we can control. As Brendon mentioned, this isn’t simple, and really, I think you have a lot of work here before you want to trust things running on a production Kubernetes cluster. Really now you need these in your CI pipeline:
- Unit tests
- Vulnerability scanning
- credential scanning ( I see this from MS for Github repos)
- every other type of scanning you can do
In this way, you ensure some quality level in your containers as far as security goes and code working. The big win here, at least for me, is that you can continue to raise the bar by adding more tests and scans, something that’s hard to do for developers without automation.

Not a lot new here, but certainly the idea of putting access and control into the CI/CD flow makes sense.
July 22, 2019