Voice of the DBA

Tag: containers

The K8s Overview

I’ve written some posts on Kubernetes (K8s) as I think this will be an important technology in the next decade for database systems. Actually, it’s not that I think Kubernetes is that important, but rather that the orchestration of containers that run database systems, along with the storage and networking, will be necessary. Kubernetes appears to be the most common and popular orchestrator, but that could change. There are other orchestrators out there.

Part of the reason that some technology gets adopted is that managers read about it and push it. Certainly technical staff have input, but managers have influence, and ultimately spend money on projects. While every manager wants to succeed and make their mark, no one wants to be left behind and have made a poor technology choice. As popular as containers and Kubernetes are, a little advocacy from management might get you moving in that direction. That’s if you want to implement containers.

If your manager knows nothing about Kubernetes, can they still recommend the platform? Sure, but you can also help them to understand how it works and what the impact of this choice might be. I ran across A Manager’s Guide to Kubernetes Adoption, which is a good read for a manager, perhaps tempering some of their enthusiasm to just install Kubernetes now. In fact, if you know nothing about containers and Kubernetes, you might give it a read as well.

I’ve changed my mind on database containers, and I do think Kubernetes has a future in the life of the data platform pro. It’s not simple, and may not be easy to run for some time, but I think that will change. I do think we’ll eventually just get a K8s service, or simple install, or use it in the cloud. I also think we may not need a lot more Kubernetes knowledge than is in the post, but we certainly will want to know what configuration items might affect our workloads, and certainly we’ll still need to think about, and set limits (high and low) for our database systems.

I don’t think that our world will change quickly, and certainly the more legacy applications you run, the longer your transition will be, but I do like the concept of Kubernetes as a platform where I deploy applications and services. I haven’t quite felt that way about AzureStack to date, nor OpenStack and other variants, but Kubernetes seems to be a better idea for what I’d want. It will take time to get there, and it’s not trivial effort to move to it, but once there, you should have reduced some of your infrastructure headaches.

As to whether your application will run better, look to your developers. All the hardware and well built platforms in the world won’t work well if you don’t write good code. Spend some effort there, and then think about moving to containers and orchestrators.

Steve Jones

Listen to the podcast at Libsyn, Stitcher or iTunes.

October 23, 2019
Local Database CI Builds with Jenkins in a Container
In two previous posts (Jenkins in a container, Adding SCA to Jenkins), I’ve showed how to configure Jenkins in a container as well as how to add the SQL Change Automation plugin. This post looks at an initial database build in Jenkins. I will build from my local repo, which gives me a bit of a pre-push check on my code.

Note: I already have a SQL Source Control (SOC) project committed to Github that contains the database I want to build. If you haven’t done that, see this post on a basic SOC->GitHub link.

Creating a Jenkins Job

Jenkins calls these build processes jobs, and you see that if you set up Jenkins in a container. By default, there are no jobs, and Jenkins lets you know this.

When I click this, I get a choice of different projects. You can read about these in the Jenkins documentation, but I am just going to use a Freestyle Project. Give it a name and click the type and then OK.

At this point, we need to configure the job. I do like to provide a description, and normally I would set this as a Github project, but in this case, this is really just a local build..

I do want to restrict which agent will run this, since I have a master agent in the Jenkins container. That one can’t see my SQL Server (and doesn’t have tools), so I’ll restrict this to the local agent on my machine.

There are tabs at the top for quick navigation, but this config is really just one long page. You can scroll down as needed. Below the agents, I find the VCS section. I’ll choose git, and then enter the path the agent will need.

When I scroll down to the Build area, I need to add a step. CI is just a step engine to assemble your software, and in this case, I see my Redgate SQL Change Automation plugin.

When I click this, it is added as the first step. There is an implicit “git clone” checkout of code. I need to configure this step with just a few items. First, build in the local folder, rather than some specific area. We’ve already specified an agent location when we configured the Jenkins agent.

Also, you need a package ID, which will be the place where the current state of all objects is stored. Unlike application code, the artifact is really the current version of the objects in source control, not any .exe or .dll.

The only way to build a database project in SQL Server is with an actual SQL Server. In this case, I don’t have any code that would error on LocalDB, so I’ll just use that. I coudl specify my local SQL Server development database if I had the need.

This is a test build, so I also don’t need any SQL Compare options or other switches.

That’s it. I click Save, and I get to the main project page. Now, the test. I’ll click Build Now.

This kicks off a build on my agent. If your agent isn’t connected to the container, you’ll need to do that now. At the bottom left of the page, I see my build working.

The build number is a link, so click that to go to the details of the build. When you get there, you can see the build working by clicking the Console output. As you see, the output is on the right side of the screen. As you can see, my local path worked fine, without any error.

I also got to this part of the process on a plane, with no Internet. While there were some warnings as the plugin and other parts of the process tried to contact the PowerShell Gallery, things worked.

At the end, I see success, which means that the code I wrote in my database, and that was committed to my VCS, is valid. It’s a low bar, but you might be surprised how often someone breaks code by making a strange change to their project.

This is a basic build process, to which I can add other steps. Testing and releasing are further steps, and Jenkins can support you in these.

Now I can move forward with additional development, knowing I have my own build system in my own environment, using Jenkins in a container.

This Series
- Jenkins in a container
- Adding SQL Change Automation to Jenkins for CI
- Local Database CI Builds with Jenkins in a Container (this post)
August 26, 2019
Expose Multiple Docker Ports
I was working with containers recently with Jenkins. I didn’t want the server process running on my machine all the time, but I did need to allow some communication. Jenkins uses 8080 by default, but agents need another port.

I figured there was a way to do this, and I found it on Stack Overflow, which is the perfect forum for a question like this. The answer? Multiple –p parameters.

Instead of
```
docker run –-name jenkinsci –p 8080:8080 jenkins/blueo
```
I do this:
```
docker run –-name jenkinsci –p 8080:8080 -p 50000:50000 jenkins/blueo
```
That opens two ports from the host to the container, which is what I need.
August 7, 2019
Learning Kubernetes Security Concepts

This post continues looking at my process of learning more about Kubernetes. I’ve been working through the 50 days of Kubernetes (K8s). Here I continue on with a few videos on secrets, configuration management and RBAC.

Secrets

I am glad that the idea of storing data securely has come to so many applications. Authentication, authorization, privileges, and more are complex and hard to get right. We continually see problems with passwords, certificates, and the link not being managed well.

Kubernetes has the concept of secrets, which can be values or file contents (like a certificates). These can then be sent or mounted to containers for use.

One problem, these are stored in the Kubernetes etcd (keyvalue store) in an unencrypted format. Kubernetes has new concepts to integrate into key management areas, like Azure Key Vault, to protect secrets, though Kubernetes privilege controls are still needed.

A gross simplification, but I learned a little more about what secrets are.

Configuration Management

This is interesting to me, because I like abstracting things away. In Kubernetes, we abstract a configuration object as a separate entity, which can then be applied to a container/pod later. This also works through deployments, allowing configurations to be part of the rolling deployment, which can prevent issues.

If you have good testing/evaluation of what is healthy. This is likely easier than comprehensive unit testing, but still.

A good idea, just not sure if this is too much.

RBAC

I always think of this as row-based, but it’s really role-based, so maybe that should be the way I translate RBAC in my head.

Like most systems that use RBAC, there is, of course, a role. The video wanders a bit in the explanation, but I’ll try to summarize what I learned. There are two facets to roles. First, these are verb/noun pairs that determine what permission is available for what resource. Like get/list for the verb, and pod, volume, service, etc. for the noun. These are setting up the permission side.

However, there is another facet. These roles can be set as cluster roles, similar to server roles for SQL Server, or at the namespace level, more like a database role. In either case, you are scoping permissions. Obviously, be careful with cluster roles.

The other side of this is the user. We can have users or groups used for these roles, similar to what is in SQL Server.

A simple explanation, but at least I have some idea of the concept.

August 5, 2019