The Database Server Upgrade for Let’s Encrypt

Let’s Encrypt is a service the provides free SSL certificates for web servers. They have grown from a small project to a critical service on the web that serves over 235mm clients each year, and often issues over 1.5mm certificates each day. They depend on a MariaDB database to support this, and recently, they released a post on the database server upgrade they made. They run physical hardware on a managed services provider (Datto), rather than using a cloud platform service, which is fine. While cloud services are great, they aren’t always the best choice. I’d like to see them talk about any evaluation as to what Datto is better than, say, AWS.

In any case, I like seeing how people upgrade hardware, as well as the impact of changes. I haven’t always done this when I’ve upgraded my home systems, though I do notice the new one always runs faster. Good for me, but for production systems, we should have better metrics, if for no other reason than to better learn how to size the next upgrade.

In this case, Let’s Encrypt has a great post that shows how the hardware changed. They moved from Intel to AMD, 48 threads to 128 threads, doubled RAM, and went from mid 500MB/s read/writes to 3200MB/s. Their response metrics also improved, with API requests going from 90ms to 9ms. CPU dropped from 90% to 25% and latency is a third of previous values. Quite an impressive change.

I like seeing the details of their change, and I think this is a nice pattern for others to think about. Whether you spec out machines, VMs in AWS, or DTUs in Azure. You ought to have some metrics that matter for your customers, and then compare the before and after. Prove your upgrade worked, or maybe prove that you need to alter your specs after the upgrade. One advantage with a cloud provider is that you can often scale up or down as needed. With an MSP, it might be harder, but I bet you can still change hardware fairly quickly if you suspect a problem.

While we all want to test a new configuration before we turn it on, simulating a workload can be challenging. Clients somehow always seem to find ways to stress systems we hadn’t anticipated. When spec’ing out a new system, make sure you have room to grow if you’ve underestimated how the new system performs. You also might set specific targets that aim to improve your metrics to some level. Even if you don’t hit the estimate, you will learn something about your methodology that can help you improve your guess for the future.

Above all, publish something like Let’s Encrypt. Share some data and help our industry move forward. We’ve love to have you write something for SQLServerCentral.

Steve Jones

Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

About way0utwest

Editor, SQLServerCentral
This entry was posted in Editorial and tagged , . Bookmark the permalink.