Tag: Cloud Computing

  • The Cloud Database Cost Analysis

    There is a skill that I think DBAs and sysadmins will need to develop: cloud cost analysis. I’ve thought this was important for quite a few years, and I’ve been (unsuccessfully) lobbying for cost information to be gathered and analyzed in Redgate Monitor. Hopefully, this work will get done soon, as I see more companies asking their technical people to provide analysis and justification of the resources being billed for in the cloud.

    Basecamp analyzed its costs in 2023 and decided it could save money by leaving the cloud. I’ve seen other companies decide they were saving money in the cloud. Many, however, are likely unsure of the total return they get compared to the costs of cloud computing. I have seen some posts (like this one) that try to help you get a handle on your costs, but there is often a lot of complexity in cloud costs when multiple departments have different accounts (AWS) or subscriptions (Azure) with a provider.

    In many ways, I think the large cloud vendors haven’t really considered how to support large enterprises that need a lot of resources, a lot of different administrators, and a wide variety of ways to both secure those resources as well as aggregate billing. I know we have an Azure account I didn’t create, where I have very limited rights to do things, but I can access some resources. However, my group has an AWS account that appears to be sending us billing statements instead of our corporate finance group.

    What can be even more difficult to understand is that applications using the cloud might want a complete picture of billing, but various different technical groups might be responsible for the different parts of the system. Data services could be managed separately from web servers or application servers, with different groups creating, configuring, and destroying them as the needs change. Billing, however, be something accountants would want to separate by function/area/application, not group. Knowing all data services for all databases cost $xx/month is different than knowing the retail website (containing web servers, databases, and networking charges) costs $yy/month.

    This is far different from the days when one group was responsible for most of our computing resources. Often we had a group of IT staffers who managed hardware, ordered it, and could link costs of different machines to a specific application or department. Now we often don’t have a central group who is even aware of all the resources that the company has provisioned.

    I suspect this will mean that many more technical people will not only be asked to account for the cloud resources being used, but also split out those costs in different ways, allowing people in finance departments to aggregate the different costs from the various technical groups. I don’t know who will actually track network resources, but I suspect applications will often have this data included with the servers or services that access public networks.

    Controlling costs and carefully removing unnecessary or unused resources is going to be a continual problem in the cloud. I suspect that quite a few data professionals are going to be integral to helping manage this, especially for dev/test systems that are easily forgotten about over time.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

  • Auto Renew Subscriptions

    This week I saw an announcement from Azure that you can set up auto-renew subscriptions for your reservations for your resources. This allows you to set up auto-renew for reservations to prevent them from not renewing and being charged at the pay-as-you-go rates. This reminds me of the old Columbia House deals, which I had for both cassettes and CDs. For some time these were good deals for me, but over time, I’d lose track of the subscription, forget to return things, and end up purchasing items I didn’t want.

    An auto-renewal is likely both a good idea and a bad one, as I suspect that many organizations lose track of their renewal dates when they have an estate of any size. Administrators or finance people will come and go, and they might not know the history of why we reserved something, like a VM, and could forget to renew these. Often we are still using resources, so renewals make sense.

    However, we also stop using resources or don’t use them at the same capacity levels, so perhaps we want to not renew. The docs for this feature note that the renewal creates a new reservation, and this setting can be changed at any time. However, some of the other docs aren’t clear to me. The descriptions read as though someone has an idea about how this works, and they don’t quite clearly detail the workings of the system for readers that are not familiar with this feature.

    As an example, the docs say if you enable renewal more than 30 days before expiration, then  you get an email explaining renewal costs. However, with automatic renewal, do you get the email? When are costs disclosed or detailed? Maybe more importantly, the doc notes the price may change between when you lock the renewal price and when the renew time occurs. Does this mean you pay the higher price? Are you notified each year?

    Maybe more importantly, if we create resources at different times, can we somehow get renewals dates to align? Whether this comes to finance people or IT people, at some point with a big cloud presence, you might have renewals coming every month or week. Who would be able to track this stuff appropriately if it’s a constant item to check? I can see plenty of people starting to ignore these notices if they come too often. There are other issues as well, such as a SKU being different (or deprecated) over time. Will someone figure out how to upgrade to a newer SKU? If so, will they remember to set auto-renewal? Does this transition to the new SKU?

    The documentation is woefully incomplete, though I’d be happy to submit PRs for edits. I just don’t know what the answers are, and I suspect that whoever built this feature might not have thought through all the scenarios. Certainly the individual writing the documentation didn’t.

    The cloud is complex, and billing is complex. Tracking and managing this is going to be partially a technical role, and I suspect many of us won’t know what the implications are for the choices we make. I also suspect we’ll get notifications for resources that we have no knowledge of and will need to research more about the item. We’ll become more of a financial DBA over time, which may or may not be the job we want.

    Steve Jones

  • Continuity Across Restarts

    There are a lot of database platforms, and each tries to convince you theirs is better. As Brent points out in that link, sometimes they just skip comparing themselves to other platforms because it makes them look better. They only look at the platforms they compete well against.

    For most of us, we often just need basic CRUD operations. I know that most RDBMS platforms would work for us, and sometimes NoSQL ones work as well, though I think that NoSQL isn’t necessarily better for many applications (maybe most). You may feel differently, but that’s my view. While I use SQL Server, I think the majority of systems I’ve managed or built could easily run on MySQL, PostgreSQL, or many other platforms.

    In any case, we want to store some data, get it back, maybe change it, and sometimes delete it. I think that last operation is fairly rare as most of us use soft deletes and because of this, our databases continue to grow like crazy. There was an interesting post from Brent on MySQL in AWS Aurora, where the buffer pool cache can survive a restart of the database engine. This is for the cloud version from AWS, not MySQL in general, but that’s fairly cool to me.

    And scary.

    If you read more, there are some limitations, and the value of this, to me, is limited. However, it’s a neat idea, since I could see some sort of restart of the database from a patch for the engine or even a configuration change. For MySQL, clients can’t connect while the buffer pool is initialized, which isn’t something that we deal with in SQL Server, but I can see this being useful. Or perhaps this helps if an Aurora node goes down and the next one picks up the database engine with a full buffer pool? Not sure if that actually happens from the docs, but if they kept the buffer pool separate enough or up to date on a second node, that would be great.

    In the real world, I wonder to what extent we find an empty buffer pool at startup to be a problem. Sure the first few queries are slow, but most of the workload isn’t the first few queries. It’s all the clients that come in the next 5, 10, 20 minutes.

    As Brent notes, this might not be something we want in SQL Server, but it is good to see cloud-enabled platforms trying something new. Some of those will turn out to not work very well, but for those that do, I bet we see other platforms start to adopt some of the ideas from new technology.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

  • Re-Evaluating the Cloud

    Last year 37 Signals (makers of Basecamp and Hey)  announced they were leaving the cloud. I wrote about the decision, and wondered if they’d look back at this as a great decision or one they’d regret and backtrack to the cloud again. They planned to build their own tooling, buy a bunch of servers, and run their own data center (or rather, rent space in someone’s data center).

    Recently there was an update, in an FAQ, about how the transition has gone. In short, very well. One of the founders, David Heinemeier Hansson answered several questions about the move and the financial status. They didn’t hire more people, so their payroll is the same. They used a service to unpack and rack their servers, so they could just connect to them remotely and not deal with hardware. They built their own redundancy across two data centers where they rent space, and they think they will save well over $7 million in the next 5 years. They had a $3.2million cloud budget per year, so that appears to be halved (3×5=15 – 7 = 8ish) with their move.

    There are some answers to various questions, which are likely of interest to many data professionals who might feel pressure to move to the cloud. First, they discuss resource optimization in the cloud. I think that’s really hard, to do well and if they gave the details of what they’d done, that might help others decide if 37 Signals did a good job, or if there are things others could do. I think many people might struggle to optimize their usage, especially if you are only responsible for one thing, like the database. In many orgs, once something gets deployed, or developers build a PoC using some cool, new service, it’s hard to get rid of it or even change how it’s used. I think 37 Signals has some agility here that many orgs struggle to implement.

    Rewriting things are cloud-native is the way to go, in my mind, and I do have customers and clients who see applications working better in the cloud. However, I might argue that rewriting applications is hard and expensive, and many companies aren’t great at writing software that’s efficient, so rewrites are hard. I also think in most organizations, developers struggle to understand what cloud-native means. I think DHH is a little off here, as their developers probably could make better software written for the cloud, but they didn’t want to spend the time and/or money.

    The points about security and reliability are fair, but I think those are a function of having good people implementing systems. You can get good security and reliability in the cloud or poor implementations of either. The thing about security is that the overall protections that detect some of the DDOS or attacks are better with Azure/AWS, but simple SQL injection or open firewalls are still a problem. Authentication and security are just hard for most people and often people do this poorly in the cloud. To be fair, most people don’t do this well on-premises, so at least in the cloud, there are some services/scanners/checks that let you know when you’ve messed up. Whether you change or implement their suggestions is another whole debate.

    Is reliability better? AGs are hard in SQL Server, as are distributed clusters. The cloud services do this better for most people, but maybe not for you. That brings me to his super engineer point. He doesn’t think he has super engineers on staff, but I disagree. I think they can pick good people, especially because of their profile, and they don’t need a lot of people, so their average engineer is likely better than the average engineer at most organizations. They built some great software, and they’ve built an amazing framework and tooling to deploy their software. Don’t tell me that’s even close to the capabilities of the staff at many organizations. We have really good engineers at Redgate, and I think we’re above average as well. I don’t know how we compare to them, but we haven’t had people publicly write and release code that thousands of other developers use. At least not at quite the same high profile level.

    The cloud has its place. It can work well and it can be very expensive. This journey is worth sharing with your management if they want to move to the cloud, especially if they want to lift and shift. That might create some flexibility and CapEx/OpEx changes that are worth it, but you ought to debate and question whether that’s reality or marketing hype from a vendor. After all, it’s not clear if a cloud move is really something that returns an ROI or one that reduces your profitability.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.