Voice of the DBA

Tag: DevOps

  • Never let a good crisis go to waste

    There have been quite a few jobs in my life where my first day was complete chaos. Some were technical and some were not. I had a first day as a bartender (in more than one job) where someone was sick or the place was busy and I was left on my own to figure things out. Fortunately, I know my way around a keg and a bottle of liquor. I’ve also started at a couple tech jobs where we had a crisis in a system. I’ve been able to pitch in and help solve problems, or even get other work done on my own. In all these cases, the boss was impressed, and I often started out my tenure with quite a bit of credibility.

    When we look to make changes, it can be hard to convince someone that the change is better for the organization. This might especially be true when working in software development and trying to convince someone to change their process or habit. I see this all the time as clients look to adopt DevOps, with no shortage of naysayers. There are always plenty of people that want to maintain the status quo.

    I heard quote from a presentation by Comcast execs on DevOps: “never let a good crisis go to waste.” The quote has been attributed to others in history, but I think it’s an apt description of how to drive your processes forward when you know there are problems and issues and no one wants to spend time fixing them. Pain often allows an opening for a better solution to be considered.

    Too often when there is no problem, no immediate pain, there is a lack of will from management to spend time improving things. Certainly there are some executives that believe in improvements, but far too many focus on adding something new rather than improving the underlying infrastructure of software development. When you see a problem, if you can’t make an improvement at the moment, keep an improvement or solution handy. You never know what a crisis might give you the opportunity to pitch a better way forward.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • A Technology Collapse

    This past week the Arecibo radio telescope collapsed, with cables and instruments falling into the massive dish. You can see images of the devastation, which saddens me. I doubt this will be rebuilt, though one can keep hope alive. there were reports of failures a month or so ago, with the intent to shut down the telescope, remove instruments, and institute a controlled demolition.

    The facility was featured in a few movies, notably GoldenEye and Contact. The latter is one that first showed me the telescope, which I first thought was a movie magic trick. When I found out this was a real facility, I was amazed that humans could conceive and build such a structure. Even more amazing is that it was built in the 1960s.

    That’s quite a lifetime for a piece of technology. While I’m sure lots of wiring, electronics, computer systems, and more were added or upgraded, the core of this telescope remained the same. The ongoing engineering effort, fitting new capabilities around legacy systems and structures, was likely a study that many would find interesting.

    I don’t know many database systems that have survived that long, but certainly there are some long lived legacy applications. The Sabre reservation system for airlines is one that stands out to me. The history of this one is fascinating, especially as it’s a system I’ve depended on in my travels.

    Many of us that work with databases likely feel that everything we do is a legacy system. The dependencies our objects have on various applications, scripts, reports, and more can limit how much repair, improvement, or replacement we can make to schema.

    There are good techniques for modifying our objects, and helping to ensure we don’t break systems, but we often do need some cooperation and collaboration with application developers to implement those changes. Much of DevOps avoids talking about the database, but we shouldn’t. Instead, we ought to embrace database refactoring patterns, both at the database and application levels, ensuring that our systems can survive for as long as we need them while adapting to the changing requirements of our clients.

    Steve Jones

  • Delivering Patches Quickly

    I love cars. In my life, I’ve owned and regularly driven well over 20 cars. If I count the ones I purchased for my kids and lightly drove, it’s over 30. Just writing this paragraph gets me a little itchy about looking for another car. Actually, I’m looking lightly now, as I expect two kids to move on next year and handle their own expenses. So, next year I’m hoping to add another vehicle.

    Recently I saw a note that there was an exploit against Tesla cars, which are rarely stolen, but there have been a few issues. This one was against the fobs that communicate with the car. Tesla is working on a patch, which is interesting. They’ve not only devised their cars, but also their fobs to get firmware updates. Good, and bad, as this increases attack surface area,

    Tesla can patch their cars in real time, and they do this fairly quickly. I have a 2012 BMW, and I have updated the firmware in it, but patches are rare, and most of them seem to be stuck in the “visit a dealer” paradigm. It seems many other makes of cars fall into the same paradigm, and aren’t built for regular updates.

    In some sense, this reminds me of the way many people deal with the their SQL Server databases. We put code out there, in tables, functions, procs, etc., and often we struggle to update that code quickly. This becomes especially true with applications that use our objects. I often find that changes seem to come slower and slower over time, much like the traditional way that your car is updated by a dealer only.

    Ideally, we’d be able to more quickly update code when there are issues. This is especially true if you have procs or other code that could potentially have security issues. DevOps asks us to move towards this type of flow, making changes, and adjusting quickly. There are certainly still challenges with changing code and adapting applications, but if front end developers and database developers worth together, we can deploy code quicker.

    Our customers don’t care about our challenges we face with writing and deploying code. What they want are new features, functions, better performance, and meeting their needs. DevOps helps, but it’s not easy. You have to automate things, deploy code sooner AND be willing to fix your code mistakes. You not only need testing, but testing that improves over time.

    It is, however, worth the effort.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • How Do You Experiment?

    One of the things that DevOps asks software developers to do is experiment. Try new ideas out, get feedback quickly, and then choose how to grow or stop your experiment. This is great for features, and it works well for application software.

    The general flow for this is to talk to customers, and then decide what to build. In some sense, this can work, but as I heard at the DOES Summit recently, if Henry Ford had asked his early on customers what to build, they’d have asked for a faster horse.

    Customers are limited by their current experience. This includes not only end users, but for us database pros, the developers that build software. When they want to experiment, they often need some backing from the database to store information and query it.

    If we want to help enable experiments, and allow our software to evolve, there are two things we need to deal with in experiments. One is schema changes, either through new data buckets in tables, or programmable objects, such as views, functions, and procedures Adding these, or removing them when experiments aren’t useful, can be cumbersome and difficult. It’s amazing how quickly we create dependencies and how slow we are to remove them.

    The other area is in ensuring that we properly or appropriately, handle resource usage. Do we go back and tune queries, or restructure the way that we’ve indexed items to ensure that our system works optimally? Some tuning can be done early, and should be, but some requires some feedback to understand query patterns or data loads.

    Today, I’m wondering how, or if, you experiment in database work. What works for you, or what doesn’t? Or do you hate the idea of experiments in the database world and want more specification up front? Let me know with a comment.

    Steve Jones

    Note: Podcasts are suspended for a week as I deal with the PASS Summit.