Voice of the DBA

Tag: software development

  • Database Checkups

    I am most definitely a fan of continuous delivery (CD) and the idea that we have the ability to both release our software at any time, for any change, but also have the process and mindset in place to rapidly push enhancements and patches through our development process. However I’m also no naive enough to think that the processes of Agile, Lean, Scrum, Continuous Integration, or any other methodology make this happen. It requires belief and effort by your entire team, including management, to evolve your software quicker.

    There is a downside. The faster, and more often, you release changes to your production system, the more likely you are to cause an issue. This isn’t really any different from traditional waterfall processes, which also cause issues in production, but it does mean that the perception of issues is a bit changed. People expect higher quality when they implement a CD process, and while quality might be higher (there is debate on this), it certainly isn’t perfect.

    Walgreens has implemented a CD process, and they rapidly deploy changes, but they’ve also had issues. In fact, there’s a piece that talks about an issue during a database deployment, and offers some guidance on preparation to deal with problems in your deployment process. The ideas of monitoring (which is critical), planning for recovery, coordinating development, etc are all things that we need to do in any production environment, but they are especially important in a CD process where people expect changes quickly. Whether those changes are enhancing the application, or rolling back a problematic deployment, they need to happen quickly.

    Over time, which may be decades, I expect that more and more organizations and end users will expect patches and changes to their software to happen in the background, while the system is being used. This might mean we need to plan for application architectures with sections that can be upgraded separately, or maybe we will develop ways to modify executable code without ending the process. In either case, the ideas behind CD will become more and more useful, and the more you understand them, the better off you’ll be.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.5MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Testing in Production

    The recent Apple keynote announcing the iPhone 6 was very heavily watched. At least, I think it was heavily viewed because I had trouble connecting, numerous pauses and regular stuttering of the video. Despite the fact that the stream was limited to Apple operating systems (iOS/OSX), it appeared that many other people had trouble watching based on the tweets I saw. On top of the scale issues, there were also different language translations overlaid in the audio and crashes of the Safari browser. Overall, the live event was a disappointment to me, though it hasn’t stopped me from upgrading my iPhone.

    The numerous problems that occurred had me wondering if any system wide testing was previously conducted. Was it possible that Apple was actually having their full scale, end to end system test in production? During the live event? I guess it’s possible, though it would be  imprudent and foolish to do so. With all the effort and expense that goes into the “Apple show”, how could a complete system test not be managed?

    Certainly a one time event like a product launch can be hard to simulate. The scale alone is difficult to predict, but certainly there are things that can be simulated. The actual people and applications can be used to record, encode, broadcast, etc. As closely as possible to the conditions of the live event, with the same size, type, and configuration of equipment should be used. The same people who will operate it should participate. We know this as technologists, and most of us would perform testing like this if we could.

    I know that resources are often constrained, and time is precious. However we need to perform some testing prior to production if we are to have confidence that everything will work during a deployment of new bits. The best way to do this is to deploy often, to a variety of environments in the same way we will to production. Execute a variety of tests each time that ensure the application functions as expected. If we find problems, we shouldn’t fix them in that environment. We should start over, fixing the issue in development, and deploying the changes again through test, pre-production, and any other environments we have. Be repeating the process over and over, we can build confidence that our production environment will work as we expect. 

    I hope Apple did this, though the end result has me feeling a bit skeptical.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.9MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

  • Untested

    The 2014 Apple keynote was today, with the announcement of the iPhone 6 and iWatch taking center stage. I was excited, with an aging iPhone 4S that needs to be replaced. I tuned in, expecting to watch the announcements while taking care of some busywork on the other monitor.

    At first I saw this screen (though at about 20 minutes beforehand)

    applekeynote2

    About 10 minutes before the event, the countdown switched, and noted that my browser (Firefox) wasn’t supported. I almost switched to Chrome, but then saw the fine print. I wasn’t thinking and didn’t take a picture as I ran to grab my iPad, but Karen Lopez posted an “altered” version of the Apple page.

    applekeynote3

    Fair enough. Apple doesn’t have to broadcast to everyone and I get that they want to let their iPhone/iPad/iTouch/Mac users see it live. The keynote was available about 30 minutes after the end for everyone, so no big deal.

    In fact, in the scheme of my life, not being able to watch live wasn’t a big deal. I can catch the “showmanship” an hour later.

    However I was surprised when my iPad kept showing a test screen from the broadcast truck. When it connected, it was sputtering and halting, even when a speed test showed me getting 20+MBps at the device.

    I did manage to watch a little, with a Chinese translation overlaid on the sound, at a higher volume than the speaker for me. Slightly annoying, but I could deal with it.

    About 15 minutes in, things started crashing, with Safari on my iOS 7 iPad and iPhone crashing. The iPhone would start Safari and connect to the Apple home page and show this:

    Photo Sep 09, 12 49 36 PM

    The iPad connected a few times and showed this:

    Photo Sep 09, 12 49 26 PM

    I gave up, going on with my day, knowing I could watch it later. However that brings up the question.

    Did Apple test in Production?

    By this I mean, did they actually test the broadcast of this event today, with the live event, or did they test things last week on a copy of what this environment would look like? Did they actually run through the same settings for broadcast that they would do today?

    I do suspect they did some testing, and certainly it’s possible that many, many more people than they expected tuned in. They may have been overwhelmed, and that’s understandable. I doubt anyone has any good way to determine what capacity to plan for, other than to just guess. This event might have been larger than the Super Bowl, World Cup, etc.

    Or maybe Apple didn’t want to bother scaling to a capacity beyond some xx% of Apple device owners.

    Any of those is fine with me, and I can understand someone deciding to spend $yyy on this, though I’d argue it was a mistake since this is a great chance for an amazing amount of publicity and hype for their products.

    However there are certainly other problems that I question the level and detail devoted to testing. The overlay of foreign languages means that someone hadn’t configured their audio correctly, and more importantly, no one was monitoring to stop the problems quickly. The problems with the stream meeting devices might have been capacity, but crashing the browser? Either there’s a major QuickTime issue, or the encoding was broken in some way (perhaps again, by capacity).

    This is a shot at Microsoft as well, but I thought Tim Ford’s summary was spot on:

    applekeynote

    We talk about thorough testing on a copy of production. A real, scale, same sized version. Few of us do it, but certainly some do, especially those with big budgets. Apple has big budgets, so I suspect that their testing was cut short, or someone assumed things would work the same way they had in the last keynote, which went off without a hitch for me.

    Complacency, coarse attention to detail, poor quality control and monitoring seemed to be in play today. None of which as a good thing.

    Perhaps Apple was testing in production.

  • Estimates

    Estimating the time it takes to build a piece of software is quite a difficult task. As many of us have seen over the years, most estimates can wildly differ from the actual time it takes to complete a project. There are lots of reasons for this, but I was surprised to find this post explaining some of the problems in a very interesting way.

    I realize that many software projects are very different in scale and scope, but I hadn’t ever considered the spectrum to be quite as wide as explained in the post. Just as in other industries, there are times that your software project needs to create its own technologies and techniques in order to be successful. While software might not be as complex as curing a disease, it might not be that far off at times.

    However it’s not just that the body of knowledge about software development techniques is so vast, it’s also the problem that each of us might have a very limited amount of knowledge about building specific types of projects. What might seem simple and easy to some developers feels complex to others. A simple example I’ll give you is with version control. I know version control isn’t the same as building software, but it’s a tool that many developers use.

    I think version control is fairly trivial to implement and use. It can be a pain to use with T-SQL code and scripts without some tooling, but it’s easy to learn how to use a version control system (VCS). However as I’ve spent two years talking about version control at various events to lots of people, and I still meet lots of people and organizations that see a VCS as an impediment to getting work done. Hint, it’s not.

    I think we share so much knowledge in the SQL Server community, but it still seems that we don’t seem to be educating people as well as I’d like, and as quickly as I’d hoped. Our Stairway Series were designed to try and organize information, but I’m sure there are other, better ways to do so. I’m open to suggestions, and other efforts to help raise the level of knowledge for SQL Server professionals. I’d like to find ways to help people learn more, write higher quality code, at a faster pace.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.8MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

    The Voice of the DBA podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music.