Tag: software development

  • Testing your API

    I think learning to better test our software, including the database objects, is one of the ways in which we’ll build better software applications in the future. Testing is a complex subject, but this is part of a series that looks at ways in which you can use tSQLt.

    Checking Table Metadata

    One of the easy tests you can write is to compare the meta data of an object to a known quantity. The easy way to do that in tSQLt is to use the AssertResultSetsHaveSameMetaData function. This function compares the structure of two result sets, covering names, ordering, and data types, to determine if they are the same.

    Here’s a quick example of how that’s done.

    Let’s assume I have this table:

     create table Articles  (
        [ArticlesID] [int] identity(1,1) not null,
        [AuthorID] [int] null,
        [Title] [char](142) null,
        [Description] [varchar](max) null,
        [Article] [varchar](max) null,
        [PublishDate] [datetime] null,
        [ModifiedDate] [datetime] null,
        [URL] [char](200) null,
        [Comments] [int] null
      );

    If I wanted to write a test in tSQLt to check this table for changes or alterations, here’s what I’d do in code:

    create procedure Articles.[test Articles_Check_metadata]

    as

    begin

      –Assemble

    create table Articles.Expected

      (

        [ArticlesID] [int] identity(1,1) not null,

        [AuthorID] [int] null,

        [Title] [char](142) null,

        [Description] [varchar](max) null,

        [Article] [varchar](max) null,

        [PublishDate] [datetime] null,

        [ModifiedDate] [datetime] null,

        [URL] [char](200) null,

        [Comments] [int] null

      );

     
      –Act

     
      –Assert

    exec tsqlt.AssertResultSetsHaveSameMetaData

      @expectedCommand = N’select * from Articles.Expected’,

      @actualCommand = N’select * from articles’

      ;

    end

    ;

    go

    Note that I don’t have an ACT section in this test.

    The Assemble section is easy. I record the size and shape of my table. This will be what I compare the actual table to in the database.

    The Assert section is a call to AssertResultSetsHaveSameMetaData, with a SELECT * from the real table being compared to the same SELECT from the expected result table I created. If these match, I pass the test. If they don’t, the test fails.

    Why?

    This seems silly, I know. What does it matter if the table changes, and it certainly will need to change. I definitely questioned the value of a test like this when I first saw the example. However when I thought about it, and thought about the places in which I’ve developed databases, this makes some sense.

    Imagine that I have 3 or 4 (or more) developers. As we get new requirements, we’ll change the schema over time. Imagine that I actually have views built on this table, and other procedures and functions, all of which have some tests on them. If I change this schema, and run a test suite, I could see multiple failures. If I did that, one would hope I realized that the addition of a column here (or a rename) would cause those issues. However if I changed a couple things before running a test, which is something I might do at times, having this test fail tells me quickly that the schema was altered. If someone else changed the schema, I also quickly see that this change was to the schema.

    It’s not a big change, but it does allow me to determine that I need to refactor all the objects (potentially) that depend on this table. I can go do that work now, or add it to the list of tasks for this particular development task, and also fix the tests, which should go quickly.

    If the work doesn’t go quickly because I have a lot of objects, then I’m really glad that I learned now this is an issue.

    This becomes even more valuable with views and procedures returning result sets. If I add a column, then I may or may not want views to change, but certainly a check of view meta data will tell me if they do.

  • Database Checkups

    I am most definitely a fan of continuous delivery (CD) and the idea that we have the ability to both release our software at any time, for any change, but also have the process and mindset in place to rapidly push enhancements and patches through our development process. However I’m also no naive enough to think that the processes of Agile, Lean, Scrum, Continuous Integration, or any other methodology make this happen. It requires belief and effort by your entire team, including management, to evolve your software quicker.

    There is a downside. The faster, and more often, you release changes to your production system, the more likely you are to cause an issue. This isn’t really any different from traditional waterfall processes, which also cause issues in production, but it does mean that the perception of issues is a bit changed. People expect higher quality when they implement a CD process, and while quality might be higher (there is debate on this), it certainly isn’t perfect.

    Walgreens has implemented a CD process, and they rapidly deploy changes, but they’ve also had issues. In fact, there’s a piece that talks about an issue during a database deployment, and offers some guidance on preparation to deal with problems in your deployment process. The ideas of monitoring (which is critical), planning for recovery, coordinating development, etc are all things that we need to do in any production environment, but they are especially important in a CD process where people expect changes quickly. Whether those changes are enhancing the application, or rolling back a problematic deployment, they need to happen quickly.

    Over time, which may be decades, I expect that more and more organizations and end users will expect patches and changes to their software to happen in the background, while the system is being used. This might mean we need to plan for application architectures with sections that can be upgraded separately, or maybe we will develop ways to modify executable code without ending the process. In either case, the ideas behind CD will become more and more useful, and the more you understand them, the better off you’ll be.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.5MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Testing in Production

    The recent Apple keynote announcing the iPhone 6 was very heavily watched. At least, I think it was heavily viewed because I had trouble connecting, numerous pauses and regular stuttering of the video. Despite the fact that the stream was limited to Apple operating systems (iOS/OSX), it appeared that many other people had trouble watching based on the tweets I saw. On top of the scale issues, there were also different language translations overlaid in the audio and crashes of the Safari browser. Overall, the live event was a disappointment to me, though it hasn’t stopped me from upgrading my iPhone.

    The numerous problems that occurred had me wondering if any system wide testing was previously conducted. Was it possible that Apple was actually having their full scale, end to end system test in production? During the live event? I guess it’s possible, though it would be  imprudent and foolish to do so. With all the effort and expense that goes into the “Apple show”, how could a complete system test not be managed?

    Certainly a one time event like a product launch can be hard to simulate. The scale alone is difficult to predict, but certainly there are things that can be simulated. The actual people and applications can be used to record, encode, broadcast, etc. As closely as possible to the conditions of the live event, with the same size, type, and configuration of equipment should be used. The same people who will operate it should participate. We know this as technologists, and most of us would perform testing like this if we could.

    I know that resources are often constrained, and time is precious. However we need to perform some testing prior to production if we are to have confidence that everything will work during a deployment of new bits. The best way to do this is to deploy often, to a variety of environments in the same way we will to production. Execute a variety of tests each time that ensure the application functions as expected. If we find problems, we shouldn’t fix them in that environment. We should start over, fixing the issue in development, and deploying the changes again through test, pre-production, and any other environments we have. Be repeating the process over and over, we can build confidence that our production environment will work as we expect. 

    I hope Apple did this, though the end result has me feeling a bit skeptical.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.9MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

  • Untested

    The 2014 Apple keynote was today, with the announcement of the iPhone 6 and iWatch taking center stage. I was excited, with an aging iPhone 4S that needs to be replaced. I tuned in, expecting to watch the announcements while taking care of some busywork on the other monitor.

    At first I saw this screen (though at about 20 minutes beforehand)

    applekeynote2

    About 10 minutes before the event, the countdown switched, and noted that my browser (Firefox) wasn’t supported. I almost switched to Chrome, but then saw the fine print. I wasn’t thinking and didn’t take a picture as I ran to grab my iPad, but Karen Lopez posted an “altered” version of the Apple page.

    applekeynote3

    Fair enough. Apple doesn’t have to broadcast to everyone and I get that they want to let their iPhone/iPad/iTouch/Mac users see it live. The keynote was available about 30 minutes after the end for everyone, so no big deal.

    In fact, in the scheme of my life, not being able to watch live wasn’t a big deal. I can catch the “showmanship” an hour later.

    However I was surprised when my iPad kept showing a test screen from the broadcast truck. When it connected, it was sputtering and halting, even when a speed test showed me getting 20+MBps at the device.

    I did manage to watch a little, with a Chinese translation overlaid on the sound, at a higher volume than the speaker for me. Slightly annoying, but I could deal with it.

    About 15 minutes in, things started crashing, with Safari on my iOS 7 iPad and iPhone crashing. The iPhone would start Safari and connect to the Apple home page and show this:

    Photo Sep 09, 12 49 36 PM

    The iPad connected a few times and showed this:

    Photo Sep 09, 12 49 26 PM

    I gave up, going on with my day, knowing I could watch it later. However that brings up the question.

    Did Apple test in Production?

    By this I mean, did they actually test the broadcast of this event today, with the live event, or did they test things last week on a copy of what this environment would look like? Did they actually run through the same settings for broadcast that they would do today?

    I do suspect they did some testing, and certainly it’s possible that many, many more people than they expected tuned in. They may have been overwhelmed, and that’s understandable. I doubt anyone has any good way to determine what capacity to plan for, other than to just guess. This event might have been larger than the Super Bowl, World Cup, etc.

    Or maybe Apple didn’t want to bother scaling to a capacity beyond some xx% of Apple device owners.

    Any of those is fine with me, and I can understand someone deciding to spend $yyy on this, though I’d argue it was a mistake since this is a great chance for an amazing amount of publicity and hype for their products.

    However there are certainly other problems that I question the level and detail devoted to testing. The overlay of foreign languages means that someone hadn’t configured their audio correctly, and more importantly, no one was monitoring to stop the problems quickly. The problems with the stream meeting devices might have been capacity, but crashing the browser? Either there’s a major QuickTime issue, or the encoding was broken in some way (perhaps again, by capacity).

    This is a shot at Microsoft as well, but I thought Tim Ford’s summary was spot on:

    applekeynote

    We talk about thorough testing on a copy of production. A real, scale, same sized version. Few of us do it, but certainly some do, especially those with big budgets. Apple has big budgets, so I suspect that their testing was cut short, or someone assumed things would work the same way they had in the last keynote, which went off without a hitch for me.

    Complacency, coarse attention to detail, poor quality control and monitoring seemed to be in play today. None of which as a good thing.

    Perhaps Apple was testing in production.