Tag: testing

  • Scripting Makes Mistakes Easier Than Ever

    A number of you likely use Atlassian products like Jira, Confluence, Opsgenie, or something else. You might have been affected by a large outage they had (post incident blog, Company Q&A, TechRepublic report) recently which lasted at least 9 days. I don’t know if all customers have their data back and are working, but this was a surprisingly poorly handled incident according to a number of reports from customers. There’s a great write-up from the outside that you might want to read.

    The bottom line in this issue is that Atlassian looked to deactivate a legacy product with a script, but they apparently didn’t communicate well among their teams. The script ended up using the wrong customer IDs and also marked the sites for permanent removal, not temporary removal (soft delete). While they supposedly test their restore capabilities, they weren’t prepared for partial restores of subsites. I’m guessing this is likely a partial database restore, which many of us know is way more complex than a full database restore.

    Leave aside the issue of a software-as-a-service (SaaS) company failing their customers, and the lack of communication with customers. The more interesting thing for me is the challenge of poor coding and communication internally. Clearly, the project to deactivate their legacy app wasn’t well planned or tested and the code used was probably executed at too wide a scale initially.

    When we deploy code changes to a large number of items, we want to test them at a small number first. Whether we are deploying to multiple databases, against many customers, or different systems, a standard method of making changes at scale involves working in rings. Azure DevOps describes this in docs, and they actually use rings to change the platform. We used the same pattern 20 years ago for software and database updates to many systems. We would internally deploy to a few users to look for issues. Then a week later we would deploy to a small number of systems to check for unexpected issues. Then typically to most systems in the third ring with a fourth ring a week later to catch up stragglers that needed more time to prepare.

    I find many customers, especially those with sharded/federated databases or many systems unwilling to spread out deployments in this manner. Often they yield to pressure from business users to ensure everyone gets the same update at the same time. I would never recommend this approach as we need to ensure we are looking at scripts in a controlled environment, or even two, before we deploy things widely. I’d be even more cautious about one-off administrative scripts that might make a change similar to the one Atlassian attempted. Those are often not seriously tested enough.

    At the very least, any of us working with multiple customers in a single database or in multiple databases ought to ensure we can backup and restore a single customer, but more importantly, can you restore a group of customers. If you make a mistake like Atlassian, which scripting allows us to do extremely rapidly, can you recover a partial set of data? Many of us don’t test this, but that’s likely something we ought to consider when we work with scripts that are designed to only change some data. Most of us don’t experience complete failures, but partial ones, usually because of human error. We ought to know how to deal with these situations.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • T-SQL Tuesday #125 Unit Testing

    tsqltuesdayThis month T-SQL Tuesday is hosted by Hamish Watson, quarantined down in New Zealand. Like me, he has a ranch, and kids, so he has space to move around a bit.

    Hamish is a DevOps and testing advocate, just like me. His topic this month is to ask “So now that I have introduced unit testing – is it valuable for implementing with databases? “

    I think it’s a great topic, and here is my take.

    Database Unit Testing Documents

    My big take is that database unit testing best documents what you, as a developer, think your part of the database code should do. You expect these things:

    • The API a table will provide
    • The way a CASE statement decides what to return
    • The formulas you use for aggregation
    • The manipulation a function uses to return a value

    And more. Ultimately, we interpret a requirement, whether written down or told to us. We often may slightly mis-interpret what someone told us, or not think of certain cases. A test forces us to document the way we think our code behaves with certain data.

    This isn’t perfect, as we still may not account for certain cases of data, we may have bugs, or we may interpret things incorrectly. However, the test documents what we think. When there is an issue, we can go look at the test and verify this behavior is what is expected. Or we can add a new test (or enhance ours) to handle the case we didn’t expect.

    There are plenty of other reasons, but I believe that testing helps improve quality because it forces us to think, it documents our actions, and it helps ensure we don’t create regressions of code that already works.

  • Code from Redgate Streamed

    I presented a session today at Redgate Streamed and did a number of demos. The code consists of a few flies, which I’ve made available here: BasicUnitTesttsqlt.zip.

    This consists of:

    • 00_tsqlt.sql – Creates a database and adds the tsqlt framework.
    • 01_GetTesting_tsqlt_Setup.sql – adds objects you need.
    • 10_SQT_SQLCop_Demo.sql – The standards test demo
    • 20_SQT_RefactorArticles_MetaData.sql – Testing a table API
    • 30_SQT_RefactorArticles_IsolateProcedure – Test a function and procedure separately
    • 40_SQT_RefactorArticles_Exceptions.sql – Testing edge cases and bad data
    • 50_SQT_RefactorGrouping.sql – Sample test class breakdown
    • 99_SQT_Cleanup.sql – cleans things up

    You do need to have the SQLCLR enabled, and you will enable TRUSTWORTHY for this test database.

    Any question, let me know.

  • Testing SQL in the Advent of Code

    I like participating in the Advent of Code each year, though my participation often varies wildly as life gets in the way. Still, trying to solve some programming challenges is a good way of practicing your skills. If you’re competitive, you can try and see how quickly you can solve things and get onto the leaderboard.

    One note, if you enjoy the challenges, support the cost of running the site. Sending $5 would make a difference to what I’m sure is a decent amount of effort and some costs. Plus, I’d certainly be happy to buy the author some sushi if I were sitting next to him, so why not send something during the holidays.

    This year’s challenge is over, but you can still work through the challenges. In my case, I’ve gone through a few and hope to get to more in a few spare moments.

    Testing Day 2

    One of the things I’ve done in the past is see a challenge and then start to write some code. I’ve worked through the puzzles in PoSh, Python, and SQL, sometimes all three. When I think I’ve solved it, I often enter a result, which is wrong, and then code some more, repeating as needed.

    This isn’t different from what I’ve done as an employee for a company, but I’ve also realized that the subtle design specification is sometimes mis-interpreted by me. In that case, I’ve essentially been bothering the “QA” people for no reason. It’s an application in this case, but still.

    It would be better to have inputs and outputs specified and checked by the computer, which is way better at checking than I am. I decided to set up test harnesses after Day 1 (which was really easy) for the problems. Here’s Day 2.

    Puzzle A

    The first part of Day 2 is a puzzle about letters, asking you to compute a checksum based on whether any letters are repeated. This isn’t a complex set of instructions, but it would be easy to make a mistake. Across any number of sets, a human might have problems verifying the actual results.

    Since the answer here is a single value, this lends itself to a test. I decided to start by creating a table and then loading the input data into the table. That’s something I often do, so the basics here were:

    CREATE TABLE dbo.Day2
    ( Boxnumber INT
    , boxid VARCHAR(100)
    )
    GO
    INSERT dbo.Day2 (boxnumber,boxid)
    SELECT  ca1.ItemNumber,
             ca2.Item
    FROM    OPENROWSET(BULK 'e:\Documents\GitHub\AdventofCode\2018\Day2\input.txt', SINGLE_CLOB) dt(FileData)
    CROSS APPLY dbo.Split(dt.FileData, CHAR(10)) ca1
    CROSS APPLY (VALUES(REPLACE(ca1.Item, CHAR(13), ''))) ca2(Item);

    Now that I had data, I can write a test. I like to use tsqlt, so I started there. Since I want something to test, I decided to start with a procedure that will hold my solution. Since I’ll code here, I can stub this out.

    CREATE OR ALTER PROCEDURE Day2a
    AS
    BEGIN
         DECLARE @i INT = 1;

    -- Solution goes here
     
    RETURN @i
    END

    With this set up, we can now build a test. The basic outline for a test is Assemble an environment, Act on your code, Assert your results. Let’s follow this template.

    The Assemble is easy. I’ll fake out my table of values and insert the test section from the calendar. I’ll also add the expected result, which is given in the puzzle as 12.

    CREATE OR ALTER PROCEDURE tsqltests.[test Day2a]
    AS
    BEGIN
         ---------------
         -- Assemble
         ---------------
         DECLARE
             @expected INT = 12
           , @actual INT;
         EXEC tsqlt.faketable @TableName = 'Day2', @SchemaName = 'dbo';
         INSERT dbo.Day2
             (
                 Boxnumber
               , boxid
             )
         VALUES
             (1, 'abcdef')
           , (2, 'bababc')
           , (3, 'abbcde')
           , (4, 'abcccd')
           , (5, 'aabcdd')
           , (6, 'abcdee')
           , (7, 'ababab');

    The Act part is easy. I’ll call my procedure and get the result back.

    ---------------
    -- Act
    ---------------
    EXEC @actual = dbo.Day2a;

    The Assert part is also easy. I’ll just compare my actual result to what I expected.

    ---------------
    -- Assert   
    ---------------
    EXEC tSQLt.AssertEquals
         @Expected = @expected
       , @Actual = @actual
       , @Message = N'An incorrect checksum calculation occurred.';

    Once this is done, I’ll run it and it fails because my stub proc returns 1. Now to code the solution, which I can easily check by running my test. I can verify things work with a first change to my procedure.

    CREATE OR ALTER PROCEDURE Day2a
    AS
    BEGIN
         DECLARE @i INT = 1;
    SELECT @i = 12
    RETURN @i

    GO

    EXEC tsqlt.run 'tsqltests.[test Day2a]';

    That’s it, and the solution is to split out the box IDs, count the letters, and where there are repeats, tally those up.

    Puzzle B

    The second part of the puzzle is always a nice twist on the first part. In this case, I get a new set of IDs, which vary by a single character.I need to pick those two box IDs and return the common ones. A new solution needed, but only a slight change to the test.

    First, we change the Assemble section because we have new results and inputs.

        ---------------
         -- Assemble
         ---------------
         DECLARE
             @expected VARCHAR(26) = 'fgij',
             @actual   VARCHAR(26);

        EXEC tsqlt.faketable @TableName = 'Day2', @SchemaName = 'dbo';
         INSERT dbo.Day2 (Boxnumber, boxid) VALUES
    (1, 'abcde'),
    (2, 'fghij'),
    (3, 'klmno'),
    (4, 'pqrst'),
    (5, 'fguij'),
    (6, 'axcye'),
    (7, 'wvxyz')

    Next, I need to change the ACT section. Since I can’t return a string from a procedure, I could use a function, but I’ll just add an OUTPUT parameter to my Act.

    ---------------
    -- Act
    ---------------
    EXEC dbo.Day2a @actual OUTPUT;

    Lastly, I change the proc.

    CREATE OR ALTER PROCEDURE Day2b
       @r VARCHAR(50) out
    AS

    That’s it.

    Good luck solving the puzzles.