Category: Blog

  • Redgate Lives DevOps

    At Redgate, we release a lot of changes to our products. In fact, this is the “About Redgate” slide I’ve been using in talks related to the company.

    2018-08-23 09_36_29-ReduceAttackSurfaceArea.pptx - PowerPoint

    If you look in the lower right, you’ll see product releases from last year (2017). We have 30-ish products, so that’s roughly 38 releases per product. Not every product releases at this cadence, but a lot of them release every week. SLQ Monitor, for example, releases every Wednesday.

    Of course, there’s the inevitable release-a-bug-and-need-to-fix-it-so-a-second-release this week, but those don’t happen too often. I’ve been tracking releases this year, and not too many were corrected in the same week, but it does happen.

    Some people say that’s the problem with DevOps, but I think it’s the advantage. I guarantee that most software releases include bugs. If you release once a quarter, are you ready to re-release in a couple days to fix something? Or do customers live with issues for a quarter? The advantage of DevOps is we can fix things quickly, in addition to adding new features quickly.

    Redgate does some amazing development work and I’m proud of the ladies and gentleman that write the code.

    I still complain, and there’s room for improvement, but they do a great job and I try to remember to thank them and complement their work when I do see them.

  • Generating a Constrained Random Date–#SQLNewBlogger

    Another post for me that is simple and hopefully serves as an example for people trying to get blogging as #SQLNewBloggers.

    There have been lots of posts on the topic of generating random values, and some great articles. One of my favorites is Jeff Moden’s Generating Test Data: Part 1 – Generating Random Integers and Floats. Part 2 deals with dates, and that’s actually what I needed, but really I needed part 1.

    In my situation, I was helping a customer generate some random data. They had filled a table, Customers, with some data.

    2018-08-24 13_05_44-Microsoft Edge
    The goal was to populate a child table with some data. The child table had a date column that was supposed to be between the Entered and Exit dates in the Customer table.

    My update would have a join, obviously, and I can reference the enter and exit date, but how to get a date between them? My first thought was that I wanted a DATEADD() function. Something like this:

    UPDATE ce
    SET ce.EventTimeStamp = DATEADD( MINUTE, SomeRandomValue, c.CustomerExitedDateTime)), c.CustomerEnteredDateTime)
    FROM   dbo.Customer AS c
    INNER JOIN dbo.CustomerEvent AS ce
    ON ce.CustomerID = c.CustomerID

    The trick is what random value to use? If you look through Jeff’s article, you will see that the trick is to use a tally table and the NEWID() function. However, this doesn’t work:

     UPDATE ce
    SET ce.EventTimeStamp = DATEADD( MINUTE, NEWID(), c.CustomerEnteredDateTime)
    FROM   dbo.Customer AS c
    INNER JOIN dbo.CustomerEvent AS ce
    ON ce.CustomerID = c.CustomerID
    ;

    What I need to do is convert the GUID to a number. In this case, I added CHECKSUM around it, again, as in Jeff’s article. Then use ABS() to enclose this to get all positive numbers.

     UPDATE ce
    SET ce.EventTimeStamp = DATEADD( MINUTE, ABS(CHECKSUM((NEWID())))), c.CustomerEnteredDateTime)
    FROM   dbo.Customer AS c
    INNER JOIN dbo.CustomerEvent AS ce
    ON ce.CustomerID = c.CustomerID
    ;

    This gives me values, but they aren’t constrained. What I need to do is limit the upper random value so that the end time doesn’t exceed the Customer.CustomerExitDateTime for that row.

    To do this, I can constraint a large set of numbers to some value with the modulo function. This will limit what values can appear. The basic script is this:

    UPDATE ce
    SET ce.EventTimeStamp = DATEADD( MINUTE, ABS(CHECKSUM((NEWID())))) % 10, c.CustomerEnteredDateTime)
    FROM   dbo.Customer AS c
    INNER JOIN dbo.CustomerEvent AS ce
    ON ce.CustomerID = c.CustomerID
    ;

    This would give me values between 1 and 0 minutes after the start time, but this doesn’t mean these values won’t be after the exit time. This is also an unrealistic window if most of the time the enter and exit times vary by hours.

    What I did instead was to use the difference between the enter and exit times, with DATEDIFF() as my modulo function. That gives me:

    WITH myTally (n)
    AS
    -- SQL Prompt formatting off
    (SELECT n = ROW_NUMBER() OVER (ORDER BY (SELECT null))
      FROM (VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10)) a(n)
       CROSS JOIN (VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10)) b(n)
    )
    UPDATE ce
    SET ce.EventTimeStamp = DATEADD( MINUTE, ABS(CHECKSUM((NEWID()))) % (DATEDIFF(MINUTE, c.CustomerEnteredDateTime, c.CustomerExitedDateTime)), c.CustomerEnteredDateTime)
    FROM   dbo.Customer AS c
    INNER JOIN dbo.CustomerEvent AS ce
    ON ce.CustomerID = c.CustomerID
    ;

    I run this, and I get the table updated with a random set of values.

    2018-08-24 13_19_08-Microsoft Edge

    SQLNewBlogger

    This was a problem in my daily work. It was a customer, but it could easily be an internal query problem. I spent about 10 minutes grabbing screen shots and taking apart the query I’d built.

    You can do this, too. Show us your mind working with the solutions you write in your own blog.

  • Bringing DevOps to #SQLSat Oslo

    2018-08-16 12_50_39-SQLSaturday #746 - Oslo 2018 _ Event Home

    As my kids get older and I spend less time with them, I look forward to visiting new countries. While I’ve been to Norway and Oslo before, I wasn’t able to make the SQL Saturday last year. I am lucky that they accepted my submission and I’m heading  SQL Saturday #746 – Oslo in a week.

    There’s a great schedule with some amazing sessions to watch. I’m on it with my Bringing DevOps to the Database talk, showing you the process and idea behind adding your database development to an application DevOps pipeline.

    I’m also looking to try and get to Oslo early Friday and spend a little time hiking somewhere and enjoying the beautiful space. It’s on my list to travel up the West side of the country some day, and even get to Svalbard, but that will have to wait for a less busy time when my wife can join me.

    If you’re near Oslo, come to an amazing event and enjoy a little time in a little city by the sea.

  • Remember the Default Window

    I ran across a question recently from a user about why they had strange results from a windowing query. This is better explained with an example, so let’s look at one.

    I have some data in a table. This is NFL data, and a sample of it looks like this:

    2018-08-22 18_57_25-SQLQuery1.sql - Plato_SQL2016.NFLAnalysis (PLATO_Steve (52))_ - Microsoft SQL Se

    What I want to do is compare the passing yards each year with the most current value for that player, showing the plus or minus. This means that for Aaron Rodgers, who threw for 1675 yards in 2017, I’d want to show this for the first few years of his career:

      PlayerName  NFLYear PassYards Most Recent Yards Difference
    ------------- ------- --------- ----------------- -----------
    Aaron Rodgers 2005 65 1675 -1610
    Aaron Rodgers 2006 46 1675 -1629
    Aaron Rodgers 2007 218 1675 -1457
    Aaron Rodgers 2008 4038 1675 2363

    This shows
    me an easy view of the years where he was better in his career than he is now. Last year was likely a down year because of injury, but we’ll see this year.

    In any case, if I run this query using LAST_VALUE() for the final year of his career, I don’t get the right results.

    2018-08-22 19_11_16-SQLQuery1.sql - Plato_SQL2016.NFLAnalysis (PLATO_Steve (52))_ - Microsoft SQL Se

    It seems as though in every row, I’m getting the current row as the last value, not the last value of the partition. My partition is by player, so I should only have a window for each player. In this case, I should have the years 2005-2017 for Aaron Rodgers. My ordering is by year, so the last value should be 1675.

    Why isn’t it?

    The reason has to do with the framing. As the window is consumed, the default values for the framing are between

    • start – unbounded preceding
    • end – current row

    That means the first row for 2005 has the range of 2005-2005. The preceding rows are this row, and the current row is this row. For 2006, we have the first row as 2005 and the current row as 2006. The last value in this case is 46.

    What we need to do is specify the entire window if we want that. In this case, we could use the current row as the start, but we certainly need the unbounded following rows.

    2018-08-22 19_18_30-SQLQuery1.sql - Plato_SQL2016.NFLAnalysis (PLATO_Steve (52))_ - Microsoft SQL Se

    This is a common mistake when writing window queries. I’d recommend you always include the partition and the framing to avoid any issues.