Author: way0utwest

  • SQL Grouping on Sums (with testing)

    I ran across a post recently that I thought was an interesting T-SQL problem. The user wanted to group values into a running total, but the groups would reset based on a sum.

    In this case, the user had this set of data:

    2017-09-14 18_09_39-SQLQuery1.sql - (local)_SQL2016.sandbox (PLATO_Steve (72))_ - Microsoft SQL Serv

    Their goal was to run through these values, in Category order, and whenever the running total sum of SomeValue exceeded 30, reset the sum. Their requirement was that this could only be a single value or two values, which boxes in the problem nicely. In other words, they wanted these results:

    2017-09-14 18_12_02-SQLQuery1.sql - (local)_SQL2016.sandbox (PLATO_Steve (72))_ - Microsoft SQL Serv

    The first two rows equal 30, so we reset for the third row. The third row is 30, so we reset for the fourth. The fourth and fifth would exceed 30, so each gets a reset. Five and six give 29, so we stop there.

    I don’t know what the use case is here, but it’s an interesting problem.

    My Solution

    I had a quick solution using Lag. I created a quick query that looked back 1 and 2 rows. I could have stopped with one, but originally I thought that the poster might go to three rows if the 30 value wasn’t met. I use a CTE to get the current row and previous values, then  a simple CASE to sum values or return the current row.

    WITH lagCTE
    AS (SELECT
              Category,
              SomeValue,
              LagValue1 = LAG(SomeValue, 1, 0) OVER (ORDER BY Category),
              LagValue2 = LAG(SomeValue, 2, 0) OVER (ORDER BY Category)
         FROM Source
        )
    SELECT
          lagCTE.Category,
          lagCTE.SomeValue,
          Sums = CASE
                     WHEN lagCTE.SomeValue + lagCTE.LagValue1 > 30 THEN
                         lagCTE.SomeValue
                     ELSE
                         lagCTE.SomeValue + lagCTE.LagValue1
                 END
    FROM lagCTE;

    I also created a test, because, why do the math. Once I’ve done this, I want to ensure any code changes, any logic changes will still pass the same test. Here’s my test code:

    EXEC tsqlt.NewTestClass @ClassName = N'tTSQLTests'
    GO
    CREATE PROCEDURE tTSQLTests.[test running total reset]
    AS
    -----------------------------------
    -------   Assemble
    -----------------------------------
    EXEC tsqlt.FakeTable
         @TableName = N'RTSource'
    
    INSERT RTSource
    VALUES ('101', 10),
            ('102', 20),
            ('103', 30),
            ('104', 12),
            ('105', 19),
            ('106', 10),
            ('107', 10);
    
    CREATE TABLE tTSQLTests.Expected
    (   Category     VARCHAR(5),
         SomeValue    INT,
         RunningTotal INT
    );
    INSERT INTO tTSQLTests.Expected
    VALUES
           ('101', 10, 10),
           ('102', 20, 30),
           ('103', 30, 30),
           ('104', 12, 12),
           ('105', 19, 19),
           ('106', 10, 29),
           ('107', 10, 10);
    SELECT
           Category,
           SomeValue,
           RunningTotal
    INTO  tTSQLTests.Actual
    FROM  tTSQLTests.Expected
    WHERE 1 = 0;
    
    -----------------------------------
    -------   Act
    -----------------------------------
    INSERT tTSQLTests.Actual EXEC RunningTotalQueries
    
    -----------------------------------
    -------   Assert
    -----------------------------------
    EXEC tsqlt.AssertEqualsTable
         @Expected = N'tTSQLTests.Expected',
         @Actual = N'tTSQLTests.Actual',
         @Message = N'incorrect query'
    GO

    Adding Counters

    The poster then asked for a group counter, which becomes much harder. I was about to try for another CTE that would give me some counter I could work with when Jeff Moden used the quirky update to build a better script. You can read his code here.

  • Cookie Consent

    More of an FYI, but the EU Cookie Consent law goes into effect on Oct 1. Maybe this matters to you, maybe not. In my case, I may be in limbo as I work for a UK company even though I live in the US. Certainly I want EU visitors to my sites, so I decided to go ahead and comply.

    For tsqltuesday.com, I went into the Plugins section of the site and clicked “Add New”. Then I had a list of plugins, but I entered “Cookie”” and searched. I got this:

    2017-09-22 17_35_57-Add Plugins ‹ T-SQL Tuesday — WordPress

    I choose Cookie Consent and added it to the site. The default activation is at the top of the site, and that’s fine. I tested this with a Private browser window and I see the notice. I’m not going to mess with options here, just let you know there are cookies.

    For voiceofthedba.com, this is hosted by wordpress.org. I did that since it was easy and I don’t need to manage WP for the blog. Here, I went to the Appearance menu and then Widgets. In this section, WordPress have provided the EU Cookie Law Banner. That’s fine with me, and I added it to the First Footer Widget area.

    2017-09-22 17_38_33-Widgets ‹ Voice of the DBA — WordPress

    Again, not messing with settings, and I confirm this is at the bottom of the page.

    2017-09-22 17_39_05-Voice of the DBA _ Writings from Steve Jones, the Voice of the DBA

    Whatever you think of this law, I decided compliance was easy and not a big deal. I’ve been seeing this on more and more sites, so it may be just something to get used to. Up to you, but I’d add this to my blogs if I were you.

  • A SQL Server Bug and Data Security

    This week I saw a post from Microsoft Tiger Team on the issues with backup compression and TDE databases. Apparently when they added compression, they didn’t test a few edge cases. Or a few regular cases, like WITH INIT. There is a possibility that your data could be corrupted, which is a major problem. I have found the SQL Server backup and restore capabilities to be very solid over the years, so this is disturbing. If you use TDE, apply SQL 2016 CU7 for RTM or SP1 CU4 ASAP. Test your restores, and be sure you know what you can recover and what you can’t. If Mr. Murphy has anything to say about it, you’ll have an issue soon, so test your restores.
    Overall, SQL Server security is very good, but there are certainly issues with applications and devices that connect to SQL Server. You never know when some item that queries or alters data in SQL Server will cause issues. This week there were a couple notes from Bruce Schneier on FaceID and Bluetooth security, the latter of which has a vulnerability issue. Be sure you are aware of issues here to actually help protect yourself with your devices, but I was amazed to see this piece on infrared camera hacking. A POC on using light to jump air gaps is truly frightening. It seems that truly anywhere that we are processing data, we need to be thinking about security.
    The last few weeks are especially scary for many people, particularly with the Equifax breach. I know they have released numbers on people’s data, but I’d assume that everyone who has ever had credit in the US has a potential issue. I know I am being careful with credit and watching for issues myself. You should, too, and demand better security from companies you do business with. We can improve systems, but it will take more pressure to get companies to put more emphasis on better software and security.
    Steve Jones
  • How Mature are you in Database DevOps?

    I remember seeing the Carnegie Mellon Software Capability  Maturity Model (CMM) when I was in university. It was fascinating, and I was sure this was the way to write software. Across many jobs and many years, I realized that few organizations even try to become more efficient and capable in how they write software.

    That’s changed a bit in the last 4-5 years as more organizations try to move to DevOps and become better at building software. Some do well, some just want to build software faster and not change the way they work.

    In any case, Redgate has built a maturity model for Database DevOps. You can take the assessment now in a few areas and get an idea how you stack up against other companies.

    Benchmark Your Database DevOps maturity level today