Author: way0utwest

  • Using Aggregates in Calculations with Other Columns Functions–#SQLNewBlogger

    Another post for me that is simple and hopefully serves as an example for people trying to get blogging as #SQLNewBloggers.

    In the last post on Window functions, I looked at ROW_NUMBER, and how I can use this to order rows. In this one, I want to look at one of the advantages of Window functions in trying to combine data and aggregates together.

    A Scenario

    In the last post, I examined the career of Ken Griffey, Jr., showing his home runs with some ordering. That wasn’t a very realistic case of using data, so let’s look at another one. Suppose I want to know the percentage of his career home runs he hit during each one of his seasons. That’s an interesting question, showing some idea of how much he improved or declined. If I try a to start combining a “normal” aggregate with other data, I can’t do it without a GROUP BY.

    2021-07-19 15_27_35-window_queries.sql - ARISTOTLE.BaseballData (ARISTOTLE_Steve (57))_ - Microsoft

    Not a huge big deal, as I can put the sum in a CTE and use it later. Here’s the code:

    WITH cteHR (PlayerID, TotalHR)
    AS (   SELECT
                     b.playerID
                   , SUM (hr)
            FROM     dbo.batting AS b
            WHERE    b.playerID = 'griffke02'
            GROUP BY b.playerID)
    SELECT
                    b.yearID
                  , b.teamID
                  , hr
                  , TotalHR
                  , ROUND((hr * 1.0) / TotalHR * 100, 2) AS percentofCareer
    FROM
                    dbo.batting AS b
         INNER JOIN cteHR
             ON cteHR.PlayerID = b.playerID
    WHERE          b.playerID = 'griffke02'
    ORDER BY       b.yearID;

    And the results.

    2021-07-19 15_30_52-window_queries.sql - ARISTOTLE.BaseballData (ARISTOTLE_Steve (57))_ - Microsoft

    That’s OK, but the code is complex, and if I wanted to break the home runs by team or some other group, it would get more complex quickly.

    Window Functions Simplify Things

    Here’s a simpler query. I’ve just added the SUM with an OVER() clause, without any order. I just want all rows summed. I added this to the query to see the value.

    SELECT
              b.yearID
            , TeamID
            , HR
            , SUM (b.hr) OVER (ORDER BY (SELECT NULL)) AS TotalHR
            , ROUND ((b.HR * 1.0) / SUM (b.hr) OVER (ORDER BY (SELECT NULL))  * 100, 2) AS TeamPercentofCareer
    FROM     dbo.batting AS b
    WHERE    b.playerID = 'griffke02'
    ORDER BY b.yearID;

    The results are the same as the other query, but it’s easy to see.

    What if I wanted to change this and order this by the highest percentage years of his career. In other words, when was he the most productive. I can easily add an ORDER BY to both queries to see this, but I lose some context.

    Look at these results. How do I know if 1997 was closed to the beginning or end of his career?

    2021-07-19 15_35_33-window_queries.sql - ARISTOTLE.BaseballData (ARISTOTLE_Steve (57))_ - Microsoft

    I want to add some context with the span of his career. I can do that easily with a few more Window functions. Here’s the result I want, with the years showing his career first. I moved some of the other data to the end.

    2021-07-19 15_38_57-window_queries.sql - ARISTOTLE.BaseballData (ARISTOTLE_Steve (57))_ - Microsoft

    Without window functions, this would be complex, as the MIN() and MAX() would be from different columns, so I’d need another CTE. Here, I can use this code:

    SELECT
            CAST(MIN (b.yearID) OVER (ORDER BY (SELECT NULL)) AS CHAR(4)) 
            + '-'
            + CAST(MAX (b.yearID) OVER (ORDER BY (SELECT NULL)) AS CHAR(4)) AS CareerSpan
            , b.yearID
            , ROUND ((b.HR * 1.0) / SUM (b.hr) OVER (ORDER BY (SELECT NULL))  * 100, 2) AS TeamPercentofCareer
            , TeamID
            , HR
            , SUM (b.hr) OVER (ORDER BY (SELECT NULL)) AS TotalHR
    FROM     dbo.batting AS b
    WHERE    b.playerID = 'griffke02'
    ORDER BY TeamPercentofCareer desc;

    If I wanted to add some math, like how many years into his career was he, I could easily do that. Here I’ve added the year number to his career, which comes from ROW_NUMBER().

    2021-07-19 15_42_31-window_queries.sql - ARISTOTLE.BaseballData (ARISTOTLE_Steve (57))_ - Microsoft

    The code? I just add the aggregates I need, which in this case are ones containing the entire set. I mix MIN(), MAX(), COUNT() and ROW_NUMBER(), and use a partition of the entire data set.

    SELECT
            CAST(MIN (b.yearID) OVER (ORDER BY (SELECT NULL)) AS CHAR(4)) 
            + '-'
            + CAST(MAX (b.yearID) OVER (ORDER BY (SELECT NULL)) AS CHAR(4)) AS CareerSpan
            , b.yearID
            , ROUND ((b.HR * 1.0) / SUM (b.hr) OVER (ORDER BY (SELECT NULL))  * 100, 2) AS TeamPercentofCareer
            , RTRIM(CAST(ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) AS CHAR(2)) )
            + ' of ' 
            + CAST( COUNT(yearid) OVER(ORDER BY (SELECT NULL)) AS CHAR(2))
            AS YearInCareer
            , TeamID
            , HR
            , SUM (b.hr) OVER (ORDER BY (SELECT NULL)) AS TotalHR
    FROM     dbo.batting AS b
    WHERE    b.playerID = 'griffke02'
    ORDER BY TeamPercentofCareer desc;

    Try doing that without window functions. It’s a nightmare to write in T-SQL.

    SQLNewBlogger

    This was pretty easy to write. The hard part was thinking of the questions I might ask of this data set and setting up the queries. Duplicating this without window functions was fun, and took more time. But it was good practice for me, and helped me to better understand why I like window functions.

    This took me about 30 minutes, and it’s a good showcase of learning a new technique and applying it. You should do this if writing reports and aggregates is part of your job and you might want to showcase this to your next potential employer.

  • The Danger of Management Access

    First we had the Solarwinds hack, and now we have a Kaseya ransomware epidemic. It seems the criminals are moving up the stack. We used to see physical attacks on tapes and keyboards, then we saw OS level attacks. Now we seem to be getting to the management layer for software that is used to help us run systems at scale. Since we often require some level of privileged access for monitoring and management systems, this is scary. I certainly wish that we didn’t require admin access for monitoring, but unfortunately platforms sometimes do.

    Many of us depend on some standardization and some sort of software to ensure we can manage systems at scale. I don’t know about the OS world, but in the SQL Server world, there are relatively few vendors that provide software for managing systems. If one of these were compromised in some way, this could be very bad for many database administrators. Fortunately, many of us know how to air gap backups and ensure that we are prepared for disasters.

    Or we should. If you don’t know how to do this, you ought to be learning right away. Review backup plans, ensure you can rebuild systems, test restores, and brush up all your recovery skills. Be ready for whatever a criminal might throw at you, including having gotten ransomware into some of your backups.

    This attack seems to have taken advantage of a zero day, or very early, vulnerability that was discovered by a Dutch security research firm. The firm looks into management software, especially admin interfaces, specifically because they are worried about the lack of security in many products. In this case, Kaseya builds tools that allow admins to distribute software to other systems on the network. In this case, criminals used the management software to distribute ransomware.

    The updates from the Kaseya are less than stellar, and if I were a customer, I’d be rather upset. They seem to keep setting unrealistic plans to restore service and then constantly revise them across a few days, all the while with customers that are likely stressed and overworked. I’d also be upset in that they claim only a few of their thousands of customers are affected, but they neglect to admit that some of those customers affected as Managed Service Providers, who themselves have thousands of customers using this software.

    There are some technical  details in this piece, in case you want to check your own systems. If you think you have multiple pieces of software that might protect you, read the article. This deployment shuts off some other products, like Microsoft Defender.

    I feel bad for many people here. IT staff at affected companies that have likely been incredibly stressed and overworked recently. The consumers of some affected customers, like those that might shop in the Swedish grocer, Coop, who shut down more than 400 stores. I don’t know the state of grocery shopping in Sweden, but this might dramatically impact many people that just want to buy food for their families.

    Ransomware continues to surprise and worry me. Large profile hacks keep coming, affecting lots of people. Often these are because of previously undiscovered software vulnerabilities or simple mistakes made by privileged users. I hope that at some point insurers and governments start to put more pressure on companies that make widely used software to ensure they are adhering to best practices and have detailed security practices in place to ensure their code is constantly checked for issues, and that they have detailed plans for responding to and patching customers when there are issues. Because, they likely will have an issue at some point.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes

  • Daily Coping 21 Jul 2021

    I started to add a daily coping tip to the SQLServerCentral newsletter and to the Community Circle, which is helping me deal with the issues in the world. I’m adding my responses for each day here. All my coping tips are under this tag. 

    Today’s tip is to pick one of your strengths and use it this week.

    The easy joke is that I used my pectorals and did some weightlifting. However, this week I picked a different strength. This week I used: people pleasing and support.

    My wife had a dressage competition this week. We ended up driving to a show and living in a camper for a few days, during some of which I worked. A show is stressful, both for her and the horse, and they have a schedule they’re trying to stick to. It was also hot, and limited amenities around.

    Instead of trying to maintain my schedule, find time to work out, etc., I decided to be completely supportive and help my wife with her schedule. I didn’t worry about my workouts, I moved my work around her competition times, and I was ready to jump in and tackle tasks she asked for or needed, subordinating what I wanted to thought I could get done.

    That’s a little stressful, as I try to still work, but I let many things go, relaxed where I could, and made the show the priority.

    It worked out well, and now I’m back into a normal schedule for me, hitting the gym and worrying about my own chores. She’s happy, too. She qualified for a season ending show, and felt the trip was a success.

  • Daily Coping 20 Jul 2021

    I started to add a daily coping tip to the SQLServerCentral newsletter and to the Community Circle, which is helping me deal with the issues in the world. I’m adding my responses for each day here. All my coping tips are under this tag. 

    Today’s tip is to pick an achievable goal for the week and make progress.

    I actually did this last week. I have had a lot of things on my list of things to do this year, and sometimes I get a little paralyzed by the length of the list. There are also some larger projects that I find myself putting off because I keep thinking “I’ll take a day” or “Take a weekend” to do this.

    The reality is that I never will likely get a large block of uninterrupted time, when I feel ready to tackle things.

    I got up, had some breakfast, and then went outside recently. There were a few other things I could do, but I started with a chainsaw and chopping down some juniper bushes in our front yard. My wife has been asking to do this, and I keep delaying rather than getting started.

    20210703_122540

    I got one out, taking a few breaks as I tried to cut down the tangled limbs and drag them to a nearby gully. My breaks were on the lawnmower, mostly because I needed to get that done as well.

    The next day I went out and tackled a second one. I managed to get down to two stumps.

    20210704_144805

    I took a few days off, but then spent about an hour one night working on the third one. I didn’t get it out, but I made more progress.

    20210711_202929

    Baby steps were helpful to me moving forward.