Author: way0utwest

  • ROLLing UP Totals–#SQLNewBlogger

    Another post for me that is simple and hopefully serves as an example for people trying to get blogging as #SQLNewBloggers.

    I was editing an article recently that talked about ROLLUP, and I wanted to play with this a bit more. I hadn’t used this much in my career, but it’s a handy operator that’s worth including in your toolkit.

    ROLLUP is used to provide additional totals for your aggregates while using GROUP BY. Here’s a little example. I’ve got some data for American Football quaterbacks. In this case, I’ve extracted some stats for a few noteworthy players today. Here’s a sample:

    2017-10-16 13_05_05-SQLQuery1.sql - (local)_SQL2016.NFLAnalysis (PLATO_Steve (74))_ - Microsoft SQL

    I want to aggregate some data together. For example, let’s say that we want to look at the total touchdowns scored per year by these players in their first 5 years. That’s a simple query:

    SELECT
       Season = qs.CareerYear
       , TDs = SUM(qs.Touchdowns)
      FROM dbo.QBStats AS qs
      WHERE qs.CareerYear <= 5
    GROUP BY qs.CareerYear

    This gives me data like this:

    2017-10-16 13_06_28-SQLQuery1.sql - (local)_SQL2016.NFLAnalysis (PLATO_Steve (74))_ - Microsoft SQL

    However, when I look at this, I don’t have a total across all years. In my set, there are players that have played up to 18 seasons, and I want to know the aggregate number of TDs. I can get that by adding WITH ROLLUP. This is added

    2017-10-16 13_07_25-SQLQuery1.sql - (local)_SQL2016.NFLAnalysis (PLATO_Steve (74))_ - Microsoft SQL

    after the group by. However, I can also change this to GROUP BY ROLLUP and include the columns in parenthesis.

    SELECT
       Season = qs.CareerYear
       , TDs = SUM(qs.Touchdowns)
      FROM dbo.QBStats AS qs
      WHERE qs.CareerYear <= 5
      GROUP BY ROLLUP (qs.CareerYear)

    This is OK, but the NULL isn’t great. What can I do here? I can add an ISNULL or COALESCE to my query and get this:

    2017-10-16 13_10_04-SQLQuery1.sql - (local)_SQL2016.NFLAnalysis (PLATO_Steve (74))_ - Microsoft SQL

    Useful, but what about if I’m aggregating by individual players? I can add the player to the column list, but I also need to add the player to the ROLLUP (or GROUP BY) list as well. If I do that, I get this:

    2017-10-16 13_11_40-SQLQuery1.sql - (local)_SQL2016.NFLAnalysis (PLATO_Steve (74))_ - Microsoft SQL

    That’s quite a bit more data, and I now see NULL values in the Player name. What’s this?

    These are the totals for that subgroup. In this case, the first NULL above, next to the 1, is the total for all players for season 1. I can use another trick to clean this up:

    2017-10-16 13_14_33-SQLQuery1.sql - (local)_SQL2016.NFLAnalysis (PLATO_Steve (74))_ - Microsoft SQL

    This now shows some data in an easier to understand fashion, with some totals. We can see for season 1 that 3 players really started their careers in a strong fashion and 3 didn’t.

    At the end, however, we get a different result.

    2017-10-16 13_17_09-SQLQuery1.sql - (local)_SQL2016.NFLAnalysis (PLATO_Steve (74))_ - Microsoft SQL

    This is because we have handled the player name as a NULL for the second column, but not the season as a NULL. For this last row, out “Total” text comes from an ISNULL of the first column. Here we’d want to do this:

    2017-10-16 13_18_30-SQLQuery1.sql - (local)_SQL2016.NFLAnalysis (PLATO_Steve (74))_ - Microsoft SQL

    This is where COALESCE  comes in handy, allowing us to account for multiple NULL values.

    ROLLUP is a nice way to get totals for each of your GROUP BY columns. This can make some of your reporting easier, and perhaps even faster by having the totals calculated in the result set.

    SQLNewBlogger

    This was a fairly easy piece to write. It took longer to get data together and find a set that had two grouping elements and was interesting than write the queries. I also had to play with ROLLUP a bit to clean up the NULLs, which always made this seem like a useless operator.

    I think I’d look to use this more if I had the chance for many reports, as summing totals in some client tools is a pain.

  • DevOps Basics – Ignoring Files in Git

    Another post for me that is simple and hopefully serves as an example for people trying to get blogging as #SQLNewBloggers. This is also a part of a basic series on git and how to use it.

    One of the things you’ll run into at times is the need to keep some scratch files, or extra files, in your Git repository, but not track them. One common type of file for me when working with SQL is a .zip file. I may zip up code to share or copy to a friend (without giving repo access).

    Ignoring files is easy in Git. We just add a .gitignore file. This is a list of files that the git repository will not track or show in status. Essentially, we see them in our file system, but git doesn’t.

    Creating .gitignore

    To create a .gitignore file, the easiest method for me is to just create a text file. I can do it like this:

    2017-10-09 16_36_08-cmd

    This gives me a new file. Certainly VS Code, Sublime, etc. will make this easy as well.  The format is simple, with a list of files and/or patterns to ignore. For example, I’ve got a .zip file in my repo.

    2017-10-09 16_34_24-GitTests

    I don’t want to see this, but I do right now:

    2017-10-09 16_37_31-cmd

    If I want to ignore this file, I’ll enter this in my .gitignore file:

    GitTests.zip

    If I want to ignore all zips, I’ll do this:

    2017-10-09 16_38_09-cmd

    This is a part of my repo, so I need to commit it.

    2017-10-09 16_38_32-cmd

    Now my status is clean.

    2017-10-09 16_39_14-cmd

    Generated .gitignore

    Some applications will generate a .gitignore. For example, my C# project gets this file from Visual Studio.

    2017-10-09 16_40_33-.gitignore — Visual Studio Code

    That’s a subset of files that are often in a VS project, but we don’t want to track in a VCS. Images, archives, executables, etc.

    You can customize this as you need, and it’s easy to just edit the text file and commit the changes.

    Hopefully this helps you understand how to best work with git and keep your repo clean. This also means your  git add –all is easy to use without adding unnecessary files.

  • Prepping for Summit 2017

    It’s about time for the PASS 2017 Summit. The event essentially starts on Monday with pre-cons and the unofficial networking dinner. Be sure you RSVP and come to the dinner if you don’t have plans. The official event is

    Many people are already traveling and packing for the event. I feel a bit behind as I won’t leave until late Monday afternoon and arrive late Monday night. I’m sure I’m not alone, but it seems like everyone’s ready and I’ve still got a day of work Monday.

    Redgate Software has a booth and a few presentations next week on Wednesday. We’d love to chat with you about ways to make database development easier, especially if you’re thinking DevOps. We also have some contests, swag and prizes.

    I’ll be around Tues, Wed, and Friday, and of course, moving from Game Night to the Redgate Party Thursday night. I’ve got other commitments Thursday day, but hope to see lots of you there.

  • Physical or Virtual Storage

    When I started working with SQL Server, every server had what we’d call das-dee, or DASD (Direct Attached Storage Devices). These were hard drives inside the same physical case as the rest of the Windows computer. I’ve added lots of drives to various server systems over the years. As databases grew, we even had separate boxes in our racks that were attached to the main server, but only filled with drives.

    Technology has changed, and today most of us work with SAN or NAS devices, where the storage is addressed across some type of network. Either a private one (copper or fiber), or the same Ethernet that connects the various computers together. A few of us might even have cloud storage that is located at Microsoft, Amazon, or elsewhere. The Stretch Database feature takes advantage of this last configuration. In all these cases, the storage that our databases see is often cobbled together from other disks that hide the underlying organization from the system.

    Recently I read a piece from Randolph West that talked about recovering data from a RAID array. That reminded me of my early career, where I had to make decisions about how to structure storage. I’ve run RAID 1, 5, 10, 0+1, 6, and maybe more in my career to store data files. However, at some point I stopped worrying about the underlying configuration. I just expected, and trusted, the storage people to ensure that space was available. I even stopped thinking of the z: or y: drives on my database server as disks. Those drives were just storage that existed somewhere in the ether, just available for the database to use.

    In thinking about Randolph’s experiences, I wondered how many of you out there might still deal with physical drives. Do you still make decisions about RAID levels? Do you even know what RAID levels are being used by your databases? If you’re a storage admin, you might, but for those of you that aren’t, do you know anything about your storage configuration?

    Really, I’m speaking of production systems, not development ones. Certainly many of us might know there’s a development server with RAID 5 that holds a bunch of dev/test VMs, but I would expect that might even be rare. Outside of our own workstation, we likely don’t know the storage setup. Plenty of development systems these days probably even use a SAN, maybe even the same one as production, for storage.

    For me, I have no idea of our systems. I used to build the SQLServerCentral servers, and when Redgate took over that part of the business, I helped spec the initial machines we rented as physical hosts. At some point we moved to virtual machines, and while I was asked about the specifications, I didn’t care about any of the hardware. I just said that I wanted enough CPU, RAM, space, and IOPS to handle the load. Deciding what that was, and ensuring it was available, was someone else’s job.

    If you spec hardware, or pay attention, let me know. There certainly are plenty of hardware geeks, like Glenn Berry, that pay attention and prefer particular configurations. Those are the people I’m glad I can ask for advice if need it. I certainly ask for help with my personal systems, but for servers, I just need capacity. Do you feel the same way?

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.7MB) podcast or subscribe to the feed at iTunes and Libsyn.