Tag: syndicated

  • Adding Row Numbers to a Query: #SQLNewBlogger

    I realized that I hadn’t done much blogging on Window functions in T-SQL, and I’ve done a few presentations, so I decided to round out my blog a bit. This post will start with the ROW_NUMBER() function as a gentle intro to window functions.

    Another post for me that is simple and hopefully serves as an example for people trying to get blogging as #SQLNewBloggers. This is also part of a series on Window Functions.

    A Basic Set of Data

    I’m going to use some fun data for me. I’ve been tracking my travels, since I’m on the road a lot. I’m a data person and part of tracking is trying to ensure I’m not doing too much. Just looking at the data helps me keep perspective and sometimes cancel (or decline) a trip.

    In any case, you don’t care, but I essentially have this data in a table. As you can see, I have the date of travel, the city, area, etc. I also have a few flags as to whether I was traveling that day, if I spent a night away from home, and how far I was.

    2024-09_0199

    I have a travelID in here, which is a sequence, but what if I wanted to the trips I took in August 2024. I’d want a distance > 0 (not at home) and filters by dates. Adding that to my query, I’d run this:

    SELECT
       TravelID
    , TravelDate
    , TravelCity
    , Area
    , Province
    FROM travel
    WHERE
       TravelDate     > '2024/07/31'
       AND TravelDate < '2024/09/01'
       AND Distance > 0
    ORDER BY TravelDate;

    This gives me these results:

    2024-09_0202

    There are 9 rows in here, but they have a weird ID number, plus these are different trips. I can just add a row_number to this data, and I’d see this result. Ignore the OVER and track I used, but you can see an incrementing number added to each row. The second column in the result set matches with the number added by SSMS on the side.

    2024-09_0204

    What if I wanted to see the separate trips with some row number for the day in each city?

    That’s where a row_number() can help.

    Creating a Window

    The window comes from the OVER() clause, which is added to a number of functions, including Row_number(). The OVER() clause lets me set a window or rows on which the function works. I can set a partition and an order.

    The partition is a column where we are essentially grouping data. For me, this would be the city. When I change city, I want to reset the number. Looking at the data above, I’d expect to see 1, 2 for the first 2 days in Minneapolis, then a 1 for a day in Fort Collins, and another 1 for day in Aurora

    The ordering is what order is the data in the partition. In this case, I want to have the data in the window ordered by traveldate, so I’ll use that. I now have this code:

    SELECT
       TravelID
    , ROW_NUMBER () OVER (PARTITION BY TravelCity
                           ORDER BY TravelDate)
    , TravelDate
    , TravelCity
    , Area
    , Province
    FROM travel
    WHERE
       TravelDate     > ‘2024/07/31’
       AND TravelDate < ‘2024/09/01’
       AND Distance > 0
    ORDER BY TravelDate;

    And I get these results, where I can see that I essentially had 4 trips (all with number 1s), and these were the trips:

    • Minn – 2 days
    • Fort Collins – 1 day
    • Aurora – 1 day
    • New York City – 5 days

    2024-09_0205

    This shows how row_number() gives me a sequence based on the partition. The select null part in the earlier query just ignores the order by, which is required for row_number(). With no order, how do we know what sequence?

    Let’s change this slightly. What if I partition by Province? Then I see this:

    2024-09_0206

    We put the data in date order, and ran through each province. In this case, my two 1 day trips around Colorado are bucketed (partitioned) together and I see one less trip. If I did this by country, I’d see all of this as one sequential list, since all my trips were in one country.

    However, if I did countries for June, I’d see this, with the raw data on the left and the row_number() on the right. There’s a weird sequence in here; can you see it?

    2024-09_0208

    The weirdness is that my trips to England were broken up by a trip to Italy. So while my sequence looks good for the first part of the trip to English for 6 days, when I returned a week later, we get numbers 7, 8, 9. That’s because the data is grouped first by country, and the sequence added. The ordering of the sequence is by date, so the later days (June 13,-15) are marked with the higher sequence that continues on.

    Hopefully this gives you a basic look at row_number() and some of the possibilities. I’ll examine it further in another post, along with various other window functions.

    SQL New Blogger

    Complex coding and finding weird situations are things employers want you to be able to do. If you work on algorithms or you’re learning new language elements, blog about them. That will impress people.

    This post took me about 30 minutes, plus about 15 minutes or playing with code to set things up. You could likely do this in an hour if you’ve never blogged, though let someone proof things for you.

  • T-SQL Tuesday #179: The Detective’s Toolkit

    It’s time for the monthly T-SQL Tuesday blog party. This month a longtime friend, Tim Mitchell is hosting and he’s got a neat invite. He’s asking us how to dig into a request and figure out th meaning in our database, our files, or something else. How do you solve a data mystery?

    If you haven’t participated in a T-SQL Tuesday blog party, start a blog and join us. If you’re written a post, host one of the months in 2025. Don’t get too caught up in the rules. At this point, I’m just trying to get

    Inspector Gadget

    2024-10_0157For some reason, the first thing that comes to mind is Inspector Gadget. My kids used to love this show and I can still hear the song and catch phrase in my head.

    Being a detective doesn’t require gadgets, but they do make things easier to use. As Tim asks in the invitation, how do you go about learning about your data?

    One of the main things I’ve done in the past is use Notepad++ to load a large text file and look at it. These days VS Code seems to work well for my scales, and I can get an idea of the structure of text files.

    If I’m looking for meaning in a database, often I need two things:

    1. An Extended Events session
    2. A business analyst or end-user

    Usually I only look at data in databases when someone asks me to find something or explain something. Often, this means relating something in an application to a data model that might not be intuitive, obvious, or even rational.

    The big way for me to figure out where data is stored is to get a user to show me something in the application, or tell me what data they see from their side. Then when they query something, I can trace what happened with xEvents and see what tables were queried.

    The does most of the detective work for me, though sometimes I’m doing some trial and error as any specific screen in an application, might query lots of tables.

  • Moving T-SQL Tuesday to Jekyll

    I got a message a few months back that Microsoft was deprecating the MySQL server version that I was using in Azure. The cost was going up, and while I don’t mind donating the $12-15/month to the site, I didn’t want this to be really expensive.

    It’s also a perfect fit for a static site as the content rarely changes and doesn’t really need to have database access to server pages. Jekyll, which powers a number of others sites (SQL Saturday, SQL Memorial) seemed like a good fit.

    This covers a few of the things I did to move the site, which was surprisingly easier than I expected.

    Exporting the Data

    There is a plugin on the WordPress marketplace that you can add to your site to export your data as markdown for jekyll. I tried to add it to the live site, but the plugin didn’t want to activate. My PHP was old.

    I could upgrade PHP, but I might break something, and ultimately, I don’t need another job.

    Instead I decided to move the site local. That’s easy with a container, and I found this post on running WP in a container. In a couple of minutes after getting the Docker image downloaded, I had a new WP site running on my laptop. With the latest PHP and WP versions.

    2024-08_0026

    While the image was downloaded, I used the main WP export to download a copy of data.

    2024-08_0029

    The next step was to import this locally. Since I had a brand new WP site in a container, the easiest thing is an import, rather than a database restore. I’m a database pro, but I prefer simple and efficient processes.

    2024-08_0030

    I picked the export file and let it run.

    2024-08_0032

    Once I ran the import, I could see the same site in my local container as I see at tsqltuesday.com. Now I need a plugin. When I searched around, the Jekyll exporter was recommended, but I couldn’t install this on older version or WP.

    However, in my container, this worked fine.

    2024-08_0027

    Once the plugin was installed. I could see it in my list.

    2024-08_0028

    I also had a new menu item under my Tools, which was export to Jekyll. Once

    2024-08_0034

    Once I picked this, I got a file to download (from the container to my local drive), and all of my files were zipped up.

    2024-08_0035

    Inside of this file, I had a bunch of folders, which were structured for Jekyll. The _posts is the main folder where the various posts from your WP site are stored. As you see here, all the files are .md, which are markdown files. The _posts are the same format.

    2024-08_0036

    Now that we have a set of Jekyll compatible files, let’s get this moving to GitHub.

    Loading into Git

    The first step was to create a repository on GitHub. I’m moving this to the SQL Saturday organization, so it can be a part of the charitable foundation (and I can hand it to someone at some point).

    I created the repo and then downloaded an empty repo to my local drive. If you want to help improve the site or change things, fork this repo and then you can have your own URL.

    2024-08_0038

    I then copied my exported structure into this folder, but I had some issues rendering with the default GitHub themes. So I deleted everything and then grabbed a fork of Beautiful Jekyll, which is the same theme running the SQL Saturday site. Essentially I copied all of the Beautiful Jekyll files to this local repo, then copied the T-SQL Tuesday export on top of those files and committed the files.

    In GitHub, I also added the automation for Actions by grabbing the workflows folder and added that. In the pages setup, I then enabled pages, which gave me a .github.io URL, which is fine for testing.

    2024-08_0039

    This worked, as once I committed code, I saw a build.

    2024-08_0040

    And it worked well.

    2024-09_0116

    Sort of. The theme stuff didn’t quite work well, so I ended up copying over the SQL Sat site with the Beautiful Jekyll theme and then deleted posts and added back the T-SQL code.

    I had some work to get the site working well, and when I did, I moved it. I created an official repo here where people can send PRs if needed: https://github.com/sqlsaturday/tsqltuesday

    2024-08_0037

    Once I had things here, I had a little DNS work to do for the custom domain, but T-SQL Tuesday was again live at https://tsqltuesday.com/

    And with an SSL cert, thanks to GitHub. However, I wasn’t done.

    Data Cleanup

    While I was hoping the plugin would convert things easily, I realized quickly there were a bunch of problems. Not a crazy number, but some.

    The data cleanup was a manual process. I could perhaps have found a way to parse and automate this, but with 174 invites, this just wasn’t enough work for me to spend time building and testing something. Plus, I needed to get this done before the next month.

    First, I had a bunch of posts converted with long WordPress like URLs. That’s fine, but I had inconsistency as some posts weren’t linked in this way. I also had some links that were absolute, using the https://tsqltuesday.azurewebsites.net/ URL as a base.

    I decided to simplify things.

    First, I added a permalink to each page, similar to what you see in this one: https://raw.githubusercontent.com/sqlsaturday/tsqltuesday/refs/heads/main/_posts/2009-12-01-t-sql-tuesday-001-datetime-tricks.md

    I set this to be /xxx, where xxx is the invite number. While this might be a problem if we get to 1000, I certainly won’t be dealing with that one.

    2024-09_0128

    Next, some links were absolute and pointing to the old site, which was on Azure. I changed these, mostly in the Host Index, to relative links using a simple formula for the posts, which was a /xxx, where xxx is the invite number. This is a simple, clean way of managing the links.

    Lastly, I had issues with some individual posts, where I’d see things like this:

    2024-08_0120

    That HTML wasn’t rendering inside the markdown correctly, so I had to delete these items. And, of course, the end tags.

    2024-08_0121

    I also had issues where people would do something like Steve (twitter | blog) and the pipe character was seen as a table indicator. I had to change those to /.

    Savings

    The old costs, which usually came out of my MVP subscription, were about $12/mo for the db and $9 for the site. The requirement to move to a Flex server increased the db to about $23/mo.

    The new Github site costs $0. Perfect for a charitable endeavor.

  • SQL Saturday Boston 2024 Slides

    Thanks to everyone that came to my talks. Slides are below.

    If you have questions, please reach out.