Tag: T-SQL Tuesday

  • T-SQL Tuesday #182: Personal Integrity

    It’s time for the first T-SQL Tuesday blog of 2025, with an invite from the first non-founder to host a party, Rob Farley. I reached out to Rob and he graciously agreed to host. His invitation this month is on integrity, leaning towards data integrity.

    I still manage the T-SQL Tuesday list, and I’m always looking for hosts. I have a few scheduled for 2025, but I can use more. If you’re interested in hosting, hit me up at one of these places:

    While data integrity is important, I think personal integrity really matters as well.

    Integrity at Work

    I used to work for a large corporation, which I won’t name here. This was a 10,000+ person organization and I helped manage part of the data group for the company. I had a number of reports, 10 or 11, that handled different aspects of production operations across a variety of database platforms.

    It was a busy job, and our environment was far from stable. As is the case, things get cobbled together, become popular, and then limp along for years. This happens in small companies and large ones, but in large ones, I think it continues because when there are problems, we can throw people at the problem. We could throw resources and fix chronic issues, but that wasn’t the case at this company.

    At the time I worked at this company, we had a stacked ranking system for reviews.  Microsoft used to do this (and stopped), but think about ranking everyone in your team from 1-5, 1 being underperforming and 5 being outstanding. The “stacked” part comes from the need to have a certain number of 2s, 3s, and 4s. 1s and 5s were rare (5s more rare). Essentially a 1 meant you were on a performance plan and on the way to termination.

    I had to rank my staff, who worked hard and kept our systems running, despite lots of incidents. We performed some large migrations and upgrades of systems where my staff worked multiple weekends, with no comp time, just meals covered.

    My first integrity stand was that even though I wasn’t officially allowed to give comp time, I did, working through my staff with extra days off that were unrecorded anywhere. This might be more of a violation, and if my boss’s boss had caught wind of this, I would have been terminated. However,  I felt my staff deserved something, as a few of these long weekends weren’t adding business value; they were things executives wanted to do for optics.

    Near the end of the year, we had to rank our workers for annual reviews. I had most of my staff at 4s, with a couple 3s. I put in one person as a 5 for some great work they did. Justifying a 5 is hard and I thought I had a good case.

    In a meeting of our IT department, the VP of Operations told us that we had too many 4s and that each director (1 level above me) would get an allocation of rankings and that each manager had to work to fit their staff inside that allocation. Anyone ranked a 1 would be outside the allocation.

    My second integrity stand was to argue with my boss on why my staff deserved their rankings. According to their job descriptions and performance, I’d ranked then well, even though his quota meant he wanted me to move a couple 4s to 3s and 1-2 people to a 2 ranking. I refused, arguing with other managers whose staff I had seen underperform through the year. I told a few others they needed to absorb the 2s and 3s and not the data team.

    I lost that battle and had some uncomfortable reviews (and the accompanying bonus/salary numbers), but I acted like an adult and told people I had ranked them a level higher and upper management lowered their ranking, not because of performance, but because of a quota.

    As we went through the next year, management decided they needed to cut costs. They had managers compile a lot of numbers that were sent to outsourcing companies in an RFP to essentially remove IT from the company’s books. In a meeting, our CTO tried to spin this as good for workers as the winning company would hire our IT staff. The framing was that this outsourcing would save the company money.

    My third integrity stand was when I questioned the way this worked in a large meeting of IT management and eventually got the CTO to admit that:

    • a) everyone would have to re-apply for their job
    • b) they might not get the same salary (higher or lower, you decide which is more likely)
    • c) not everyone would be hired

    This might work out for some people, but likely those who got jobs at a new company were risking their salary and workload in a new situation. I questioned this as being good for the company overall as we would likely lose lots of knowledge.

    I left shortly thereafter, voluntarily, but I’m sure I would have been let go at a layoff that occurred near the end of that year.

    I hated that job in many ways and was glad to eventually leave. I decided I would never work for a stack ranking company again.

  • T-SQL Tuesday #181: A Technology Present

    It’s the last T-SQL Tuesday of the year, and it’s amazing to think we’ve gotten to #181. That’s over 15 years of monthly blog parties.

    This month we have a slightly different invite from Kevin Chant. Kevin usually participates in the Festive Tech Calendar, which I’ve never been a part of. He wanted to combine those to together in a crossover, which is an interesting idea. I wonder how it will go.

    In any case, there’s a bit of explanation, but the essential invite is this: My invitation to yourselves is to write about a Microsoft Data Platform announcement that you considered to be as amazing as a present. In other words, something which made you go “wow”.

    It’s a good invite for the end of the year and my answer is below.

    Think about a VCS Diff

    I could say git diff since git has won the VCS battle, but in any tech work, it’s important to be able to tell what things have changed and evaluate if the change is helpful or hurtful.

    Lots of tools, especially visual ones, haven’t always considered this. They’ve often built a tool that doesn’t easily make comparing code easy. The first example of this was DTS/SSIS (SQL Server Integration Services), where this was a visual tool. The configuration was stored in an XML file, but every visual change created a change in the XML, and the structure was fluid enough that it was often hard to determine what changed between versions.

    Microsoft has a lot of tools that did this, including Power BI. However, a preview mode of the code was announced earlier last year. Power BI Developer mode includes Git integration. This gives us a way to edit the file as code, something that should be required of all tech tools. The visual stuff is great, but give us a code option.

    You can enable this in the PBI Desktop options.

    2024-12_0143

    To me, this is fantastic as it enables this to really work as a code tool, which it is. More importantly, as we get changes made by Copilot or other AIs, we need to easily see the differences that exist between versions. That’s important for troubleshooting and governance.

    To me, getting a Power BI project file that works in a VCS is a great present.

  • T-SQL Tuesday #179: The Detective’s Toolkit

    It’s time for the monthly T-SQL Tuesday blog party. This month a longtime friend, Tim Mitchell is hosting and he’s got a neat invite. He’s asking us how to dig into a request and figure out th meaning in our database, our files, or something else. How do you solve a data mystery?

    If you haven’t participated in a T-SQL Tuesday blog party, start a blog and join us. If you’re written a post, host one of the months in 2025. Don’t get too caught up in the rules. At this point, I’m just trying to get

    Inspector Gadget

    2024-10_0157For some reason, the first thing that comes to mind is Inspector Gadget. My kids used to love this show and I can still hear the song and catch phrase in my head.

    Being a detective doesn’t require gadgets, but they do make things easier to use. As Tim asks in the invitation, how do you go about learning about your data?

    One of the main things I’ve done in the past is use Notepad++ to load a large text file and look at it. These days VS Code seems to work well for my scales, and I can get an idea of the structure of text files.

    If I’m looking for meaning in a database, often I need two things:

    1. An Extended Events session
    2. A business analyst or end-user

    Usually I only look at data in databases when someone asks me to find something or explain something. Often, this means relating something in an application to a data model that might not be intuitive, obvious, or even rational.

    The big way for me to figure out where data is stored is to get a user to show me something in the application, or tell me what data they see from their side. Then when they query something, I can trace what happened with xEvents and see what tables were queried.

    The does most of the detective work for me, though sometimes I’m doing some trial and error as any specific screen in an application, might query lots of tables.

  • Moving T-SQL Tuesday to Jekyll

    I got a message a few months back that Microsoft was deprecating the MySQL server version that I was using in Azure. The cost was going up, and while I don’t mind donating the $12-15/month to the site, I didn’t want this to be really expensive.

    It’s also a perfect fit for a static site as the content rarely changes and doesn’t really need to have database access to server pages. Jekyll, which powers a number of others sites (SQL Saturday, SQL Memorial) seemed like a good fit.

    This covers a few of the things I did to move the site, which was surprisingly easier than I expected.

    Exporting the Data

    There is a plugin on the WordPress marketplace that you can add to your site to export your data as markdown for jekyll. I tried to add it to the live site, but the plugin didn’t want to activate. My PHP was old.

    I could upgrade PHP, but I might break something, and ultimately, I don’t need another job.

    Instead I decided to move the site local. That’s easy with a container, and I found this post on running WP in a container. In a couple of minutes after getting the Docker image downloaded, I had a new WP site running on my laptop. With the latest PHP and WP versions.

    2024-08_0026

    While the image was downloaded, I used the main WP export to download a copy of data.

    2024-08_0029

    The next step was to import this locally. Since I had a brand new WP site in a container, the easiest thing is an import, rather than a database restore. I’m a database pro, but I prefer simple and efficient processes.

    2024-08_0030

    I picked the export file and let it run.

    2024-08_0032

    Once I ran the import, I could see the same site in my local container as I see at tsqltuesday.com. Now I need a plugin. When I searched around, the Jekyll exporter was recommended, but I couldn’t install this on older version or WP.

    However, in my container, this worked fine.

    2024-08_0027

    Once the plugin was installed. I could see it in my list.

    2024-08_0028

    I also had a new menu item under my Tools, which was export to Jekyll. Once

    2024-08_0034

    Once I picked this, I got a file to download (from the container to my local drive), and all of my files were zipped up.

    2024-08_0035

    Inside of this file, I had a bunch of folders, which were structured for Jekyll. The _posts is the main folder where the various posts from your WP site are stored. As you see here, all the files are .md, which are markdown files. The _posts are the same format.

    2024-08_0036

    Now that we have a set of Jekyll compatible files, let’s get this moving to GitHub.

    Loading into Git

    The first step was to create a repository on GitHub. I’m moving this to the SQL Saturday organization, so it can be a part of the charitable foundation (and I can hand it to someone at some point).

    I created the repo and then downloaded an empty repo to my local drive. If you want to help improve the site or change things, fork this repo and then you can have your own URL.

    2024-08_0038

    I then copied my exported structure into this folder, but I had some issues rendering with the default GitHub themes. So I deleted everything and then grabbed a fork of Beautiful Jekyll, which is the same theme running the SQL Saturday site. Essentially I copied all of the Beautiful Jekyll files to this local repo, then copied the T-SQL Tuesday export on top of those files and committed the files.

    In GitHub, I also added the automation for Actions by grabbing the workflows folder and added that. In the pages setup, I then enabled pages, which gave me a .github.io URL, which is fine for testing.

    2024-08_0039

    This worked, as once I committed code, I saw a build.

    2024-08_0040

    And it worked well.

    2024-09_0116

    Sort of. The theme stuff didn’t quite work well, so I ended up copying over the SQL Sat site with the Beautiful Jekyll theme and then deleted posts and added back the T-SQL code.

    I had some work to get the site working well, and when I did, I moved it. I created an official repo here where people can send PRs if needed: https://github.com/sqlsaturday/tsqltuesday

    2024-08_0037

    Once I had things here, I had a little DNS work to do for the custom domain, but T-SQL Tuesday was again live at https://tsqltuesday.com/

    And with an SSL cert, thanks to GitHub. However, I wasn’t done.

    Data Cleanup

    While I was hoping the plugin would convert things easily, I realized quickly there were a bunch of problems. Not a crazy number, but some.

    The data cleanup was a manual process. I could perhaps have found a way to parse and automate this, but with 174 invites, this just wasn’t enough work for me to spend time building and testing something. Plus, I needed to get this done before the next month.

    First, I had a bunch of posts converted with long WordPress like URLs. That’s fine, but I had inconsistency as some posts weren’t linked in this way. I also had some links that were absolute, using the https://tsqltuesday.azurewebsites.net/ URL as a base.

    I decided to simplify things.

    First, I added a permalink to each page, similar to what you see in this one: https://raw.githubusercontent.com/sqlsaturday/tsqltuesday/refs/heads/main/_posts/2009-12-01-t-sql-tuesday-001-datetime-tricks.md

    I set this to be /xxx, where xxx is the invite number. While this might be a problem if we get to 1000, I certainly won’t be dealing with that one.

    2024-09_0128

    Next, some links were absolute and pointing to the old site, which was on Azure. I changed these, mostly in the Host Index, to relative links using a simple formula for the posts, which was a /xxx, where xxx is the invite number. This is a simple, clean way of managing the links.

    Lastly, I had issues with some individual posts, where I’d see things like this:

    2024-08_0120

    That HTML wasn’t rendering inside the markdown correctly, so I had to delete these items. And, of course, the end tags.

    2024-08_0121

    I also had issues where people would do something like Steve (twitter | blog) and the pipe character was seen as a table indicator. I had to change those to /.

    Savings

    The old costs, which usually came out of my MVP subscription, were about $12/mo for the db and $9 for the site. The requirement to move to a Flex server increased the db to about $23/mo.

    The new Github site costs $0. Perfect for a charitable endeavor.