Author: way0utwest

  • Porting SQLServerCentral

    Like a few of you, I’ve been working with WordPress as a blogging platform. Over the years I’ve tried a few different pieces of blogging software, but I really like WordPress. I’m not alone as there are estimates that 20-30% of all websites run on this platform, including a few you might not expect. I thought UpperCup and Krispy Kreme UK are sites that don’t really look like they’re powered by WordPress. Those make my own blog and T-SQL Tuesday look pretty bland. Maybe I’ll do a little design work at some point on those. That’s after SQLServerCentral moves over to the platform.

    SQLServerCentral started as a custom ASP site many years ago, then upgraded to ASP.NET at some point. This was a joint effort from the founders to build in new functionality and features as we needed them, purchasing components (like the forums) where we found a suitable product. This first evolution of the site lasted for many years until Redgate Software acquired the property. We then underwent a second platform shift to NHibernate, which has been underpinning the site for a decade. We now move forward with our third evolution.

    We have a project underway that is porting our site to WordPress, for a variety of reasons. Like many of you, I struggle to get resources assigned from my employer for the projects that I’m passionate about if they don’t rise in importance above other things being worked on. There are only so many resources available, and they must be shared by the company. While Redgate values SQLServerCentral, we have a site that works well, and has worked well for many years. Thus, it’s not the same priority as some of the other projects in the company. Since we have some requirements around better mobile support thanks to Google, we had to move in some direction.

    We have struggled with skillsets over the years as most of our web developers aren’t well versed in NHibernate as we’ve moved many of our other web projects to WordPress or more basic technologies like React. Building all the various features from scratch would be a big project, not to mention a constant maintenance headache, so after reviewing some responses to our RFP, we decided to go with WordPress, under Project Nami. This is an open source project that replaces MySQL with SQL Server. While I run MySQL on T-SQL Tuesday, one of our key requirements was that we use SQL Server as a database, and Project Nami allows us to do this. Since there are numerous people with WordPress skills, and lots of plugins that can be easily added (or removed), our view is that WordPress will allow us to grow and change the site over time with fewer resource constraints.

    The last few months have been a long, drawn out project as we needed a number of custom plugins written, or existing ones adapter for some of the functions on the site. At its heart, SQLServerCentral is a rather unique publishing platform, and we needed to preserve much of this functionality. As with most projects, we’ve run over time and budget a bit, but we’re now getting close. I don’t have a date yet, but I anticipate we’ll add more user testing in January and then make a switch sometime later in the month.

    I hope that you’ll find the new platform to be very similar to what we have now. Our goal was to change relatively little in terms of functionality and minimize the look and feel changes. There are some, but I don’t think they are too disruptive. However, we will be looking for feedback and make decisions on what things we’d like to change or adapt for the future. Keep an eye out for more announcements and fingers crossed that everything goes smoothly during the deployment.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 5.6MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Identity Gaps–#SQLNewBlogger

    Another post for me that is simple and hopefully serves as an example for people trying to get blogging as #SQLNewBloggers.

    Many people think that that an identity property will ensure a consistent, increasing numerical value. I ran across this tweet that indicates that situation.

    2018-12-21 12_13_51-Krista on Twitter_ _#SQLHelp Is there any other reason (other than a DELETE) for

    This isn’t really true, for many reasons, but in this post I’ll look at the possible reasons we get gaps in identity values.

    Normal Operation

    Let’s start with a basic table that contains an identity value. I’ll use this code:

    CREATE TABLE dbo.SalesOrderHeader
    ( OrderKey INT IDENTITY(1, 1)
    , CustomerName VARCHAR(30)
    )
    GO

    Now I can insert a few rows. Note that the results shown below the code will contain increasing values for the OrderKey.

    INSERT dbo.SalesOrderHeader (CustomerName) VALUES ('Andy')
    INSERT dbo.SalesOrderHeader (CustomerName) VALUES ('Brian')
    INSERT dbo.SalesOrderHeader (CustomerName) VALUES ('Steve')
    INSERT dbo.SalesOrderHeader (CustomerName) VALUES ('Anna')
    GO

    Each of these inserts is a separate transaction, and they cause the identity to increment.

    2018-12-21 12_04_17-SQLQuery5.sql - Plato_SQL2017.sandbox (PLATO_Steve (59))_ - Microsoft SQL Server

    Deleting Rows

    This is noted in the tweet as a cause, but let’s test this.

    One of the common ways that we get gaps in identity values is when rows are deleted. Let’s remove the row with Steve in it.

    2018-12-21 12_06_46-SQLQuery5.sql - Plato_SQL2017.sandbox (PLATO_Steve (59))_ - Microsoft SQL Server

    I clearly have a gap in OrderKey here now. What happens if we add a new row? The identity value is built for (some) efficiency and doesn’t fill the gap. Only the next value is kept. We insert a row and get a 5.

    2018-12-21 12_08_00-SQLQuery5.sql - Plato_SQL2017.sandbox (PLATO_Steve (59))_ - Microsoft SQL Server

    As a side note, there is no index on this table, and no ORDER BY clause, so you can clearly see that there isn’t a reason why I should expect the ORDERKEY column to be returned in numerical or even insert order.

    The Rollback

    One of the more common occurrences with inserts is a problem with the value. For example, in this table, I have allocated 30 characters. What happens if I run this code?

    INSERT dbo.SalesOrderHeader (CustomerName) 
       VALUES ('A Really Long Name Van Something The Third')

    I get an error, which is shown here.


    Checking the table, I have no value:

    2018-12-21 12_11_17-SQLQuery5.sql - Plato_SQL2017.sandbox (PLATO_Steve (59))_ - Microsoft SQL Server

    Let’s insert a new value and see.

    2018-12-21 12_12_00-SQLQuery5.sql - Plato_SQL2017.sandbox (PLATO_Steve (59))_ - Microsoft SQL Server

    We get a gap. The value “6” was skipped because of the error. The identity was allocated, but the rollback of the transaction due to the error did not rollback the identity sequence.

    Reseeding the Property

    One of the other ways to miss a value is directly reseeding the table. I can use the DBCC CHECKIDENT function to accomplish this. In my case, let’s run this code and set the identity value to 20.

    DBCC CHECKIDENT(SalesOrderHeader, RESEED, 20)
    GO

    Now I can insert new values and I’ll get these results.

    2018-12-21 12_17_28-SQLQuery5.sql - Plato_SQL2017.sandbox (PLATO_Steve (59))_ - Microsoft SQL Server

    The identity value was set to 20 and the next insert will increment this and take 21, leaving a gap from 8 to 20.

    Be Careful

    Don’t depend on the identity property to give you uniqueness, consecutive values, or avoid duplicates. It is up to you to code properly to account for these values.

    SQLNewBlogger

    This post came about from helping someone understand the problems and limitations. I wrote this in about 20 minutes (with setup and testing) to ensure that I understood what I was explaining to someone.

    You could write something similar to show that you know the ways in which identity works.

  • Better Static Code Analysis and Security Scans

    I was listening to a talk from Stefan Simenon on their CI/CD transformation within ABN AMRO, a large financial company. One of the interesting things he noted was that they consider open source to be less secure, possibly with more vulnerabilities than in house written software. Their build pipeline will fail if a developer starts using new OSS components.

    I find that interesting, as the DORA 2018 State of DevOps report sees more use of OSS software in companies that are adopting DevOps. In general, I think that having many people able to view the source and find errors makes companies feel that open source is more secure. I think that’s likely more true, though it’s a bit of a philosophical argument. We can look at some data, but it’s hard to prove that one or the other is empirically more secure.

    The thing I agree with is that using new components without some review is not a good idea. Whether this is written in-house, copied from an Open Source project, or purchased from a vendor, we need to perform some testing and analysis of the code or component.

    This is also true in database code. When we get a query from a developer, it’s often easy to determine what is happening, but when the size of code grows, or there is a large stored procedure, we often don’t perform a detailed analysis. What’s worse, we don’t have good static code analysis tools for database languages. As much as I like what Redgate Software has done with SQL Prompt, I know this is rudimentary and is built to avoid code smells. There isn’t any detailed look at whether the code is secure, or if there might be unintended effects.

    There aren’t really any good tools I’ve seen, though I’m not even sure what I’d want here. How can a tool tell me that querying 4 tables and updating 3 more is OK, but an insert to some other table in a separate database is bad. That insert to the other database might be what a malicious actor wants to copy data elsewhere. The best thing to me would be some analysis of what objects are being touched and how, which could help alert developers to potential issues.

    Building static code analysis tools for database languages is hard, but it’s something that our industry needs to do. This is even more true when we start to have more programmability features, like the ability to execute other languages inside of our database engines. In those cases, not only do we need to ensure the code for another language passes test, but that we understand what types of interactions our database code has with those modules.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.5MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • 2018 Advent of Code–Day 1

    I enjoy when the Avent of Code comes around each year. I seem to make this a December (or sometimes New Year’s) resolution to get through them all, but life usually gets in the way. In any case, I decided to at least start this year and see how far I get.

    Day 1 – First Puzzle

    This is a simple one, and one that seems to lend itself to T-SQL. We have an input file that looks like:

    +11

    +9

    -10

    -5

    etc.

    This asks us to walk through the file, summing the values together and getting a new value. So the first row ends with 11. The next ends with 20 (11+9). The next is 10 (20-10), and so on. This feels like a simple calc, so let’s get it.

    I wanted to load this with BULK LOAD, so I started with a table:

    CREATE TABLE Day1(rawdata VARCHAR(20))

    I know I’ll need to change this, but let’s make this easy. I use this command to now load my data.

    BULK INSERT dbo.Day1 FROM 'C:\Users\way0u\Source\Repos\AdventofCode\2018\Day1\input.txt'

    Once this is done, I’ll move on. Since I need to get this into some numeric values (this is a math problem), I’ll make another table.

    CREATE TABLE Day1_a(frequency INT)

    Now I move the data.

    INSERT dbo.Day1_a
    (
         frequency
    )
    SELECT CAST(rawdata AS int)
    FROM dbo.Day1
    GO

    That seems to work fine. How do I get the end result? Well, addition doesn’t matter here, so I can do this:

    SELECT SUM(frequency) FROM dbo.Day1_a
    GO

    I get an answer, plug it in, and viola, I’m right. That feels good.

    Day 1 – Second Puzzle

    This one is a little harder. I’m supposed to find out the first time that the end result repeats it’s value. The test cases show this working as follows:

    Value    New result

    0       0
    1       1
    -1      0

    If I walk through this, the 0 repeats. The other test cases show this, but with the large input set, I need to change a few things.

    1. I need to preserve ordering
    2. I need to process this row by row.

    The second item doesn’t mean that I’m looping necessarily, but I need to calculate out the sums as I go and potentially repeat the list.

    To get started, let me modify my Bulk Insert and table to keep the ordering. I created this table.

    CREATE TABLE Day1b(datakey INT IDENTITY(1,1), rawdata VARCHAR(20))

    I then ran BULK INSERT. I got this error:

    2018-12-03 15_24_00-SQLQuery5.sql - dkrSpectre_SQL2017.sandbox (DKRSPECTRE_way0u (53))_ - Microsoft

    I tried a number of items, but nothing really worked. This was a very, very annoying error, and the main solution I saw on Stack Overflow was to add a column to the input file, which I don’t want to do. I initially thought this was a problem with the encoding, but it’s really the identity.

    The best solution was a lower down answer, which was to create a view without the identity.

    CREATE VIEW vDay1b
    AS
    SELECT rawdata
      FROM dbo.Day1b
    GO

    If I run the BULK INSERT to this view, it works fine.

    OK. We’re moving and I have the data in order. Let’s move it to get the integer results we need.

    CREATE TABLE Day1_2
    ( n INT, frequency INT)
    GO
    INSERT Day1_2
      SELECT datakey,
             CAST(rawdata AS INT)
       FROM dbo.Day1b

    If I run a quick query that does a SUM() OVER(), I get a series of results. I can see there are no duplicates here.

    2018-12-03 15_32_29-SQLQuery5.sql - dkrSpectre_SQL2017.sandbox (DKRSPECTRE_way0u (53))_ - Microsoft

    OK, this means I need to repeat the data. I can re-insert data into the table, but that feels inefficient. I ought to be able to group data together.

    Let’s do this by selecting the data as a group, but adding a value to it. I can do that with a cross join. Here’s a short example that illustrates this. Suppose I have a table with the values “Broncos”, “Chiefs”, “Raiders”, “Chargers”, I get select data like this in groups.

    2018-12-03 15_36_36-SQLQuery5.sql - dkrSpectre_SQL2017.sandbox (DKRSPECTRE_way0u (53))_ - Microsoft

    With that in mind, let’s create a tally table and start to duplicate data. I have no idea how many times, but having done the Advent of Code before, I’m guessing 5 groups isn’t enough. Let’s start with 100 repeats.

    One note, I do need to start with 0, so we’ll use a UNION to add the 0 row. We don’t want the 0 row repeated, so we don’t add that to the table.