Author: way0utwest

  • Next week is SQL in the City Streamed

    This picture is out of date. I’ll make sure we get a new one next week at SQL in the City Streamed that includes Kendra Little.

    IMG_2469

    On Sept 5th, is SQL in the City Streamed. Register today and watch us talk about Azure, Machine Learning, full stack development, and more. We’ll talk a little Redgate as well.

    sitc-201809-social-grant (1)

    I’m off today to Norway for SQL Saturday Oslo, but I’ll also be prepping for SQL in the City Streamed. Hope to see you online.

  • Protecting Our Stream of Data

    Protecting the data our companies collect is important. Many of us go to great lengths to secure our databases, firewall connections, limit access, and more. However, we can’t secure the data before it gets to us, and that can be a problem. I ran across a link on Bruce Schneier’s blog that shows a criminal placing a skimmer on a credit card scanner in a convenience store. The original video is gone, but there are plenty more.

    In general, the loss of data from the application (and physical hardware in this case), isn’t really a database issue. After all, the data is essentially split into two streams, with some going to the legitimate database and other processes while another stream goes to the skimmer. If this happens, and the skimmer isn’t discovered, however, it’s entirely possible that any data loss might be blamed on the database or IT infrastructure.

    If someone suspected you were hacked because of data being lost, could you prove you weren’t hacked? Or that the data didn’t come from your database? This is impossible, since you can’t prove a negative, but would you have any evidence that could be used to bolster your claims? Is there auditing or other tracking of activity? Does your organization keep any logs that would show a lack of activity and are protected against tampering? Ideally you would find the source that lost the data, but if you can’t, it can be difficult to prove that the losses didn’t come from your systems.

    For most of us, we might not have much in the way that would show our systems have only had legitimate access. Instead we’d depend on the limited SQL Server logging, as well as other infrastructure tracking, such as firewall activity logs. The strength of our presentation would likely determine whether security staff or management accepted our claims as valid.

    Point of Sale systems are different than the applications that most of us use, but certainly we have database security concerns that we should address. I would hope that SQL Server would bolster its capabilities in this area, providing a way to set up a tamper proof log easily that accepts writes of activity from a database in some structured text file that uses minimal resources. I’m not even sure what I’d want here that I can’t get from XE, but I do think having something more robust and standardized would be nice. If nothing else, a local SQL Server could stream XE events out to a file target in a remote location that only supports writing. This wouldn’t necessarily prove anything if the connection were disrupted, but it would ensure that a business was aware of the breakdown between the instance and audit log.

    I know the ways in which people attempt to access and steal data will continue to evolve and become more complex and creative. We can’t protect against many of them from the database side, but I’d like to think that we could protect the data we have. We should ensure it is safe from theft, loss, or inappropriate access and prove we have done so.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.2MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Redgate Lives DevOps

    At Redgate, we release a lot of changes to our products. In fact, this is the “About Redgate” slide I’ve been using in talks related to the company.

    2018-08-23 09_36_29-ReduceAttackSurfaceArea.pptx - PowerPoint

    If you look in the lower right, you’ll see product releases from last year (2017). We have 30-ish products, so that’s roughly 38 releases per product. Not every product releases at this cadence, but a lot of them release every week. SLQ Monitor, for example, releases every Wednesday.

    Of course, there’s the inevitable release-a-bug-and-need-to-fix-it-so-a-second-release this week, but those don’t happen too often. I’ve been tracking releases this year, and not too many were corrected in the same week, but it does happen.

    Some people say that’s the problem with DevOps, but I think it’s the advantage. I guarantee that most software releases include bugs. If you release once a quarter, are you ready to re-release in a couple days to fix something? Or do customers live with issues for a quarter? The advantage of DevOps is we can fix things quickly, in addition to adding new features quickly.

    Redgate does some amazing development work and I’m proud of the ladies and gentleman that write the code.

    I still complain, and there’s room for improvement, but they do a great job and I try to remember to thank them and complement their work when I do see them.

  • Generating a Constrained Random Date–#SQLNewBlogger

    Another post for me that is simple and hopefully serves as an example for people trying to get blogging as #SQLNewBloggers.

    There have been lots of posts on the topic of generating random values, and some great articles. One of my favorites is Jeff Moden’s Generating Test Data: Part 1 – Generating Random Integers and Floats. Part 2 deals with dates, and that’s actually what I needed, but really I needed part 1.

    In my situation, I was helping a customer generate some random data. They had filled a table, Customers, with some data.

    2018-08-24 13_05_44-Microsoft Edge
    The goal was to populate a child table with some data. The child table had a date column that was supposed to be between the Entered and Exit dates in the Customer table.

    My update would have a join, obviously, and I can reference the enter and exit date, but how to get a date between them? My first thought was that I wanted a DATEADD() function. Something like this:

    UPDATE ce
    SET ce.EventTimeStamp = DATEADD( MINUTE, SomeRandomValue, c.CustomerExitedDateTime)), c.CustomerEnteredDateTime)
    FROM   dbo.Customer AS c
    INNER JOIN dbo.CustomerEvent AS ce
    ON ce.CustomerID = c.CustomerID

    The trick is what random value to use? If you look through Jeff’s article, you will see that the trick is to use a tally table and the NEWID() function. However, this doesn’t work:

     UPDATE ce
    SET ce.EventTimeStamp = DATEADD( MINUTE, NEWID(), c.CustomerEnteredDateTime)
    FROM   dbo.Customer AS c
    INNER JOIN dbo.CustomerEvent AS ce
    ON ce.CustomerID = c.CustomerID
    ;

    What I need to do is convert the GUID to a number. In this case, I added CHECKSUM around it, again, as in Jeff’s article. Then use ABS() to enclose this to get all positive numbers.

     UPDATE ce
    SET ce.EventTimeStamp = DATEADD( MINUTE, ABS(CHECKSUM((NEWID())))), c.CustomerEnteredDateTime)
    FROM   dbo.Customer AS c
    INNER JOIN dbo.CustomerEvent AS ce
    ON ce.CustomerID = c.CustomerID
    ;

    This gives me values, but they aren’t constrained. What I need to do is limit the upper random value so that the end time doesn’t exceed the Customer.CustomerExitDateTime for that row.

    To do this, I can constraint a large set of numbers to some value with the modulo function. This will limit what values can appear. The basic script is this:

    UPDATE ce
    SET ce.EventTimeStamp = DATEADD( MINUTE, ABS(CHECKSUM((NEWID())))) % 10, c.CustomerEnteredDateTime)
    FROM   dbo.Customer AS c
    INNER JOIN dbo.CustomerEvent AS ce
    ON ce.CustomerID = c.CustomerID
    ;

    This would give me values between 1 and 0 minutes after the start time, but this doesn’t mean these values won’t be after the exit time. This is also an unrealistic window if most of the time the enter and exit times vary by hours.

    What I did instead was to use the difference between the enter and exit times, with DATEDIFF() as my modulo function. That gives me:

    WITH myTally (n)
    AS
    -- SQL Prompt formatting off
    (SELECT n = ROW_NUMBER() OVER (ORDER BY (SELECT null))
      FROM (VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10)) a(n)
       CROSS JOIN (VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10)) b(n)
    )
    UPDATE ce
    SET ce.EventTimeStamp = DATEADD( MINUTE, ABS(CHECKSUM((NEWID()))) % (DATEDIFF(MINUTE, c.CustomerEnteredDateTime, c.CustomerExitedDateTime)), c.CustomerEnteredDateTime)
    FROM   dbo.Customer AS c
    INNER JOIN dbo.CustomerEvent AS ce
    ON ce.CustomerID = c.CustomerID
    ;

    I run this, and I get the table updated with a random set of values.

    2018-08-24 13_19_08-Microsoft Edge

    SQLNewBlogger

    This was a problem in my daily work. It was a customer, but it could easily be an internal query problem. I spent about 10 minutes grabbing screen shots and taking apart the query I’d built.

    You can do this, too. Show us your mind working with the solutions you write in your own blog.