Author: way0utwest

  • Is This Data Science?

    In looking at the preliminary results of his salary survey, Brent Ozar noticed that female salaries seemed to be lower than males. The post focused on a simple analysis of data, the kind that many of us have done in our organizations. We will look at some data, notice some anomalies, and produce some report that allows others to look deeper or make a decision.

    The examination of the salary data was taken a bit further by Eugene Meidinger, who wrote a post that started the title with “Practicing Statistics“. His analysis was interesting, not trying to determine the reasons or causes, but just decide if there were some significant patterns in the data. Whether you think there are or aren’t issues with salary, I thought the statistical analysis was done well, and really a basis for what I would call data science.

    Data Science is a hot topic right now in many organizations. In fact, for the last year, quite a few organizations are trying to incorporate more data science into their applications, and the hiring of “data scientists” is rising, with higher salaries being paid. There are various definitions of the practice, with none being standardized. There are many curriculums out there from colleges, and even one from Microsoft. In fact, quite a few SQL Server people have completed that coursework.

    Data Science seems to mean many things, which is both good and bad. Like being a DBA, there is a lot of room for interpretation and quite a bit of variance in what we may do as a job. I’ve often made a good living as a DBA, being slightly out of the normal reporting structure, having autonomy at work, and usually able to make a difference to a variety of groups. However, I’ve also found that many places don’t want to hire a DBA or don’t think they need one, preferring to let some Windows admin or developer perform those duties. Microsoft doesn’t have DBAs as a job title inside the company, usually using IT Ops staff for those duties. That means it can be hard to find a job at times if many companies don’t think they need that position, which might be the case for data scientists as well.

    I prefer to think positively, that data science and data scientist positions are going to grow and be profitable for some of us. We’ll need to learn to have some statistical basis for our analysis, and certainly regularly improve our knowledge of the tools for things like machine learning, but we will find ways to perform data analysis that’s beyond what most business people would complete in Excel. I think we’ll find that those of us that work with data analysis will have lots of opportunities in the future, no matter what we’re called.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.3MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • It’s the Engineers

    There are all sorts of articles and blogs on the Internet that try and teach you how to use a particular technique or technology to solve a problem. Plenty of this information is well written, and showcases the knowledge of that individual (or team) and how they’ve built a fantastic system. Every vendor building software platforms has reference case studies that show how well their system has worked for some customer. These are all good models that might give you confidence why SQL Server or MongoDB or Entity Framework or GoLang are good choices for your company.

    In reality, it’s not as simple as just choosing a platform or framework or language to make your application perform better. You really need the staff that understands how to use your choice of X and has some skill in building a system in that manner. It’s why I think that the cost of the software is a pittance compared to the cost of your people. If you have to train them, or they need to learn how technology X works, then you’re going to pay more in salary than you’d save in software licenses.

    There’s an article that I think illustrates this well, and might be worth passing along to your developers. It’s called Why Amazon DynamoDB isn’t for everyone and it’s worth a few minutes of your time to read. The gist of the article is that a NoSQL system, like DynamoDB, isn’t as simple and easy as you expect. It also says that for many applications, a relational database is a better choice because it’s often better understood and will solve most problems at small scale. For most of us, we don’t really get past what I’d consider small to medium scale, and so we should stick with relational systems.

    This isn’t to say that DynamoDB will be a problem for you, but there will be a learning curve for your developers if they haven’t used it in production and at your scale. The same thing occurs if a development team decided to switch from MongoDB to SQL Server because they like automatic tuning. They won’t necessarily configure and use SQL Server efficiently, and might not have their application perform well. Switching your development paradigm or technology is hard, and it will take time for your staff to get up to speed.

    Ultimately it really comes down to having good engineers that can write good code and understands the ins and outs of administering that particular platform. The database is extremely important here as it is the central location of your information, but the same arguments apply to frameworks and languages. As much as we want to learn and try new technologies and incorporate them into our work, we need to realize that it takes time to learn to use them well. Make small experiments, try POCs and slowly build up skills before you decide that your mission critical system needs some new technology that you read about and are excited to try working with.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.1MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • It’s Time to Patch and Upgrade

    I don’t want to be chicken little here, but the Meltdown/Spectre bugs have me concerned. I don’t know the scope of the vulnerabilities, as far as exploits go, but I do know the lax ways in which humans interact with machines, including running code, opening untrusted documents, and just making silly mistakes. No matter how careful you think you are, can you be sure everyone else in your organization is just as careful? Are you sure they won’t do something silly from a database server? Or do something from a server (or workstation) that has access to a database server? Or use a browser (yes, there’s an exploit)

    PATCH your system, soon.

    Vulernabilities in hardware are no joke, and even if you think you’re fairly safe, it’s silly to let this one go by and assume you won’t get hit. The advent of widely deployed scripting tools, botnets, and more mean that you never know what crazy mechanism might end up getting to your database server. Is it really worth allowing this when you can patch a system? This is a no brainer, a simple decision. Just schedule the patches. With all the news and media, I’m sure you can get some downtime approved in the next few weeks. After all, your management wouldn’t want to explain to their customers any data loss from this any more than you’d want to explain it to your boss.

    We’ve got a page at SQLServerCentral that summarizes the links I’ve found for information, patches, etc. I’m sure things will change rapidly, and I’ll update the article as I get more information. The important things to note are that not all OSes have patches yet, and there are situations where you might not need to change anything. That’s good, as there are some preliminary reports of patches causing issues with performance (degrading it) for PostgreSQL And MongoDB systems. I did see this tweet about no effects on SQL Server, which is good, but YMMV.

    Most of us know patching matters, and we need to do it periodically (even if it’s a pain), however, many of you are like me in that you rarely upgrade systems. Once they work, and because I have plenty of other tasks, I don’t look to necessarily upgrade a database platform for years. One downside to that is that a major vulnerability like the Meltdown/Spectre attacks is that patches likely won’t come out for old system and versions of SQL Server. That is the case here.

    That means that if you’re on SQL 2005-, or even on older Windows OSes, you might really consider planning an upgrade. Even if you aren’t overly worried about this exploit, you won’t want a vulnerability to live for a long time in your environment. You never know when a firewall will change, server will move, or some malware will slip through (did I mention the browser exploit?). Plan on an upgrade. I’ve started asking about accelerating our upgrade plans, and you might think about that as well. I know management doesn’t want to spend money unneceesarily, but this feels necessary, and a good time to refresh your system to a supported version.

    In general I like to delay my patches slightly from the world and not be on the bleeding edge. That’s fine, but don’t wait too long with this one. I would hope that most people get systems patched in the next month. If not, don’t expect any sympathy if you lose data.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 5.4MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Advent of Code 2017 Day 1–#SQLNewBlogger

    Another post for me that is simple and hopefully serves as an example for people trying to get blogging as #SQLNewBloggers.

    I like the Advent of Code. I stumbled on the site a few years ago and enjoyed working through some of the challenges in 2015. I tried solving them with three languages (Python, PoSh, and T-SQL) but only managed to get through about half the items before life got busy. Last year I was worn out and too busy to mess with the code.

    This year I was busy in December when the puzzles came out, but decided to work through the puzzles again as a break from work and life, and to keep my mind flexible. I’ll do a series here, hopefully all 25, but here’s my view of Day 1.

    Part 1

    Each puzzle has two parts, and you need to solve part 1 to get part 2. In this case I need to take a list of input and compare each digit to the next digit and see if they match. If they do, then you add that digit to your sum.

    This sounds like a perfect case to use the LEAD/LAG functions in SQL Server, so I did that. Since I get a long list of input, I also decided to use a string splitter. Here’s the first part where I split the string:

    DECLARE @i VARCHAR(5000);
     --SET @i  = '1122';
     --SET @i  = '951344679963668529';
    
    WITH myTally(n)
     AS
     (SELECT n = ROW_NUMBER() OVER (ORDER BY (SELECT null))
     FROM (VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10)) a(n)
     CROSS JOIN (VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10)) b(n)
     CROSS JOIN (VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10)) c(n)
     CROSS JOIN (VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10)) d(n)
     )
     , cteSplit (i)
     AS
     ( SELECT
     i = SUBSTRING(@i, n, 1)
     FROM myTally
     WHERE n <= LEN(@i)
     )
    
    select i
    
    from cteSplit

    If you run this, you’ll get your @i string out a character at a time. This get’s me the data in a set of rows, similar to an array.

    Now I need to compare each row with the next one, and I’ll use the LEAD function. My data is ordered, so I don’t need an order. That means I’m looking at this:

    LEAD(i, 1) OVER (ORDER BY (SELECT NULL))

    I’ll compare that with the current value and if they match, I’ll return the value. If not, I return a zero, adding nothing to the sum.

    There is one special case. The list is circular, so if the last digit (when we get there) matches the first digit, I need to include that. I’ll add that as special FIRST_VALUE, LAST_VALUE function match. Here’s my code:

    WITH myTally(n)
     AS
     (SELECT n = ROW_NUMBER() OVER (ORDER BY (SELECT null))
     FROM (VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10)) a(n)
     CROSS JOIN (VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10)) b(n)
     CROSS JOIN (VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10)) c(n)
     CROSS JOIN (VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10)) d(n)
     )
     , cteSplit (i)
     AS
     ( SELECT
     i = SUBSTRING(@i, n, 1)
     FROM myTally
     WHERE n <= LEN(@i)
     )
     , datacte(j)
     AS
     (
     SELECT j = CASE WHEN i = LEAD(i, 1) OVER (ORDER BY (SELECT NULL))
     THEN cteSplit.i
     ELSE 0
     END
     FROM cteSplit
     UNION all
     SELECT TOP 1 j = CASE WHEN FIRST_VALUE(i) OVER (ORDER BY (SELECT NULL)) = LAST_VALUE(i) OVER (ORDER BY (SELECT NULL))
     THEN i
     ELSE 0
     end
     FROM cteSplit
     )
     SELECT SUM(j) FROM datacte

    When I do this with the input string, I get a result. It happened to be right Winking smile

    Part II

    Part II is a variation of Part I. Instead of looking at the next digit, I need to look halfway around the list. The list is an even number, so if it’s 10 digits long, I need to start with char 1 and look at char 6. If they are equal, add to sum. Then look at 2 and 7, and repeat. Since it’s a circular list, when I get to char 5, I look around to char 1. There’s an optimization here, but when I solved this, I decided to take an easy way out.

    I first considered adding the first half of the list to the end and only running through the first len(input) characters. Then I thought, I could easily do a LAG. So, I modified my datacte query to do this:

    , datacte(j, k, i, l, n)
     AS
     (
     SELECT j = CASE WHEN i = LEAD(i, @len) OVER (ORDER BY (SELECT NULL)) AND n <= @len
     THEN cteSplit.i
     ELSE 0
     END
     ,
     k = CASE WHEN i = LAG(i, @len) OVER (ORDER BY (SELECT NULL)) AND n > @len
     THEN cteSplit.i
     ELSE 0
     END
    
    )

    This quickly counts the matches up to the middle and then counts the next bunch.

    There’s a better way, but I’ll leave you to figure that out in the comments.

    In any case, I solved both of day 1. Now on to day 2 during the next break.