Voice of the DBA

Author: way0utwest

  • SQL Saturday #390 – Philadelphia Recap

    I was at SQL Saturday #390 in Philadelphia last week and it was an exciting, but short time. The event was at the end of a  five day trip for me, Seattle first and then Philadelphia, so I was tired by Saturday morning. However I enjoyed it, met lots of people, and learned a few interesting things.

    Unit Testing with SQL Server and tSQLt

    Friday I taught a pre-con event with Sebastian Meine on unit testing your T-SQL code with tSQLt. We had 9 people signed up, which was encouraging. Not a raging success, but certainly better than the 2 people that some friends had predicted. Everyone was interested in unit testing and producing better code.

    The day went well, and we ended a touch early. There were some good comments and suggestions from people on unit testing techniques, including a solution to one of our exercises that I hadn’t expected. I learned a few places need the demos to be expanded a bit, but overall it felt like a good start to teaching unit testing. I’m looking forward to trying this at a few other events in the future.

    standing in front of the class

    Better Development at SQL Saturday

    I was really only able to attend a few sessions (in addition to the one I taught) at SQL Saturday #390. However I was pleased to see that after my CI session, there was one on Continuous Delivery, and then one on Git for DBAs. Both of these topics are important, and well attended, which makes me think that more people are starting to lean towards building more engineered database development processes.

    My CI session went well. I didn’t get to show the failing CI process based on a test and more data, but I did get lots of good questions that I need to answer in the next few weeks. A few of them (for me to remember).

    • How do I get started with a small development team?
    • How do I start adding tests in an existing software project?
    • Can you demo a column rename (or table rename)?
    • Show static data flowing through the process

    It’s good to get questions, and I hope I answered them all. If I didn’t, or you need clarification, let me know.

    Once I finished, I wandered over to see Mark Wojciechowicz talk about Continuous Delivery. Mark is a consultant that’s trying to keep a CD process working at a client’s site. He’s been working at it for awhile and we talked about it Friday (he was in my class) and Saturday. It sounds like he has some good things going, and using a home grown process that works well.

    He showed some of the reasons why CD matters, especially for his client. They make lots of changes, and use a smooth process to push through bug fixes as well when there are issues with something in production. I hope he gets his deck up soon because he had some good quotes and notes in the PPT.

    I especially like his thoughts that if something hurts in software, you should do it more. That comes from Thoughworks and some of the progressive companies that are rapidly building high quality software and don’t let the problems of deployment slow them down. They tackle those issues just like others, and you should, too.

    From there I saw Justin Dearing talk about Git and DBAs. He’s a developer and tends to like the command line, but he showed how we can use Git for a VCS in the command line, but also from SSDT. That’s an interesting approach, and while I liked his message, I think some of the people there using a CVCS, like TFS, didn’t understand why Git matters. I need to write more on this as well.

    Justin had a minor issue with branching, not because he doesn’t do it well or understand it, but because he’s moving fast and talking. That’s what I am, in general, against branching. The real world moves fast, it’s easy to get caught up with some other thoughts and get lost with branching. I need to formalize my thoughts as I’ll be talking branching at DevConnections (register and come see me in September).

    Overall a good time and people seemed to really enjoy the SQL Saturday. I had about 20% of my attendees that were new, which is great. I hope to see more and more people coming to the events in the future.

  • When Should We Be Cleaning Data?

    I was reading Grant Fritchey’s update from the Business Analyst Conference  and noticed this quote: ” There were lots of sessions on the need to clean data, which implies that we need to do a better job at data collection in order to support this wild new world.” Grant was talking about the fact that many of the sessions seemed to imply that it’s mostly the processes around data that cause issues with data, not the people.

    However is that really what we should do? Do we want to do more and more cleansing at the point where people enter data? I’m not so sure that’s the case. The more that I deal with various applications and forms, the less I want to see too many fields and values required. Certainly there are pieces of data that are necessary for action, but I often find there are lots of additional fields that analysts want, but are more of a burden than a necessity.

    Most of us as data professionals design tables to handle the needs of an application. We put fields in tables that we expect to fill with data, based on a specification. However the real world is messy, and often the data we want to collect and store isn’t always available. The question is, do we force failures in the application or do we handle missing data?

    I don’t want to eliminate DRI, or go to an EAV models for lots of data. However I do think that we need to ensure we allow defaults and work with developers to allow data in our systems that might not be complete now, but perhaps will be in the future. We should work with analysts to help them build reports that can handle defaults or missing fields. We can work with developers to allow applications to request updates to data later and then design ETL that can efficiently fill in the updated information.

    Applications and software need to be flexible to work with problematic data. We have the ability, as data professionals, to help our clients still find meaning in data that might not be as complete as we’d prefer, or they’d prefer. However we can still find valuable information in what data they have..

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.5MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Architecture for Auditing

    In the analog world, when we need to provide an auditing system for checking on physical activity, we usually separate out the roles of people that might record or track activity from those that perform activities. The better separation we have between individuals, the better we can usually ensure our auditing records are intact, complete, and trustworthy. We certainly have issues in the real world achieving these goals, but overall things work well.

    In our digital systems, however, auditing seems to be bolted on to the systems and not designed with the need to separate duties and protect our auditing records from tampering. I would have hoped that many of the software platforms (Oses, RDBMSs, etc) over the years would have recognized this (along with security) and enabled auditing as a function separate from Administrator/root/sa/etc, but it hasn’t happened. Far too often, we’ve operated under the assumption that administrators can, and should, be trusted.

    In SQL Server, most all of the auditing mechanisms are enabled, managed, and reviewed by administrators. While viewing is certainly helpful, and perhaps necessary, for good administration, there ought to be a way to enable auditing that doesn’t require DBAs to manage it. There ought to be some role for auditors, one that allows an independent individual (or group), to access the record of actions taken on the instance.

    Perhaps it’s not as big a concern for the world as I think it is. In most cases, auditing data exists to help solve problems, not to ensure a legal record of activity for sensitive actions. If that’s the case, then SQL Server provides lots of choices and options for the auditing of your system. Extended Events, SQL Audit, and more are valuable tools for the DBA. I’m just not sure their value extends beyond being useful tools.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.4MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • FORMATing Dates

    I’m writing this post as a way to help motivate the #SQLNewBloggers out there. Read the bottom for a few notes on structuring a post.

    FORMAT is a function that was introduced in SQL Server 2012. It is designed to format dates and numbers as date/times. It was added to try and reduce the complexity and cumbersome nature of CONVERT and CAST.

    What I didn’t know, which I did like, is that the FORMAT command can be driven by language settings. For example, if I take a specific date, like today, I can reformat the date based on a language code.

    DECLARE @d DATETIME = GETDATE();
SELECT FORMAT ( @d, 'd', 'en-US' ) AS 'US English Result'
      ,FORMAT ( @d, 'd', 'en-gb' ) AS 'Great Britain English Result'
      ,FORMAT ( @d, 'd', 'de-de' ) AS 'German Result'
      ,FORMAT ( @d, 'd', 'zh-cn' ) AS 'Simplified Chinese (PRC) Result'

    The results of this code are different, based on the cultural settings. Of course, the Chinese settings is really the only good way to show dates without confusion.

    dateresults

    This is interesting in that if you can get the regional settings for the client, you can easily return data in a format that makes sense to the user. Of course, we typically don’t want to do too much formatting on the server.

    The default language is that of the current session. For dumb uni-lingual people like me, this isn’t an issue, but it might matter for people that speak multiple languages and might move from system to system.

    It is interesting in that you can easily format the data from a date with FORMAT. For example, I change orders to match the Chinese format:

    DECLARE @d DATETIME = GETDATE();

SELECT FORMAT( @d, 'YYYY/MM/DD');

    That gets me “2015/07/20”.

    I can easily change to other formats, adding in times, or even partial times that might not make sense. For example, the result below named “Hours” has only the date. The “Date and hours” has the date and the hour only.

    DECLARE @d DATETIME = GETDATE();

SELECT  'Default' ,
        FORMAT(@d, 'd')
UNION
SELECT  'Full Date' ,
        FORMAT(@d, 'YYYY/MM/DD hh:mm:ss.ffff t zzz')
UNION
SELECT  'Date and hours' ,
        FORMAT(@d, 'YYYY/MM/DD hh')
UNION
SELECT  'Date and minutes' ,
        FORMAT(@d, 'YYYY/MM/DD mm')
UNION
SELECT  'Hours' ,
        FORMAT(@d, 'hh');



--      ,FORMAT ( @d, 'yymmdd', 'en-gb' ) AS 'CleanUS'

    The results are:

    Date and hours    2015/11/20 04

    Date and minutes  2015/11/20 11

    Default           5/20/2015

    Full Date         2015/11/20 04:11:28.9600 P -06:00

    Hours             04

    I think that there are lots of reporting queries where the formatting of dates would be handy and easier with FORMAT than CONVERT. It certainly is more intuitive to read than seeing something like “,110” in code.

    I still have the habit of using CAST and CONVERT when I’m changing types, and I’ll continue to do that. Especially as FORMAT is really limited to date types.  However when trying to get dates to render in proper formats, it’s a good choice.

    SQLNewBlogger

    This post came about while I was checking on another issue. I happened to run into the FORMAT command and hadn’t used it much, so I spent a few minutes messing around. This post came out of around 5 minutes of experimentation and 15-20 minutes of writing.

    References

    FORMAT – https://msdn.microsoft.com/en-us/library/hh213505.aspx

    Formatting Types in the .NET Framework – https://msdn.microsoft.com/library/26etazsy.aspx