Author: way0utwest

  • Making VSTS Deployment Changes to Databases without Breaking Your Application

    One of the things that I do a lot is demo changes to databases with CI/CD in a DevOps fashion. However, I also want to make some application changes to my sample app without breaking things. As a result, I’ve built a few ideas that work well for both situations. I found recently these ideas can help me when I need to actually upgrade or change my CI/CD pipeline.

    Note: DevOps isn’t a thing, it’s a set of principles, ideas, and culture that produces results. I will endeavor to ensure I call out the specific principles I use to adhere to DevOps ideas. In this case, automation and CD.

    Release On Your Schedule

    One of the principles of DevOps is that we use feedback loops to ensure information moves from right (operations) to left (development).  One way of doing this is to release more often, though this isn’t required. What we really mean is release on your schedule, when you want, and allow developers to get feedback on their work quickly.

    If a developer takes a month to build a feature, they don’t need hourly or daily releases. They need a release after the month is over (assuming testing, review, etc. has taken place). If releases occur on the 5th of the month, and the developer finishes on the 6th, they must wait a month before they get feedback. What I want to ensure is that we can release on the 5th, and also on the 6th if I need to.

    Upgrading my VSTS Pipeline

    I wrote in another post that I planned on upgrading my Redgate DLM Automation tasks in VSTS. I was doing this on a trip to a conference, and I didn’t want to start making the demo changes I’d make as I’d have to undo them, and it’s possible that I’d forget to clean something up. I hate making that silly mistakes, so I needed a way of testing my build and release without breaking demos.

    I decided to use a technique that I use in presentations when I’m talking process and not code. In those cases, what I deploy doesn’t matter, but if I change random objects, sometimes I break the application using the database. As you can see here, I needed to do a bunch of tests, and repeat some.

    2017-04-01 11_52_03-ST Pipeline_Mobile - Visual Studio Team Services

     

    Deploy Often

    When I talk with Redgate clients, and they are starting to get comfortable with the idea of deploying on their own schedule, they will sometimes make innocuous changes that trigger a deployment they can test, without breaking things. For example, a user might take this stored procedure (partially shown):

    ALTER PROCEDURE [dbo].[GetEmailErrorsByDay]
      @date DATE = null
    /*
    Description:

    Changes:
    Date       Who         Notes
    ———- —         —————————————————
    2/14/2017  WAY0UTWESTVAIO\way0u   
    */
    AS
    BEGIN

    IF @date IS NULL
      SELECT @date = DATEADD(DAY, -1, GETDATE())

      SELECT Errcount = COUNT(*)
       FROM dbo.EmailErrorLog
       WHERE EmailDate = @date

     

    and make a small change. Perhaps they add SET NOCOUNT ON, or maybe they’ll add a comment. I’ve even see someone add this code:

    …

    AS

    BEGIN

    SELECT 2 = 2

    …

    These aren’t big changes, and certainly choose which stored procedure to change (one that isn’t used a lot or is critical). These are changes to test your process, gates, approvals, deployments, etc.

    I decided to try something else. I tend to write this code when I want to test changes, since I can be additive with a procedure like this:

    CREATE PROCEDURE Get7

    as

    SELECT 7

    Or I can modify things with this:

    ALTER PROCEDURE Get7

      @plus int

    as

    SELECT 7 + @plus

    In either case, I can see if my changes go through. Since I often deploy to multiple environments (QA, Staging, UAT, production, etc.), and I don’t always have deployments go all the way through (see my image above), I will usually end up creating Get7, Get8, Get9 as subsequent procedures. This way I can continue to commit new changes to the VCS, get new builds, and get new releases.

    I can also do this with tables. My favorite is MyTable, MyTable2, etc. I usually just have an integer MyID column, but I can add other ones to test the ALTER process. I can even use this to test data movement, static (reference/lookup) data, or anything else to do with tables.

    Eventually (hopefully) I get clean deployments all the way through to all environments.

    Cleanup

    I sometimes get collissions, where a test will return the error that “Get10 exists”, and I’ll move on to Get11. However, I don’t want to leave those objects in all environments. After all, likely I’ll find a way to improve things in the future and I’ll want to repeat this testing.

    This usually is one last deployment for me. I’ll delete these objects in dev, and then deploy the changes all the way through to production. This allows me to test my checks to prevent data loss, if I have any. Including if approvers actually read scripts Winking smile

  • More Power in Your BI

    I’ve seen lots of data visualization tools over the years. Cognos, Microstrategy, ProClarity, and more. While I haven’t been a BI person for most of my career, in many of my positions, someone has wanted to try some new tool and eventually I’ll get involved somewhat. I remember first seeing the OLAP cube browser in SQL Server 7, and showing my boss a quick demo. While it wasn’t something that you could let an end user have, it also got my boss excited, and they started a project as I was leaving to implement some OLAP for the sales department.

    When I first saw Tableau years ago (2008?) at TechEd, it was the first tool I’d seen that was really slick for the end user. I expected they’d grow quickly, and they have. Many people have looked at the Tableau tools as the standard for data visualization. For years nothing looked close, and as much as I appreciated Microsoft moving PowerPiviot and other tools to Excel, they weren’t as nice to use as the graphical tools.

    That changed with Power BI. When I first saw this tool, instantly I could start to see the power of the tool. Even altering and changing a tool had an ease and power that I hadn’t seen since the Tableau tools. The initial release had lots of limitations and was just on the web. It was slow, and refreshes from remote data weren’t great. There were security limitations, and I wasn’t sure that the vision for the product made sense from Microsoft, even though I could see tremendous potential.

    Things have improved, and I am somewhat amazed with the power of Power BI. A simple sales report is interactive. I can click on something intuitively and the values I see will adjust to focus on that item. What’s more, other graphs on the screen adjust. However, I don’t have to stick with simple graphs, I can add better imagry, such as this wine report, or this airline maintenance report that drives work. I was especially impressed with the analysis of Stephan Curry (which won the contest). Some of those reports I couldn’t imagine trying to build with other tools, especially SSRS.

    Power BI is a great tool, and if you need to provide visualizations for end users, I’d urge you to take a look. The Power BI Desktop tool is great for modeling and working with data. I find myself using that for quick looks at data, building a graph in a way that’s easier than in Excel. I went through Microsoft’s EdX course, and was somewhat amazed to see the vast capabilities available in the product. Guy in a Cube, Adam Saxton, has a YouTube channel where he and Patrick LeBlanc (@PatrickDBA) bring you constant training and tips on how to get the most out of the tool.

    What’s more, I keep finding more and more Power BI blogs and posts that I can add to Database Weekly every week. So many people are experimenting and finding ways to better analyze data with Power BI. This week we have a continent slicer, data privacy, collaboration, and more. With montly releases, I find that people are constantly digging into the possibilities and helping you learn with them. There are even developers building custom visuals that you can download. There’s even one for acquarium lovers. If you want to learn about these, Devin Knight blogs regularly about custom visuals and we include many of those links here and on SQLServerCentral

    Power Bi is a great way to empower your users, reduce your reporting load, and make everyone happier. Power BI is a great way to let users experiment with data and actually decide what reports are worth investing in more with IT resources to perhaps optimize the performance of certain visuals. Give it a try to day, especially the desktop version, and see how you can easily start to examine data that might be important to you. Even the SQLDBAWithaBeard finds Power BI helpful.

    Steve Jones

  • AI Helpers or Replacements

    It’s interesting to look at the data business and how companies view DBAs, database developers, BI developers, data scientists, and more. There are some companies that really see value in our services, and I’m grateful for that. I’ve been gainfully employed for well over two decades to work with data. There isn’t much standardization in our jobs or what we’re expected to do, but I’ve grown comfortable with that. That’s one of the reasons the people working with me, my coworkers, are more important than the work. The work is the work.

    As our systems become more advanced, there is concern over how much less some of our skills might be needed with new AI type systems. Will the move to smarter systems mean there will be more opportunity for us? Or less? Certainly in the long term there might be less jobs if systems become really capable, but in the short term I don’t think so. As much as Microsoft has improved SQL Server, and they’ve done great things with easier HA configs, the Query Store, Adaptive Query Processing (coming) and more, they aren’t replacing many of us. Maybe a few, but I think there’s still lots of technical work.

    Darmesh Shah wrote a nice piece on AI and how it will help many of us in our jobs, providing the easy information and guidance for us to focus our skills. He sees bots as helpers, which to me presents new opportunities to interact with and work with customers and data. We will find ourselves more capable with help from AI, not replaced. As we get better machine learning or other adaptive algorithms, we’ll actually find new ways to work with data. This should provide us with new opportunities and new types of jobs that we might grow into. Plenty of people want to dismiss the data science jobs as a popular area where anyone can claim those skills if they know a statistical function and can query data in R, but there are real jobs in those areas, and there are opportunities in new companies that might not have existed in years past.

    There are going to be some amazing new ways that data and more intelligent algorithms will help us see our world in the future. There will also be scary ones, and many we don’t know if we can trust. Somewhere in there, the world will become very, very interesting for those of us that work with data on a daily basis, looking for new ways to extract information from all the bits and bytes that we store. Once again, this is something I do look forward to as a data professional.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.6MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • The Value of Your Personal Data

    Years ago I read Being Digital by Nicholas Negroponte of MIT. In the book, he relates a story about crossing the border from the US into Canada and a Border Patrol agent asking the value of his laptop. He said millions, not because of the hardware, but because of the data. I thought that was interesting, and I somewhat agree. The data I have on my machines, or really now, in cloud services, is more valuable than the hardware. The hardware is barely worth anything to me, including the time it takes to reinstall my applications, because I really treat my laptop as a piece of cattle. I could drop it, render it completely unusable and I’d be back up, running, and productive with new hardware in less than 4 hours.

    Then I saw this article on ransoming data for iCloud users, where the request is for US$75,000. This is for Apple itself and not individuals, but what if it were for each user? What would I pay to avoid losing all the data on my mobile device?

    It certainly wouldn’t be $75,000 despite the fact I think some of the data I have is precious. I love the pictures and video of my family most of all, but if I lost some of them, maybe all of the digital storage, would I just pay a ransom? Probably not. What about all the code in a VCS for my company? Would we pay a ransom for that? Likely we would, though perhaps not. We might think about reconstructing the code, especially if we were a small company.

    Ultimately I think there isn’t a lot of money to be made with individuals’ data. Too many of us older folks remember when we did lose physical objects and they were just gone. The idea of losing things is painful, but not inconceivable. For younger people, it seems that much of the data we produce and consume is transient and isn’t necessarily that valuable to us. My kids love Snapchat for the ability to create and lose memories. I dislike it for the same reason.

    Much of the data we personally have can be backed up, and possibly recreated if it isn’t. Many of our digital records have the characteristic that while we may have a copy, often some business or organization has another copy. Requesting new data is easy, and while it might cost some time and money, it may be preferable to the idea of paying a ransom, and potentially having to continue to pay again in the future with some virus still on our systems.

    I don’t quite know what to think about ransomware and how this might evolve. I suspect that this will always be a problem in some way, just as hardware failures will likely plague us forever. The only solution I have is to create backups regularly that contain versions, and can be restored to separate, clean devices. I’ve gotten away from my own personal offline backups, mostly because of data size, but I do continue to try and keep at least 2 or 3 backups going to different locations and with different services because my data is at least that important.

    Steve Jones