Author: way0utwest

  • More Azure Database Creation in Powershell

    I wrote about Azure SQL Database for T-SQL Tuesday #82, but I had a few mistakes in my process, at least from a cost standpoint. Since I’ll probably use this a bit more as I flesh out some CI/CD, I decided to document a bit more about the database creation process.

    The Old Code

    One of the problems I noticed right away was that I created an S2 level database in my post, which was too large. That wasn’t needed, and it might eat up my database credits. My original code was:

    New-AzureRMSqlDatabase –ResourceGroupName “Default-SQL-WestEurope” –ServerName “dkranchapps” –DatabaseName “TSQLTuesday” –Edition Standard –RequestedServiceObjectiveName “S1”

    In here, there are a few things that are obvious to me, and some that might not be. I won’t delve into detail on how to choose some of these, but I’ll define what they mean. There is a doc page for the cmdlet, which I used to check what I knew.

    ResourceGroupName

    This is the way you organize Azure resources. In my case, I have one resource group, which I named “Default-SQL-WestEurope”. This was the first group I made, intending this to be for SQL servers, and it’s in the West Europe region. Hence the name. 

    ServerName

    This is self explanatory, but you create servers to host resources in Azure. The one I created a long time ago for experiments was “dkranchapps”. I re-used it here, though I could have created another one.

    DatabaseName

    I’m not defining this. If you don’t know, stop reading.

    Edition

    This is the basic edition for your database. There are multiple options, but I used “Standard”, which was in the sample documentation code. However, a Standard incurs a charge from $0.0202/hr to $0.2016/hr. That doesn’t sound like much, but there are 720 hours in a 30 day month. That’s $14.44 to $145.15 a month. Again, not much, but this is per database. This could cause me issues with my $150/month credit.

    ServiceObjectiveName

    This is the scale within the Edition. I’m not sure why this the Edition is needed if we specify this here, but we can use a name or a value (GUID). In my case, I neglected to check the S2 syntax when I made the database. What I’d want for most things is a Basic or maybe an S level if this is short lived.

    Other Options

    There are other options. I can specify the subscription in a parameter, do a “WhatIf” to see what this would do, Force the command without user conformation, set a maximum size, and more. 

    While you might not need these options, I’d encourage you to look through the list and make sure that it’s not your intention to omit something that might be important in your situation.

    Creating a Basic Database

    One of the things I want to try is scripting the creation of a database in my CI process. That’s not quite simple, and it’s a task for another day, but here’s a basic call that I can work on adding to my builds.

    New-AzureRMSqlDatabase –ResourceGroupName “Default-SQL-WestEurope” –ServerName “dkranchapps” –DatabaseName “” –Edition Basic –RequestedServiceObjectiveName “Basic”

    Once I run this, I can check the portal, and sure enough, I see my database at the correct size.

    2016-09-21 10_52_49-SQL databases - Microsoft Azure

    That’s the first step to automating some builds. Now I need to worry about security and API keys, but for now, I can build a database.

    And, of course, remove it with Remove-AzureRMSqlDatabase.

  • Capture the Flag

    I’ve read a lot of science fiction in my life. Quite a few books talk about cyberspace and the interactions of humans and software. In Snowcrash there are AI systems defending systems. The Unincorporated Man series has AI systems that not only have sentience, they hide their capabilities from humans. Numerous other examples exist, including movies such as The Matrix, which popularized system programs independently able to evaluate and react to security issues.

    Science fiction is here, showcased recently at the DARPA Cyber Grand Challenge. You can watch a bit about the challenge or read a short article about what happened. Overall, it’s amazing. Seven systems ran various pieces of software that were targeted at finding vulnerabilities, patching their own and exploiting those of other systems. All without human intervention.

    In a visualization, I picture a team of Agent Smiths (from the Matrix) seeking out rogue programs that seek to wreck havoc inside of the operating environment. Or maybe a series of automated developers tracking down bugs, such as buffer overflows and correcting the code to remove the issue.

    Is this where static code analysis is heading? Automated repair, or maybe even real time repair? Could we actually have a software bot that might run through your Entity Framework (EF) application, tracking down SQL Injection vulnerabilities and correcting them? Even at a rudimentary level, I can imagine a bot that examines incorrectly parameterized queries and rewrites the code to properly manage the .NET code. Perhaps at some point we’ll even have bots that can understand queries from some application and build stored procedures on the fly that accept parameters, are called the next time a particular method is called, and can even grow to evolve with schema changes.

    I know the simplistic ways in which we might build this code would have their own sorts of issues, but I’m hopeful that at some point we’ll have semi-intelligent bots that can fix code to ensure it’s secure and reliable, while keeping the creativity originally built by the human developer.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.2MB) podcast or subscribe to the feed at iTunes and Mevio .

  • Analyzing Data in Real Time

    A few years ago I was listening to a presentation from Microsoft where they touted a customer that had real time analytics with Analysis Services. Since my exposure had been that cubes needed to be processed, a resource intensive process, I was wondering what this meant. Imagine my surprise to hear that the cube was being updated within 5 seconds of the changes in the OLTP system. Talk about real time.

    Of course, the definition of real time isn’t really well known. We linked a piece in this week’s newsletter that looks at a survey of companies and their investment in real time, streaming, data movement pipelines. The definitions from companies about real time range from sub-second to 24 hours. I’m sure I think that 24 hours is real time, but if you’re used to processing cubes weekly, maybe that feels real time. I tend to see real time on the order of minutes, maybe a few tens of minutes. Once we get to hours, that feels more like batch processing, albeit fairly rapid batch work.

    I am sure there are systems that need real time analysis, especially in some areas where decisions must be made quickly. As we move to more and more machine learning algorithms and automated intelligence, there will certainly be more call for real-time data movement. Yet another opportunity for data professionals, especially those that work with ETL pipelines. There will be tough problems, not only with moving data, but tracking lineage, recovering from issues, even weeding out bad data quickly.

    However, that’s an opportunity for you, not me. I bet there are a few data professionals out there that feel the way I do. There are great challenges in solving real time problems, and building systems that can handle high volumes of data. The thing is, I value my sleep. I value not working all-nighters, I value a balance in my life. While I find the problems fascinating and the money involved tempting, I prefer to work on more pedestrian systems.

    I’m glad there are people that want to work on very difficult problems, and I wish you all the best in taking advantage of these opportunities. I hope you’re well paid, and you have a great time building these impressive systems. I also hope to read about some of the amazing things you do, so please, share the knowledge where you can. This is a great, exciting time to work with data, and it’s one that I continue to enjoy every day.

    Steve Jones

     

  • Restaurants At Summit

    At the PASS Summit this year, there won’t be a Thursday night party. I think that’s fine, as it never seemed like a highly attended event, and the costs are high.

    That means that Thursday night is open, and I know lots of you will be looking for something to do. Or you’ll know someone that’s looking for something to do. In either of those cases, Andy Warren and I are looking to try and organize a few things. One of these will be around dinner plans for attendees.

    That’s where I’m looking for help from you. I need a list of restaurants in the downtown Seattle area.

    We’re looking for recommendations from those of you that have eaten dinner in Seattle. Give us a name/address, cost ($ or $$ or $$$) and how busy it was. We’re looking for those places that are walking distance or short Uber/Lyft/cab from the Convention Center area.

    Leave a comment with your recommendation and watch out for our announcement soon on the meetup plans.