Tag: python

  • Do I Need to Learn Linux and R and Python?

    The big changes coming in SQL Server 2017 are the Linux version and the addition of Python as a language used to analyze data. In SQL Server 2016, we had the R language added, and quite a few people have been on crash courses learning R, or worrying about the fact they don’t understand how to write an R script. Now many of these same DBAs and data professionals worry that they also need to find time to pick up some Python and Linux.

    I really like Python. I spent some time with the language and still dabble here and there. It’s a clean, easy to write and read language. My son took an intro CS course this spring in college and we shared some programming thoughts and ideas in Python. If you’re interested, I’d recommend the Dive Into Python book, as well as the meeting series from SDTIG. If you want to learn R, I’ve been dabbling at TryR from CodeSchool. There are also resources from Grant Fritchey and Buck Woody. For Linux, there are tons of resources out on the Internet. I don’t have a recommendation, but feel free to leave one in the discussion.

    If you are a data professional that works with data scientists or others that might want to incorporate some machine learning or complex analysis inside SQL Server, do you need to learn R and Python? Is that the feeling that you have? Are these languages require to ensure that you can perform your job?

    I’d argue that for many of us, we don’t need to understand the language or be able to write R or Python scripts that are being run in SQL Server, or even in a tool like Power BI, to support our users. We should understand how to execute these scripts, and maybe how to correct simple typos. A working knowledge of HTML has come in handy many times for me when helping users display their data and a quick change was needed. However, I wasn’t the person you’d call to design a complex CSS and interactive HTML pages. I can just fix a few simple mistakes.

    You might feel differently, but I don’t think most SQL Server professionals, developer or DBA, really need to learn R or Python. Feel free to do so if you like, and I’d put in a recommendation for Python, which I really like, but I don’t expect that either one will be a significant part of your job in the future. Learn Linux if you want to experiment with another OS, but for the most part, SQL Server on Linux is just SQL Server. Learn some rudimentary Linux skills, but spend most of your time making sure you understand the SQL platform and you’ll be fine.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.7MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Python in SQL Server

    The Microsoft Data Amp event was this week, with a number of SQL Server announcements. We now know this is going to be called SQL Server 2017, which is no surprise. CTP 2.0 was also released, which includes a few improvements, but one in particular caught my eye. Python code can now run inside SQL Server, meaning we are not limited to the R language for data analysis. That’s exciting to me and I think that’s a great change. It’s one that’s been rumored for some time. I like Python, and think it’s a nice, elegant language. My son has been learning it in college and I’ve spent some time playing with it after the SDTIG got me interested with their beginning Python and Data Science modules. Since then I’ve spent a little time learning R, but find it to be a cumbersome language to work within. Having code embedded in a string in a SQL Server stored procedure made things harder, and I kept hoping that Python would be added.

    It’s there now and you can play with it. While you might debate whether this is a good use of SQL Server resources, I think for quick, short, specific analysis calls, it’s a good choice. Certainly complex calculations and data manipulations are not always efficient in T-SQL and having another option is great. You’ll want to be careful when you run Python or R code, but these are good options to have. There is also GPU processing support is available to take some load off your main CPU.

    CTP 2 is supposedly feature complete and there are more changes than just Python. Tempdb has a better setup experience, we get the first version of graph data and query support, resumable online index rebuilds, adaptive query processing, and more. You can read the announcement, which isn’t heavy on technical details, but gives you an idea of what’s changing. I’m not much of an SSAS person, but there are some enhancements in this platform as well.

    There are other improvements and announcements this week. Azure Analysis Services is generally available (no longer beta), as as Azure Data Lake Analytics and Data Store. DocumentDB is improved and some Cognitive Services are GA, and in general, Microsoft is all in on the analytics and deep learning.

    The world of data in the Microsoft world is growing and changing rapidly. We’ll have a new version this year, just a little over a year after SQL Server 2016. I expect that we will continue to see more, and new changes, occurring at a rapid pace in the future. None of us will completely keep up and become experts in all areas, but we have lots of options for new learning and experimenting with advanced features. I’d encourage you to take a little time here and there and read about and play with the new features. If you have Docker, you can get a version of SQL Server 2017 up and running without installing anything on your machine to see if there are any changes that might help your organization take better advantage of  the data you work with every day.

    Steve Jones

     

  • A Python SQL Server App

    I got a link recently from Microsoft on building apps easily for SQL Server. At the top of the page, they ask you to pick a language and OS. Since I’ve done a little Python and I used Windows, I chose that option.

    The page looks to walk you through a simple app. I decided to try this out and see if it works, following the instructions. Since I already had SQL 2016 installed (and newly upgraded to SP1), I just connected and verified it was there.

    2016-11-22 11_50_06-powershell

    I didn’t want to go back to Python 2.7, and I’d rebuilt this machine for testing, so I used Chocolatey to quickly get Python 3.5. I’ll have to adjust code to match this version.

    2016-11-22 11_52_38-cmd - choco  install python (Admin)

    One python was installed, I needed to get virtualenv and then create a folder. I used my regular git folder since it’s really, really simple to get a project with version control running. Unfortunately, when I went to get pyodbc, I had an error.

    2016-11-22 12_18_53-cmd (Admin)

    This is going to use C++ tools to build the pyodbc files, so I need to get those installed. I can do that from Visual Studio.

    Once I run those and install them, I can now get pyodbc. There is a build error, but I seem to have the module. My next step is to create my db. I also create a login that is db_owner for this.

    2016-11-22 12_41_22-powershell

    Now it’s time for the code. That’s easy enough, so I copy the code into VSCode and change the instance and user information.

    2016-11-22 12_42_11-crud.py - Visual Studio Code

    I also had to replace the print statements with parenthesis around the quoted items, which is required in Python 3.x. Once that’s done, I start the program and it appears to work.

    2016-11-22 12_39_25-cmd (Admin)

    If I check from SSMS, sure enough the program has worked.

    2016-11-22 12_39_54-SQLQuery1.sql - localhost_SQL2016.SampleDB (PLATO_Steve (62))_ - Microsoft SQL S

    The second part of the tutorial uses the Django Web Framework to setup a simple site. Once again, I get a pyodbc error and there’s nothing that works. Fortunately I found the issue is with Python 3.5 and not having the proper binaries in the default PyPI directory. I resolved it with this link.

    2016-11-22 13_26_45-crud.py - Visual Studio Code

    However, there were other errors, which I suspect are related to Python 2.7 v Pyhton 3.5. Rather than solve those, I went on to the columnstore demo. In this, you create a table with 5mm rows and then run a query against it from Python. I did that, then created the columnstore index, then ran it again. The results are below.

    2016-11-22 13_46_23-columnstore.py - Visual Studio Code

    Note: I had to create the variable, tsql, and used this line of code:

    tsql = “SELECT SUM(Price) FROM Table_with_5M_rows”

    Python Works

    Despite some issues, which are probably my fault, this is a good introduction to how you might use Python with SQL Server. Since we now have SQL Server on Linux coming, and there are lots of Python developers, this might be a good place for some of you to learn a bit about connectivity with Python if you ever need troubleshoot that kind of setup.

  • Advent of Code Day 1, Puzzle B

    As I continue through the Advent of Code, albeit slowly, I’m solving each puzzle 3 days. I worked through 6 of them in Python before moving on to other languages, and this is my chance to catch up with both PowerShell and T-SQL. I likely won’t post all solutions, but I was having fun rewriting code, so here are the ways I looked at things.

     

    Note: You should try this on your own. I logged into the AventofCode with GitHub and things worked great for me.

     

    Go on, give it a try.

     

    I’ll wait.

     

    Solutions coming.

     

     

    Python

    I started here, using iterations, which are very powerful in Python. In this case, I took advantage of the multiple variable assignment in Python to enumerate the array and get each value and index. I think use a comparison to determine if I add or subtract one. Finally, an IF returns the current index if I hit –1. I should probably have a break in there for efficiency as well.

    def calculate_negative(directions):
    start = 0
    for i, c in enumerate(directions):
    if c == '(':
    start += 1
    else:
    start -= 1
    if start == -1:
    print(i)

    PowerShell

    This was a bit trickier for me. I wasn’t sure how to work with a string and pull out values. I did some searching and ran across the .ToCharArray function. That doesn’t feel like the best way to do this, but I decided to use it.

    The rest of the function is similar and gave me the correct answer, so there you go.

    $count = 0
    $floor = 0
    foreach ($c in $input.ToCharArray())
    {
    if ($c -eq '(') {$floor += 1}
    elseif ($c -eq ')') {$floor -= 1 }
    $count += 1
    if ($floor -eq -1)
    {
    $count
    break
    }
    }

    T-SQL

    A more complex situation here, given that we need to work through a string, calculating a running total. I broke the input up using a tally table and a splitter with substring. This was fairly easy, and not complex as I was just getting individual charaters.

    Then it was a simple running total of the CTE to get me the totals at each point in time. This worked well, but I needed the first –1 total.

    Finally, I used the outer query to get the min value out of the code and use that, which gave me the lowest value where the –1 occurred.

    WITH tally (n)
    AS
    ( SELECT ROW_NUMBER() OVER (ORDER BY (SELECT NULL))
    FROM (VALUES (0), (1), (2), (3), (4), (5), (6), (7), (8), (9)) a(n) -- 10
    CROSS JOIN (VALUES (0), (1), (2), (3), (4), (5), (6), (7), (8), (9)) b(n) -- x 10 = 100
    CROSS JOIN (VALUES (0), (1), (2), (3), (4), (5), (6), (7), (8), (9)) c(n) -- x 10 = 1000
    CROSS JOIN (VALUES (0), (1), (2), (3), (4), (5), (6), (7), (8), (9)) d(n) -- x 10 = 10000
    )
    , direction (n, d)
    AS
    (
    SELECT tally.n
    , d = CASE WHEN SUBSTRING(@input, n, 1) = '(' THEN 1 ELSE -1 end
    FROM tally
    ), currfloor
    as
    (SELECT
    d.n
    , 'currentfloor' = SUM(d.d) OVER (ORDER BY d.n ROWS UNBOUNDED PRECEDING)
    FROM direction d
    )
    SELECT MIN(currfloor.n)
    FROM currfloor
    WHERE currentfloor = -1