Voice of the DBA

Tag: career

  • Practical Web Scraping–Getting Started

    As part of my learning goals for 2018, I wanted to work through various books. This is part of  my work with Python.

    After going through a few first chapters, I decided to start my February learning with Practice Web Scraping for Data Science, which looks at data acquisition using Python to pull data from the web. I found the book interesting and also this would be a nice setup for two of the other books (Power BI and natural language processing).

    Like many people, I find lots of data on the web, but I’m constantly struggling to get it into a database. I find myself going through gyrations at times to get data. Even with the cool features of Power BI, it hasn’t been as smooth as I’d like to get data, so I thought this would be a good book.

    Part 1

    The first few chapters of the book are about the basics of web scraping. We learn what this means, and get a little bit of a tutorial on who uses this technique, with some specific examples. We also get a basic python tutorial, which I skimmed. I know a bit about Python and this was a very basic, getting started.

    The next part of the early book deals with the basics of what http transport looks like and how some networking works. This is interesting to me, though not sure it matters for scraping. We’ll see. There is some discussion on GET and the http standard, so perhaps that’s helpful. It is good to at least know what codes might come back or what headers or parameters you need to use.

    The third chapter starts to get code working.  It opens with a discussion of HTML and how you can examine the structure of pages in your browser. This is a good reminder and basic tutorial of some of the web page developer tools that exist in your browser and that you might want to use when trying to build applications, especially those that scrape pages. There is also a basic CSS tutorial, which was good as I needed a little refresher. I rarely deal with CSS stuff, leaving that to others.

    The last part of the chapter starts with the BeautifulSoup library, which is built to parse out text, and specifically, makes working with HTML easier. The examples are with a Wikipedia Game of Thrones page, but I added some examples, trying to translate this to a sports page. It worked OK, and I learned a few things.

    The last part looks at Regular Expressions with BeautifulSoup and how you can search out elements and then start to copy data. It’s more complex and tedious, but then again, lots of programming is tedious. Once it’s working, it’s amazing.

    Experimenting

    I started to work with this in Azure Notebooks as a different way of tracking some work in Python. I’ll want to store things in a file at some point, but for now, this lets me start and stop learning and keep track of where I am without worrying about files and names.

    Not sure if anyone can access it (it’s marked public), but my project and notebooks are here: https://notebooks.azure.com/way0utwest/projects/web-scraping-with-python

    I ran some of the early scripts, which are just getting you used to working with Python and accessing web pages. I then copied some examples from my Calibre view of the book and executed them. I even tried to experiment a bit.

    One note: copying the code seems to leave some invalid character in there for Azure Notebooks, so I ended up editing the beginning of every line to remove the offensive character.

    This got me the basics of working with web scraping. Now to try and grab some data from another page and see what I’ve learned.

  • What is my why?–T-SQL Tuesday #111

    tsqltuesdayThis is an interesting question from Andy Leonard for T-SQL Tuesday: What Is Your “Why”? He is this month’s host and asks us to talk about our motivation, our passion. The question is: Why do you do what you do?

    Why I Do What I Do?

    I have had a great career in technology. Moving from developer to system administrator to developer to manager, and back through various incarnations. I have had success in lots of positions and helped improve systems and code, teaching and mentoring others along the way.

    Almost twenty years ago, I participated in a user group and helped someone solve an issue. That inspired me to start writing articles that detailed how to build solutions that had helped me and my employer. Over time that led to SQLServerCentral and regular speaking and writing.

    My career in databases really started for the money. I worked with a DBA that was better paid than any sysadmin or developer, and began to study and work with SQL Server. As I earned more, I thought I’d made a good decision, but over time I realized that there was a point where money was less important to me.

    Time became the most important thing to me.

    There have been many times where I would have traded more time for less money, though that isn’t quite the case now as I have two children attending college next year. Still, in the middle of my career I wanted to, because of the value that I placed on time with family and friends and living my life.

    I started to give back with writing, speaking, teaching because I had people do that for me, and because I wished I’d had more. I wished that more people had shared solutions and ideas with me earlier in my career.

    I also believe that we fundamentally ought to help others in life when we’ve been helped or been successful. We ought to pay it forward where we can, but more so we need to pay it back. It’s a bit of a social contract with others, a bit of a desire to be the rising tide that lifts all boats, and a bit of a moral obligation I feel to help the world because I’ve been blessed with success.

    Everyone walks their own path, makes their own decisions, and follows their own moral code. I try to follow mine, and I hope to inspire others to help others where they can.

    It’s why I do what I do, and I’m lucky that I’m able to do so, both as my job, but also because I am able to help others.

  • Pro SQL Server Linux–Moving to Linux

    As part of my learning goals for 2018, I wanted to work through various books. This is part of my series on Pro SQL Server on Linux from Bob Ward.

    It’s not until Chapter 10 that we get back to Linux specifics. Most of the chapters in the middle are general SQL Server items that are good to know, but not are useful for experienced SQL Serve professionals. They’re worth skimming as you might learn some things you didn’t about the features.

    In Chapter 10, the focus is migrating to Linux. Bob presents a number of options, including one I hadn’t heard of: the DEA. This is the Database Experimentation Assistant, and I hadn’t heard of it. I don’t do a lot of migrations as I start and stop on SQL Server, but this allows you to do some A/B testing of upgrades. I’m sure it’s not perfect, but that’s something that is definitely needed.

    Most of the chapter is about versions, from Oracle, or from PostgreSQL. The Oracle parts walks through the migration, while the PostgreSQL part tries to show where features in SQL Server exist or don’t in PostgreSQL. In a number of places, Bob found projects that provide some functionality in PostgreSQL that is in SQL Server, but didn’t test things.

    The last chapter of the book deals with containers, and that was one of the more interesting things for me to read. I’ve been on a bit of a container kick, digging in more and more, so I was curious to see how Bob described and talked about the technology. I think he did a good job, but I still think a couple courses I’ve watched on Pluralsight from Nigel Poulton really covered this well, so perhaps I glossed over this a bit.

    I liked that Bob includes lots of code and links to help you get started, so if you read this book, spend time on containers. They’re the future, and for now, Linux containers rule.

    Overall Review

    I learned a few things about SQL Server from the book, but far too much of it was basics and beginning SQL Server stuff, not Linux stuff. It was a good refresher, and I’m not disappointed in the time I spent, but I was hoping for more stories and more Linux.

    The lesson really is that SQL Server is SQL Server, with few differences on the platform. As Bob mentions, you shouldn’t care about the platform and just pick what works.

    The basics of SQL Server did make this hard to read, and as a result, I started strong at the end of December and hit a lull for almost two weeks where I wasn’t motivated to skim through basic T-SQL, security, etc. I realized around the 20th that I was running out of time and crammed a bit in.

    If you don’t have a lot of experience with SQL Server, this is good. If you know SQL Server, but want to know Linux differences, I might recommend digging into some of the more specific Linux blogs, or the SQLServerCentral Stairway series.

  • Learning for Feb

    By the time you read this, I’ll have decided, but I just finished the first book and need to pick another. This post is more for me than anyone else, helping me to sit down and think for a minute.

    I don’t have any pressing need to work on any of the other books for work or career. It’s more that I’m interested in various topics, so I need to pick a direction. I’m also cognizant of the future and which books I want to go through.

    Recap

    Here are the books I purchased and plan to read this year:

    The question is, what’s next?

    I’m really not sure, but I do know that I’ll be relatively less busy in February, really busy in March, and April will likely be like January. As a result, I want an easier book in March.

    The challenge for me is understanding what’s easy and hard. I think the Azure/Xamarin one will be hard as I’ll need to practice skills that aren’t useful or tangential to work. The Power BI stuff is likely the opposite. The Python stuff is probably in the middle.

    The Plan

    My thoughts right now are that I need to research for a couple days. It’s Jan 31 as I write this, and I’m on vacation for a couple days. That’s both good and bad. Bad because I want to get away, good because I can find some spare time to just read and no do.

    With that in mind, I’m going to read a chapter of 3-4 books and then decide in which direction to go. By the time this publishes Monday, I should have made a choice, but I’ll be busy working on other stuff, so I won’t get to update this post.

    The four to get through a chapter on are:

    Wish me luck.