Voice of the DBA

Tag: python

Practical Web Scraping–Getting Started

As part of my learning goals for 2018, I wanted to work through various books. This is part of my work with Python.

After going through a few first chapters, I decided to start my February learning with Practice Web Scraping for Data Science, which looks at data acquisition using Python to pull data from the web. I found the book interesting and also this would be a nice setup for two of the other books (Power BI and natural language processing).

Like many people, I find lots of data on the web, but I’m constantly struggling to get it into a database. I find myself going through gyrations at times to get data. Even with the cool features of Power BI, it hasn’t been as smooth as I’d like to get data, so I thought this would be a good book.

Part 1

The first few chapters of the book are about the basics of web scraping. We learn what this means, and get a little bit of a tutorial on who uses this technique, with some specific examples. We also get a basic python tutorial, which I skimmed. I know a bit about Python and this was a very basic, getting started.

The next part of the early book deals with the basics of what http transport looks like and how some networking works. This is interesting to me, though not sure it matters for scraping. We’ll see. There is some discussion on GET and the http standard, so perhaps that’s helpful. It is good to at least know what codes might come back or what headers or parameters you need to use.

The third chapter starts to get code working. It opens with a discussion of HTML and how you can examine the structure of pages in your browser. This is a good reminder and basic tutorial of some of the web page developer tools that exist in your browser and that you might want to use when trying to build applications, especially those that scrape pages. There is also a basic CSS tutorial, which was good as I needed a little refresher. I rarely deal with CSS stuff, leaving that to others.

The last part of the chapter starts with the BeautifulSoup library, which is built to parse out text, and specifically, makes working with HTML easier. The examples are with a Wikipedia Game of Thrones page, but I added some examples, trying to translate this to a sports page. It worked OK, and I learned a few things.

The last part looks at Regular Expressions with BeautifulSoup and how you can search out elements and then start to copy data. It’s more complex and tedious, but then again, lots of programming is tedious. Once it’s working, it’s amazing.

Experimenting

I started to work with this in Azure Notebooks as a different way of tracking some work in Python. I’ll want to store things in a file at some point, but for now, this lets me start and stop learning and keep track of where I am without worrying about files and names.

Not sure if anyone can access it (it’s marked public), but my project and notebooks are here: https://notebooks.azure.com/way0utwest/projects/web-scraping-with-python

I ran some of the early scripts, which are just getting you used to working with Python and accessing web pages. I then copied some examples from my Calibre view of the book and executed them. I even tried to experiment a bit.

One note: copying the code seems to leave some invalid character in there for Azure Notebooks, so I ended up editing the beginning of every line to remove the offensive character.

This got me the basics of working with web scraping. Now to try and grab some data from another page and see what I’ve learned.

February 15, 2019
Learning Goals–May Update

It’s 5 months in, and this is hard. I find that the time I’ve picked each week sometimes gets bumped by other items, which do have a higher priority. More on this later.

Extended Events

An epic fail here for May. I started with one week and managed to get an hour watching part of a PASS session and building a few questions. However, the second week I got busy with some family stuff that impacted my work week. With deadlines and travel the third week, I skipped this.

The third week I was gone, and the fourth, well, only a little focus, though I did start to write a short piece on using Jon Kehayias’ script to convert a trace to an XE session. We’ll see if I get this done.

Fifth week, not sure. That Tuesday will be in the June update.

Python

Not a lot better than the XE stuff. Week one I had some family issues get in the way of my Friday learning and ended up skipping this as I was away from home. With a busy weekend, I didn’t make this up.

Week 2 I was prepping for week 3. Week 3 I was gone.

This brings me to week 4. While I like to try and do something, I really didn’t get much done during the week. I am looking to add Python questions to the mix on SSC, so I have taken a little time during the week to build a couple questions. Going through basics like lists and dictionaries helps me come up with things that I think people might not know and can get inspired to learn a little with a question. That helped me get a few things done this month.

I did take time to go through part of a Pluralsight course in week 4, about 90 minutes of total time, some of that pausing the course so I could generate a question and practice some Python skills.

I keep a few browser tabs open with Python links to try some problem solving, but I keep not making time.

Moving Forward

I didn’t travel much until mid May, but I had a very busy life outside of work and didn’t make much time to learn away from work. Fortunately my work is related to these topics, so I can take a little time for learning.

As I close out the first half of the year, I’m somewhat disappointed with progress. I definitely continue to learn, as my job requires it, but I’m not getting the focused time in these areas. Looking forward to next year, I want to aim for topics that are more tightly integrated with work so that I get more practice. Here, even though I’m doing something, I’m not as focused.

For June and further, I’m going to shift my times around and pick what I think are quieter times for me to get something done. Since I can ride a bike at home and watch Pluralsight, I’ll pick those times for at least passive learning. Then I’ll set aside a shorter time period, 30 minutes, for practice at other points in the week when I have found myself with time.

May 30, 2018
Learning Goals–April Update

Here is the update: I did better.

Extended Events

I did a little more multi-tasking this month, ensuring that if I rode the stationary bike for exercise, I had my tablet and watched some Pluralsight or read some content.

Here I learned a few things:

Predicates

These short circuit, and this means that we want to write our predicates to end early if possible. We also want to write our predicates to look at local data before global data so that we can stop event code as early as possible.

There is also a 3000 byte limit, so be careful with structuring the event predicate statement.

I also found that u_tables.sql lets me rebuilt the default sessions for an instance. It’s in the install folder. That’s handy as there are times that I’ve removed the default sessions or changed them and I’d like to get back to the beginning.

Event Sessions

The structure for event sessions is like DDL for other objects. These are always created with events, but we can remove all events. I’m not sure why you want the placeholder, but it’s an interesting anomaly. I suspect that this is a hole in the DDL parser.

I also learned you can drop sessions without stopping them. Convenient? Sure. However, this means that I could make a mistake and not get a question about dropping a session that I, or someone else, is using.

Changing event or target configuration means dropping and adding back the item. That’s good to know. No way to alter an event predicate.

Python

I made it a point to do some coding every week, and work on Python. I did some fundamentals, ensuring I understood how modules work and are imported. I spent some time on variables and mutable/immutable items. I wanted to be sure I understood how these worked compared to other variables. It’s good to know that some of the assignments don’t create a new variable, but merely set a pointed to the existing item.

I set up a Jupyter notebook in Azure, which was interesting. I need to use this more, and perhaps do some coding here on the Advent of Code or other projects. I solved a couple more days of puzzles, which was fun. These are challenging, and it took a little time to work through items.

I didn’t do this every week, as I had some travel this month, but I did make an effort to work in python a bit more.

April 30, 2018
Python Jupyter Notebooks in Azure

There’s a new feature in Azure, and I stumbled on it when someone posted a link on Twitter. Apologies, I can’t remember who, but I did click on the Azure Notebooks link and was intrigued. I’ve gotten Jupyter notebooks running on my local laptop, but these are often just on one machine. Having a place to share a notebook in the cloud is cool.

Once I clicked on the link, I found these are both R and Python notebooks, as well as F#. These allow you to essentially build a page of code and share it. It’s kind of like a REPL, kind of like a story. It’s a neat way of working through a problem. I clicked the Get Started link to get going and was prompted for a User ID.

Once I had my moniker setup, the next step was to edit my profile. That’s more important than a library, right?

Of course, I needed to fill out the profile with my avatar and some information.

Next I need to create a library, which I’m guessing is a collection of notebooks. I clicked the link and had to enter a name. I decided on the classic HelloWorld name. I decided to keep this public, as I might want to share this with others.

I’ve got a library, now let’s add something. I clicked the Readme.md, but it didn’t load. There was nothing there, as this is a blank file.

I discovered I could right click the file in the list. This lets me edit it. Strange UX, but whatever. The file uses markdown as editing, which is fairly simple, but consists of a few characters to designate titles, lists, etc.

I entered some text, and then my readme appeared below my notebook list, much like it goes on Github. My screenshot got taken after I’d experimented a bit, so you see a couple python notebooks as well.

From there, I could add a notebook. I have choices. I started with Python, since that’s one of my learning goals.

I give the notebook a name and create it.

Once this is created, it appears as a Jupyter notebook. Essentially I have a repl-like command area, and once I enter code, I can click “Run” to execute it. You can see that my Hello, World program ran.

I can enter other code, and I’ve done a few things, just to practice some basics in Python. I’m working through some courses, and I’ll enter code in here to practice concepts.

Jupyter notebooks are a good way of working through a problem and showing flow. They’re especially useful for sharing information with others and letting them follow your thought process.

April 18, 2018