Voice of the DBA

Tag: goals

  • Learning more Kubernetes

    I’ve been slowly working my way through the 50 days of Kubernetes (K8s). As you might have guessed if you remember my first post, this has been more than 30 days. Life and work get in the way, but I’m working through the series of posts and videos.

    A couple of interesting things I’ve watched lately:

    Serverless Kubernetes and Serverless on Kubernetes – I worry about the idea of containers needing to spin up to meet serverless workloads, but maybe not. It’s a few ms for these containers to spin up, especially for small functional environments like Python or .NET. I suppose if there are issues, you can schedule more pods to be available or more replica sets to spread the load. This makes sense to me, and containers are often a perfect use for this on Kubernetes since you want to have lots of small items, but one large endpoint for people to hit things. I suspect this is how the FaaS implementations work on Azure and AWS.

    This video also talks about the need for a virtual kublet to allow the API to get things ready, without having underlying VMs, a prerequisite for scheduling items on the cloud. This is the serverless kubernetes concept. I’m not sure I completely understand this, but I get the idea here. We don’t have hardware provisioned, we’re running Kubernetes, and we want to push some of our load into the cloud. Since we don’t have a node assigned to us, we want to schedule on a virtual node that the cloud provider will actually spin up when we hit it.

    Fascinating and not something I’d have thought of before I saw this video.

    How the Kubernetes Scheduler Works –  This is interesting. Scheduling workloads and pods in different places is important. Flexibility while meeting demands matters. The idea of hard (required) and soft (optional or preferences) constraints for choosing where to run pods is fascinating. Another video by Brendon Burns. Most of us might care about these items since we may want some spreading assurances for instances or minimum resource requirements for our instances.

    For example, we might have a hard constraint that our pod (container) needs 128GB of RAM. This might limit the nodes that we want to run this pod on, and the scheduler takes this hard constraint into its decision making process.

    For soft constraints, we might prefer that a reporting instance not run on the same node as a OLTP instance, but if there are no nodes available, perhaps we’d live with this. That’s a soft constraint, and the scheduler tries to honor these, but it isn’t bound to prevent scheduling on those nodes.

  • Pausing the Learning for March

    In my last update, I was disappointed with my February progress. Busy with life and work, I didn’t do much reading or practicing. The latter is really the important part.

    I started March thinking I’d do a bit more, but I realize that won’t happen. I tried a little over the weekend, but with coaching responsibilities and family items, I just couldn’t really focus.

    March is a busy month for me outside of work. I have 3 major weekend volleyball tournaments with my daughter and my team, some travel for work, and still family commitments. In addition, I’ve started authoring a course with Packt, so my nights will be busy.

    As a result, I’m going to pause learning for March. Once I get past a few of these deadlines, I’ll go ahead and start reading again. For now, March will be heads down with SQL Server 2019.

  • Practical Web Scraping–More Reading

    As part of my learning goals for 2018, I wanted to work through various books. This is part of  my work with Python.

    It’s the end of the month, and I’m behind. I know it’s a short month, but it’s not that short. I had time to read the last couple week and I did some reading, but I didn’t get much practice.

    First the positive, then the excuses.

    Reading

    I did manage to read through the basic tutorial in Part I for how to use he BeautifulSoup module to read through an http page. This provided more ideas for finding sections of code and extracting out sections.

    I started Part II, which talks about more in depth http, with forms and posting. I’m still inside that one, and as of this point, I’m only 30% through the book.

    The plan for this weekend is to stop and practice a bit from BeautifulSoup and see what I can accomplish by reading a few web pages.

    Life Gets in the Way

    This month I had two long events on weekends that ate into my reading/working time, as well as a work project. I’m in the middle of volleyball season, and February brought me two 3 day tournaments to coach or watch. One was coaching a 13 year old team, and that occupied a lot of time. The evenings were spent with other coaches and I didn’t bring a laptop because I knew time would be limited. Add to that the need to get a few things done before and after the trip and I couldn’t practice web scraping.

    The second trip was for my daughter, and apart from watching her, my wife was in charge of team activities and I had to help. I had my laptop, but again, no time. I also chose to enjoy some time with my wife in the evening, which was worth it.

    Work was busy. We’re trying to migrate SQLServerCentral to a new platform and I’ve had to test things, as well as try and get other work done. As a result, I’ve been busy during the week and couldn’t even make time to incorporate any web scraping into my job. I also had to get other work done, so it’s been too busy to do much learning.

  • Practical Web Scraping–Getting Started

    As part of my learning goals for 2018, I wanted to work through various books. This is part of  my work with Python.

    After going through a few first chapters, I decided to start my February learning with Practice Web Scraping for Data Science, which looks at data acquisition using Python to pull data from the web. I found the book interesting and also this would be a nice setup for two of the other books (Power BI and natural language processing).

    Like many people, I find lots of data on the web, but I’m constantly struggling to get it into a database. I find myself going through gyrations at times to get data. Even with the cool features of Power BI, it hasn’t been as smooth as I’d like to get data, so I thought this would be a good book.

    Part 1

    The first few chapters of the book are about the basics of web scraping. We learn what this means, and get a little bit of a tutorial on who uses this technique, with some specific examples. We also get a basic python tutorial, which I skimmed. I know a bit about Python and this was a very basic, getting started.

    The next part of the early book deals with the basics of what http transport looks like and how some networking works. This is interesting to me, though not sure it matters for scraping. We’ll see. There is some discussion on GET and the http standard, so perhaps that’s helpful. It is good to at least know what codes might come back or what headers or parameters you need to use.

    The third chapter starts to get code working.  It opens with a discussion of HTML and how you can examine the structure of pages in your browser. This is a good reminder and basic tutorial of some of the web page developer tools that exist in your browser and that you might want to use when trying to build applications, especially those that scrape pages. There is also a basic CSS tutorial, which was good as I needed a little refresher. I rarely deal with CSS stuff, leaving that to others.

    The last part of the chapter starts with the BeautifulSoup library, which is built to parse out text, and specifically, makes working with HTML easier. The examples are with a Wikipedia Game of Thrones page, but I added some examples, trying to translate this to a sports page. It worked OK, and I learned a few things.

    The last part looks at Regular Expressions with BeautifulSoup and how you can search out elements and then start to copy data. It’s more complex and tedious, but then again, lots of programming is tedious. Once it’s working, it’s amazing.

    Experimenting

    I started to work with this in Azure Notebooks as a different way of tracking some work in Python. I’ll want to store things in a file at some point, but for now, this lets me start and stop learning and keep track of where I am without worrying about files and names.

    Not sure if anyone can access it (it’s marked public), but my project and notebooks are here: https://notebooks.azure.com/way0utwest/projects/web-scraping-with-python

    I ran some of the early scripts, which are just getting you used to working with Python and accessing web pages. I then copied some examples from my Calibre view of the book and executed them. I even tried to experiment a bit.

    One note: copying the code seems to leave some invalid character in there for Azure Notebooks, so I ended up editing the beginning of every line to remove the offensive character.

    This got me the basics of working with web scraping. Now to try and grab some data from another page and see what I’ve learned.