Voice of the DBA

Author: way0utwest

  • Finding Legal Data

    Many of us have likely been asked about data science or machine learning in the last few years by someone in our organization. This has become a hot field, with many companies looking to try and find ways to use the technology to improve their work in some way. While I don’t know how successful these projects have been in organizations, I know that some areas seem to be finding practical applications. Image recognition, translation, and even some application scoring systems have benefited from machine learning algorithms.

    To build a successful model that works well, we need training data. Often lots of training data, and then have some metadata about how our training data might be applicable to a particular question. For example, if we have lots of pictures of dogs, we might need to tag the different breeds in order for a system to differentiate among them. If we want loan applications scored, we should have a corpus of documents that are already scored. This metadata allows the system to learn.

    There is a company, Clearview AI, that markets itself as a facial recognition system. To build their model, they scraped images from YouTube and other Internet sources, without consent from Google or the subjects of the videos. This is interesting, as the data itself is publicly available, but gathering it into a database and using it for other purposes might run afoul of data privacy laws, like the GDPR.

    I don’t quite know how to feel about this use of public data. While I don’t mind people viewing my pictures, I’m not sure that I like the idea of them being copied into a database for some other purpose. That might seem silly, or even strange, but I do think there is power in data and more power in more data. Allowing others to put my images in a database and use them, perhaps to train a model to recognize me, feels like overreach.

    If you need data for your company, or your idea, what can you do? Many people just scrape Google, Facebook, etc., and get data. That might cause you some legal issues in some places, and you ought to be aware of the implications if you choose to do this, or you are asked to do this. I don’t think this is how we want data to be gathered. I know there are some guidelines for responsible AI at Microsoft, but not necessarily rules in place for many companies. Hopefully that will change over time.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Daily Coping 19 Apr 2021

    I started to add a daily coping tip to the SQLServerCentral newsletter and to the Community Circle, which is helping me deal with the issues in the world. I’m adding my responses for each day here. All my coping tips are under this tag. 

    Today’s tip is to make sleep a priority and go to bed on time, turning off devices well before you turn in.

    I don’t stay up late. I tend to try and relax at night with a book or a little TV with my wife. As I get tired, I’ll usually take off my glasses and then try to go to sleep rather than try to stay awake.

    I had my first COVID shot recently, and I had a rough day afterwards. Between that, the shingles vaccine and some busy days, I’m trying to stay a bit more relaxed and take care of myself. While I am still trying to enjoy a busy life, I am also trying to not push too hard.

    This week, I’ve been going to bed a little earlier, and trying to sleep well. I may watch a little TV with my wife, but I’m trying to keep it light and mindless, nothing that’s intense or excites my mind. Usually some silly comedy I can laugh at, and if I miss a bit of dialog, I don’t care because the story doesn’t really matter.

  • 2020 Advent of Code – Day 4

    This series looks at the Advent of Code challenges.

    As one of my goals, I’m working through challenges. This post looks at day 4.

    Part 1

    This is an interesting data set. It’s ugly, inconsistent, and spans across lines. In fact, to determine what any particular “row” is, you need to read line by line and process the data until you find a blank line. An SSIS or ADF exercise indeed.

    However, I decided to try this with Python first. I thought the loading and splitting of items on consecutive lines would be easier. I opened the file and then scanned it for values like this:

    for row in passports:
    if row not in ['\n','\r\n']:
            currpassport += row.replace('\n',' ')

    This let me look for a blank row. If I didn’t find one, I added the row to my current passport value. If I did have a blank value, I split the row into a dictionary:

    currdict = dict(x.split(":") for x in currpassport.split(" ") if x)

    From here, I could check the length being either 8 entries, or 7 entries if the cid was not present. I counted all these up to get to the answer.

    Part II

    This was annoying.  Validating each one of the entries based on years or a set of values. I knew I could build a number of validation functions, which is the better way. I ended up just using a series of IF statements to check values and set a validation variable. A sample of them is here: 

    if currdict["hgt"][-2:]=="cm" and ( int(currdict["hgt"][:-2]) < 150 or int(currdict["hgt"][:-2]) > 193):
                    valid = 0
    if currdict["hgt"][-2:]=="in" and ( int(currdict["hgt"][:-2]) < 59 or int(currdict["hgt"][:-2]) > 76):
                    valid = 0
    if currdict["ecl"] not in ["amb", "blu", "brn", "gry", "grn", "hzl", "oth"]:
                    valid = 0
    if currdict["hcl"][0] != "#" or len(currdict["hcl"]) != 7:
                    valid = 0

    This let me tally up the valid passports.

    I still need to work on these in PoSh and SQL, but life has gotten in the way of things outside of work.

  • Finding a Break

    Microsoft was one of the companies that dramatically pivoted to remote work. They enhanced their Teams product in a number of ways to help themselves, and their customers collaborate when people are not located in the same physical space. They also started to study the way in which people worked. I wrote about this last year, and they continue to study how work has changed, even though they are allowing workers back into offices.

    There is a new article that talks about the differences between workers and managers. Managers think they are thriving, and workers are not. There may be many reasons for this, but I suspect one of these is the privilege of space. Many managers and senior workers likely have bigger residences, and can spread out. They can change their locations and experience a new environment. Since so many of them might get by with less powerful laptops, and often work on a single item at a time, it’s easy to move from the kitchen to a dedicated office, to a bedroom, or even out on a deck.

    As a technical person, I’ve often felt cramped and crippled when I don’t have two monitors where I can see a lot of code and perhaps have a document or other information on another screen. Even when working with VSCode, I like having a separate terminal or place to test code, as opposed to ALT+TABing between windows. It’s not easy to move a couple of screens around my house.

    After a year of remote work, I still find many co-workers in cramped space situations. Working in a bedroom, a living room, or something else. Most of us haven’t built houses with the idea that we’d be in them working and living full time. With less places to go, I’m sure many in apartments, shared spaces, or even in houses with family feel cramped, stagnant, and even incarcerated.

    I also think that for many workers, there is unseen, and maybe implied, pressure to get more done. While some of this has existed in office spaces, I think the lack of contact with a manager might cause many workers to feel they have to prove they are really getting lots of work done.

    Some of the ideas from Microsoft, all of which appear to be connected to digital services, seem to miss the things most of us need. We need contact with other humans, a break from the digital world, and time in the analog world. We need a way to get away from devices, screens, and digital touch.

    I’m trying to do better myself, and I am encouraging those I know to find ways to take more breaks outside, in other rooms, or even a drive in a car. Now if we could find a way to ensure more management would support this, I think that we might get better engaged and creative workers.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.