Seriously Real Time Data Processing

There are many of us that work with systems where data is processed in real-time and then used to make decisions. These might be humans viewing reports and then taking action, or some automated system that might react based on a value changing. In many cases, however, the amount of data, timing in which to react with a decision, and the implications for mistakes aren’t that critical. We have some leeway for the processing not being perfect.

For vehicles traveling at 180mph, the tolerance for mistakes is low, with the chance of a catastrophic crash looming constantly. That is what is happening with a competition at the Indianapolis Motor Speedway, where university students are competing to develop race cars that can move around that track at these speeds. This would be quite a challenge for vehicles moving around the track by themselves, but in this case, it’s a race with multiple cars.

This might seem silly, but it’s a step towards understanding just how much data needs to be processed and how the results can deal with the chaos in the real world. Each car is independent, so has to react to the other vehicles and make decisions on how to adjust its own operation in real-time, with a sub-second response to prevent accidents. This isn’t different from the decisions human drivers have to make in a race, and there are plenty of mistakes that result in crashes. However, humans can think in new situations and react. They don’t need to have every possible response programmed in.

These AI-driven race cars will be similar, but how well they perform remains to be seen. This is the type of test environment that will help us move forward in using technology and AI models in less constrained environments, like a public highway. Lots of technology was tried on race tracks before it became available to consumers, and I think this will make its way to retail cars as well at some point.

There are already some companies trying to build this into cars. Tesla famously has Full Self Driving, although this has been in beta for a long time, and its results are less than stellar in some cases. Waymo has been working on the problem, and I actually had the chance to ride in a self-driving Uber in Las Vegas, though that experience was less than thrilling. The driver had to take control a number of times, so this wasn’t quite self-driving.

There is a lot of work still to be done here, and I don’t know how quickly this will become safe enough for general driving. I suspect that we’ll see this in very limited areas first, like a zone in a city that only allows these types of cars, or maybe specific highways, like HOV lanes. Somewhere the problem domain is simpler, with less decisions that need to be made.

There are already lots of safety features in modern cars that help prevent mistakes and accidents, but most of these are simple systems that aren’t making decisions in many ways. Moving to more complex driving operations will require some heavy data gathering, processing, and analysis, something that should be of interest to data professionals. This is a problem domain that will be fascinating to watch in the next few years.

Steve Jones

Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

Posted in Editorial | Tagged | Comments Off on Seriously Real Time Data Processing

Daily Coping 19 Jul 2021

I started to add a daily coping tip to the SQLServerCentral newsletter and to the Community Circle, which is helping me deal with the issues in the world. I’m adding my responses for each day here. All my coping tips are under this tag. 

Today’s tip is to start the week with some movement. Exercise in some way today.

An easy coping tip for me, as this is something I regularly do. I’m writing this a few days early, but in this case, I’ve scheduled a yoga class for Monday afternoon. Now that it’s on my schedule, I’ll get up early, get to work and be ready for a break after.

Posted in Blog | Tagged , , | Comments Off on Daily Coping 19 Jul 2021

Service for Brian Moran

If you are in the Reston, VA area and would like to attend, the service information is here: https://www.eventbrite.com/e/brian-morans-celebration-of-life-tickets-163260712185?fbclid=IwAR3uwlJgfqXJrQf998pyGaZtyDupT_Vi6mwQIqaVufV9cJkNMfi-6LmrA7w

Also, if someone going can livestream, there are quite a few of us that would appreciate you doing so.

Posted in Blog | Tagged , | Comments Off on Service for Brian Moran

Messy Job Descriptions

I saw a job description recently for a DBA that asked for SQL Server experience, but also “other RDBMSes, like Cassandra and PostgreSQL”. Not sure Cassandra fits there, or why this says like. I’ve seen some other ads that ask for C# or Python. Some asking for MDX/DAX knowledge along with AWS Cloud Formation and programming APIs. Some have a required and optional or “nice to have” sections, but many include a laundry list of technologies and skills. For software developer roles, the list of skills often exceeds what I think any person in the world might know.

There was a debate about this at SQL Server Central in one of the threads. It seems some people are split on whether this is a problem or a good thing. Quite a few people noted that they wouldn’t apply when there are so many items listed that they don’t have experience in. For others, we wouldn’t hesitate if we had around 50% of the items listed. I’m in the latter category, as I’ve had plenty of friends, and myself, get jobs that might have seemed out of reach based on the description and our skills.

In my experience, often a job description is put together on the fly and in a hurry, usually by someone that isn’t familiar with the job. They ask others what to include, and we end up with a large list of things that would be nice, but not necessary. The end result isn’t always what the job entails, at least not completely. Often I’ve found as a developer or operations person I might end up lightly touching parts of different roles, but not regularly and not too deeply.

I don’t know that I think it’s worth effort to tightly define a job for a new hire, as the job requirements can change, and we might adapt a job to the individual. Not completely, but if someone knows more about reporting or BI than HA/DR, we might have them tackle more of that work and only partially work on clusters or AGs. Others might fill in with more HA/DR and less BI work. The reverse also could be true, so should we have a job description that is narrowly defined to DBA work with an HADR focus? Or one for BI? I don’t know, but I learn towards not tightly defining a job description.

Hiring is a difficult enough process, especially today, without too tightly defining the roles. I do think it’s worth spending time with the team doing the work to list out the necessary skills (and levels) needed to help them, but adding in other items is useful. It casts a wider net, and it helps you as the hiring group, think about what tradeoffs you might make. If someone is weak with replication, and we use it, but they have some strong Azure skills, maybe we accept that. We know we’ll need to train this person on replication, but they might help us better understand the cloud. Perhaps that’s a better choice than someone highly skilled with replication but without a lot of other experience.

Or maybe not.

Ultimately, on the hiring side, include what you want. It doesn’t have to be perfect. If you don’t get candidates, then re-examine it, but list what you really need, and separately, what you want.

On the candidate side, don’t be intimidated. If you get to 45% of the skills, apply. If you hit 75% of the requirements, apply. Maybe even if you’re a little short on those but you have other skills. Use what you know, and what’s been listed, to help you drive the interview. Emphasize your strengths, and convince them you can learn. Ask questions, use “I don’t know, but” often with examples of how you might gain the knowledge or get help. That often is more important than what you have done in the past.

Steve Jones

Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

Posted in Editorial | Tagged | Comments Off on Messy Job Descriptions