Voice of the DBA

Category: Editorial

  • The Rise of Vector Databases

    I had never heard of a vector database. I assumed this was a specialist type of database used for a particular problem domain, like a streaming database or graph database. There is a need for specialized platforms in certain situations, but I wasn’t sure what a vector was. The description I saw for a vector database was that they “… are specifically designed to work with the unique characteristics of vector embeddings. They index data in a way that makes it easy to search and retrieve objects according to their numerical values.”

    That sounds like any database. However, I saw a few more articles on the hype and then some details about the ways in which this type of database is helpful. Essentially, this is a database designed to store the outputs from various Artificial Intelligence (AI) and Machine Learning (ML) models that examine unstructured data. Things like images, video, audio, and even text are turned into numerical values, or vectors. The vector database is designed to help index and then search these vectors.

    What is interesting about the possibilities here is that the entire image, video, or whatever isn’t turned into a single numerical hash of some sort. Instead, the AI/ML process might identify that Steve Jones is in this video. That he is wearing a hat, or that he’s wearing a kilt. If I wanted to search for other videos of Steve Jones, or if this is the type of hat he’s wearing, a vector database can help. It’s much more powerful than simple tags that might be placed on a video because the details of the content are rendered into vectors which can be compared to other vectors. Not for exact matches, but likely ones.

    One interesting example in the second link above is that content could be “vectorized” to determine if an apple in the content refers to a fruit or the company that Steve Jobs and Steve Wozniak made famous. Not easy to do with a tag, but more possible with a vector database.

    And lots of data. Lots of vectors specifically, whose inventory is growing all the time. As more software is built to analyze unstructured data, and as organizations collect more unstructured data, the need to apply database techniques to this data becomes important.

    For those of us working with databases, I’d expect a lot of the mechanics of dealing with a database would still apply. Things like security, backups, and indexing will be needed with vector databases. We’ll get calls about slow performance, missing data, or strange results, and we’ll troubleshoot the system. How we do that specifically might vary, but those are just details we’ll work out.

    I like the idea of new databases, which provide more tools, challenges, and opportunities for us as data professionals. I haven’t met anyone using a vector database yet, but I’m looking forward to the day when that happens.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • The Complexity of Metrics

    Monitoring your SQL Server instances is important to ensure you can meet your SLAs. Availability, performance, reliability, quality, whatever you care about, it’s important that whoever is responsible is looking at how the database is performing. At Redgate, we have multiple teams working on SQL Monitor to enhance and grow it to meet your needs.

    A short while ago there was an internal conversation recently about page life expectancy. We’ve had some customers ask about this and setting alerts to watch this value. Our developers and sales engineers asked for a few thoughts from Grant and others on how to respond. There are a variety of opinions, some saying monitor it, some saying don’t bother.

    I think both pieces of advice have merit, which is to say that this isn’t a metric that you can look at in isolation. There is no value of PLE that is good or bad, or that says x is wrong or y is right. There is both a subtlety and a complexity to understanding what PLE is telling you about your system. If PLE is growing, you have to look deeper. If it’s falling, same thing. If it suddenly drops, there are multiple possible causes, and you need to examine other things. However, in many cases, this isn’t an actionable metric, but one that provides context about what might be happening in the database when combined with other values you monitor.

    This certainly isn’t a metric that you want to set an alert on because it can rise or fall and many times the change isn’t indicative of an acute problem.

    This is just one metric of many that are available in SQL Server, and knowing which ones to monitor is something good administrators learn. They know that very few values they instrument have a good or bad value, and often the rate of change needs to be combined with the actual reading to determine if there is a problem. We also often want to know if a high (or low) reading appears for an extended period of time. Having 100% CPU being used for 3 minutes likely isn’t an issue. If it lasts for 3 hours, I might feel differently.

    Metrics have more complexity than just having a range in which we ignore them and a limit at which we alert people. They are intended to be combined with each other, with observations by clients, and with the experience of looking at past observations over time. Our systems often develop patterns, and we don’t get too concerned about any values when the pattern repeats. It’s when something new happens and someone complains that we dig in to determine if there is a problem or the start of a new pattern.

    We definitely need monitoring of our database metrics, but we also need to understand why values move and the implications of them doing so. That’s something which isn’t as simple as setting alert for each one based on some value we think should never be exceeded.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Speakers Are Normal People

    The #SQLFamily is amazing, at least I think it is. Like many families, it’s welcoming, supportive, and comforting. It’s also maddening, frustrating, and exacerbating at times. Like most families, or at least the ones I know, it’s not perfect, but it’s what we have and at the end of the day, most of us get along with each other.

    It’s also an open group of people. In general, we welcome people with open arms and smiles. Those of us that are more visible or prominent are willing to listen to, help, and support anyone. I was overjoyed during the recent PASS Data Community Summit, where I had the chance to see so many people that I haven’t seen in person in 2-3 years. I met many other interesting people for the first time and enjoyed the experience.

    Not everyone feels the same way. I loved seeing Kimberly Tripp and Paul Randal for the first time in years and was honored to share the stage with them for the Community Keynote. I enjoyed the time we spent together, but afterwards Paul wrote about some people not feeling included or welcome.

    I understand that feeling. In many ways, much of adult life can mimic teenage years in high school with cliques and pettiness. I won’t pretend that doesn’t happen in the #SQLFamily, but I find it quite minimized, especially compared with many other communities of which I’ve been a part in the past. I haven’t seen the higher profile speakers and leaders in the SQL Community dismiss someone for asking a question or expressing an opinion. I have, however, seen that in other communities.

    It can be intimidating to walk up to someone that you don’t know who you might feel is famous or well known. It can be intimidating to just walk up to a group of people who are talking when they appear to know each other. I have that feeling at times even today, so I appreciate feeling like an outsider. In the Summit keynote, I talked about the thrill in meeting Kalen Delaney in 1999 and shaking her hand. I was nervous and intimidated to ask her a question after her presentation. At the time I hadn’t delivered a talk in front of anyone outside of school environments and was a fairly introverted geek. It was hard to step up and make that effort, but I’m glad I did.

    As Paul writes, anyone is welcome in the #SQLFamily. Anyone can join. You don’t have to come shake a hand or say hi, but I’m happy when you do. Will we be best friends right away? Probably not. Will we go out to dinner that night? Maybe. I’ve certainly met attendees at events and then had dinner with them. I know plenty of other speakers who have as well.

    Many of the people who speak, organize, and write/blog/tweet/etc. in our industry are friends. We do value time with each other, and that can feel like a club, but it’s not. We enjoy seeing each other and want to catch up, like any group of friends. However, we are also welcoming of newcomers, so feel free to introduce yourself.

    Ultimately those of us who engage in these highly visible, extroverted acts are often just like the rest of you. We’re a mix of people that are mostly introverted, with a few extroverts thrown in. Some speakers are very smart and talented, some are more like me: we know enough to get the things done that we’re asked to do.  Some of us love to go out and sing karaoke until all hours of the night and others prefer a small dinner or a little tabletop gaming in a quiet environment.

    My encouragement to get people to meet others, network, set up events or meetups, and more isn’t to try and convince any of you to join the cool kids club. It’s not to get you to change who you are.

    It’s to help you find your tribe. To find your kind of people.

    I’d love to greet all of you with a hug at events, call you by name, and go out to dinner with you. I can’t because there isn’t enough time in the day. And quite frankly, I really, really value my alone time. What I want more than anything is for you to be successful, find a great job or career you relish, and for you to enjoy spending time with those you enjoy. Whoever and wherever that is. That takes some effort, but it’s worth the energy involved.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Losing Track of Data

    I saw this article a few months ago, which talks about engineers at Facebook not knowing where their customers’ personal data is stored. The engineers were being questioned in a legal matter, where they were asked to definitively state where all personal PII data for any human was stored by Facebook. Their answer was that they didn’t think anyone in the company would be able to answer that question.

    Facebook has been controversial over the years, and plenty of people dislike the way the company conducts business. I noticed no shortage of data people (and many others) commenting on this situation, saying that Facebook should be shut down because they don’t know where data is being stored.

    However, I don’t agree. In working with lots of customers, on all aspects of how they handle, process, and manage data, I expect this to be a problem in many organizations. Whether large or small, whether they have few or many software engineers, it is highly possible that there isn’t a good list of where personal data is being stored. As we work with customers to classify data with SQL Data Catalog, that process takes a long time, and very often the system administrators or developers who undertake take the task are unaware of all the places where data is stored.

    That’s just in relational databases, ignoring all the Excel spreadsheets, text exports, mail merge operations, and uploads to services for mailing, analysis, or something else. Very often the control of personal data is fragmented among groups, with there being few efforts made to coherently manage a customer’s data.

    The world has adopted computing at an incredibly fast pace, often by people with little knowledge or forethought of the implications of gathering and processing data. In many cases, probably most cases, there is no overriding strategy. Just like with applications slapped together quickly, we find data being gathered and stored based on the requirements and demands of business people, with no planning for management or archival, and often not even with any security requirements.

    I liked the GDPR as a step forward, asking companies to not only handle data appropriately, but remove it when not needed, not use it without consent, and to be able to keep track and delete it if not necessary. I don’t know that this has been successful, but it has changed handling practices in some organizations. At least in responsible organizations, and many of them have had to track down personal data to delete it. I’m not sure they know where it all is, but I at least assume they know where all of the data about a person is in their various relational stores.

    As a technical person, do you know where all data is stored about a customer? Are you sure you know where marketing has been keeping information and what other mailing, analysis, reporting, CRM, etc. systems they’ve put data? Any idea how many copies the operations group keeps? Test systems, QA, UAT, and others? What about test data sets, are they sanitized? Perhaps legal or finance has gotten extracts of data to reconcile their systems.

    Tracking down all data can be hard, and I’m not surprised Facebook struggles. I would guess engineers in many organizations would have similar answers.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.