Voice of the DBA

Tag: data

  • Protecting Data Between Services

    I saw an announcement this week that Microsoft is shutting down their HealthVault service, where someone might have chosen to store their medical records in a way that it could be shared with health professionals, but an individual could maintain control of the records. They could share them out to friends, download them, etc. It was a good idea, and I was interested in it for the future. Since I have been fortunate to mostly avoid doctors and hospitals, I never signed up, but I still could sign up as of this week.

    Google used to have a similar product, but shut it down a few years ago. I suspect that this commercial space just isn’t as lucrative and valuable as either company thought, as I don’t really see any competitors out there that might allow a user to transfer their records. While I don’t know that there needs to be a service for this, I did think of the hassles and potential issues that might exist if there were. Imagine your mother or grandmother keeping their health records here for the last decade and now needing to download them and manage them before they could be used again. What if someone downloaded these to a PC and had a hard drive failure?

    Backups are needed, we know that, and I’d hope most consumers know that, but in a rush or in the worry that these need to be copied, but securely, would you want to keep this data on a tablet or PC? Or would you want an encrypted drive. If that’s the case, would you want to ensure you have 2 (or more) of them? Easy to plan this out, hard to think about if you get a notice about the service shutting down. Imagine that your spam filter knocks this down or you’re inundated and miss it and get a final notice on Sept 1 or Oct 1? A real hassle.

    The higher level view of this in my eyes is that we need better data formats for capturing and keeping lots of our data in systems. Those of us that are impacted by the GDPR (or similar laws) might be thinking about this already, as we have a need to provide data in response to requests. Providing a report, and packaging this up, is no small task. I assume I’d use a .zip file, but maybe a .nuget with a manifest is actually a better idea for customers.

    As the use of services grows, I expect that we will want to get more portability for data over time. Certainly vendors that provide services have an incentive for tooling. WordPress has import/export for other blog services, Microsoft will help you move data from Oracle (as will Oracle in the other direction), and there are specialized vendors in niche applications doing the same, but really I’d like to see us have more open, and extensible, data formats that relate to the types of data in our lives. I know we’ll have more disparate types of data, in various formats, so why not an easy way for each of us to store text, images, and more as a service that contains some metadata, some indexing. Something like a personal data lake of sorts.

    We could have other services, like image services, visualization services for numbers, and more be authorized (or de-authorized) for our data. Imagine a way for us to allow a company to hold our data, but we disperse that to other vendors as needed for services. I could easily imagine various “storage” vendors competing and allowing us to “port” our data to a new service as easily as we port phone numbers for mobile phones. Perhaps a whole new era of data storage and management is coming.

    Or maybe we’ll just stick with the current separated, proprietary, limited view of data that we manage on hard drives and USB sticks. It’s more likely, but much less preferable in my mind.

    Steve Jones

    Listen to the podcast.

  • Republish – Normal Data

    Busy today at the InfoSec conference, so republishing Normal Data.

  • Fitting Into RAM

    RAM has always been a fairly limited resource in most of the computer systems I’ve worked with in my career. Often there is never enough RAM, and I’d always like more, often to speed up the systems. That has somewhat changed with laptops, as 16 GB really works well for me most of the time. Not that I wouldn’t take a 32GB machine, but I’m waiting for them to become more common and smaller.

    This has especially been true for database servers. It seems that I’ve rarely had a database server that could fit my entire database in RAM. Even now, I have an over-provisioned server for SQLServerCentral which has plenty of spare capacity, but I’m still slightly short on RAM. The target level for SQL Server is about one GB more than I have set. Not really worth complaining about, but still I don’t have the RAM I’d like.

    Last week I wrote about someone that attacked the RDBMS as old and troublesome technology. As a part of this, a method of storing all data in memory was presented. I’m not sure I think this is actually a good or practical idea for most systems, but I did wonder about the idea of data space and size. Certainly I have seen plenty of index space in databases, and certainly there is more index data than other data at times, but I suspect that’s not the case for many databases.

    Regardless, I was curious if anyone has large databases that couldn’t fit into RAM these days. If you think about the largest database you have, how big is it, in terms of data size. Not allocated size, but the total data space used. Would this fit into RAM if you could get 1TB or 2TB of memory? If you can, what about index sizes, are they large? There are a few scripts in this thread if you need one.

    I suspect there are certainly databases that don’t fit into RAM, and likely plenty of instances with more than 1 database that don’t have enough RAM. I still see plenty of people with less than 64GB on their servers, so that’s a battle still being fought. I certainly wouldn’t advocate an in-memory only database, likely because there are going to be other issues, but it’s still an interesting thought. Certainly my server has only 60GB allocated and the databases are well over that in aggregate.

    Maybe asking for a bit more RAM on those critical servers is the way to go, especially if you think you can get the entire database into memory.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.5MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • We Don’t Have Perfect Information

    I was discussing the PASS Summit with someone and they were wondering about building their schedule. Actually, they wanted to pick sessions, but see the choices in a calendar format, but the schedule wasn’t out at that point in time. My suggestion was to just build the schedule and then sort out conflicts later.

    A few people have mentioned over the years that they want to build a schedule and be ready for the event to maximize their experience and be efficient. I think that’s a common, normal thing that many technical people like to do. Many of us are Type-A, and we like knowing our plans and having a schedule prepared in advance. We can then decide which sessions might have greater priority for us during a time slow and adjust our choices to have the best experience. After all, we may want to see two speakers, but if they present at the same time, we have to make a choice.

    The problem is that we don’t have perfect information. Even if the descriptions and abstracts included perfect information about the agendas, what is covered, and to what depth, including demos, we’d still not necessarily assimilate and recognize all that data in a way that makes sense, even to us. There are no shortage of people that make plans today and are unhappy a few weeks later.

    Even if we knew what we wanted to watch, many of us might think a session on database design has to cover third normal form, even when the text said this examines PKs and FKs. We might assume an SSIS data load talk included something on CSVs when the presenter described the talk as being for ragged right text files.

    We’re human, and that means we have flaws in how we deal with the world. This includes the ways in which we model and analyze data. We can make mistakes in our analysis often when we simplify our view of a problem to the point where our analysis is inherently flawed. If we don’t account for this and assume we’re flawed, we may overweight our conclusions.

    I try to remember this when I write reports from systems that others will use. I won’t have every piece of information that might affect a decision, but I try to ensure I have the most important, or significant, data. Or at least, the data I and the users feel is significant. The important thing to remember is that out data is always incomplete, and it’s entirely possible that we have missed a valuable piece of data.

    When that happens, we have to adapt and adjust our report, our application, or our conference schedule. We’ll learn more across time and we can use that information to change our system. I know that my view of a conference like the PASS Summit today, or even a week before the event, will be different than how I feel at the event. I should have a plan, but be willing to flex as circumstances change. And, always have a backup. I like to pick two or three sessions for every time slot, just in case.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 5.3MB) podcast or subscribe to the feed at iTunes and Libsyn.