Voice of the DBA

Tag: big data

  • Big Data or Small Data

    I went to San Francisco for Small Data SF, a conference sponsored by Mother Duck. The premise of the event was that smaller sets of data are both very useful and prevalent. The manifesto speaks to me, as I am a big fan of smaller sets of data for sure. I also think that most of the time we can use less data than we think we need, especially when it’s recent data. That often is more relevant and we end up with contorted queries that try to weight new or old data differently to reflect this. Maybe the best line for me is this one:

    Bigger data has an opportunity cost: Time.

    I think time is a very valuable commodity and large sets of data can slow you down. There’s also the chance that looking at too much data starts to blur the lines of understanding. We may start to miss information in our dataset, or we may find people arguing about different things the data means, because we have so much data that we can find support for any position somewhere in the vast sea of numbers, strings, and dates.

    Big data also has a real cost in resources, often money. One of the examples was from the organizer, who once gave a demo on stage, querying a PB of data.  That’s impressive, and lots of us would want to be able to query our very-large-but-less-than-PB-sized data in minutes. However, the thing that wasn’t disclosed in the demo was the query cost over USD$5k.

    I’ve heard from a number of customers and speakers that most people don’t have big data. Most of us have 100s-of-GB-sized working sets of data, sometimes with TB-sized archives in the same database that slow everything down. If we could easily extract out the useful data, we could query those hundreds of GB more efficiently.

    This is especially true in the era of small devices that can handle something close to a TB of data in a small form factor. With some of the columnar systems that compress data, a TB of raw data might be substantially compressed in Parquet files or an analysis system like DuckDB. In that case, we might realistically search and analyze 1TB of data on a laptop.

    I know that big data is relative, but many of us face challenges with data sizes and query performance. I know lots of you embrace the challenge and see working with TB (or larger) systems as a badge of honor. I also know the reality is that most of us struggle to separate our archive data from current working data in our systems. However, if we could, would most of you want to work with smaller data sets or do you enjoy large ones? I know which way I lean.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Big Queries, Big Money

    I had been meaning to post this, so as I finished a piece that referenced this, I decided to post the picture. This was from Small Data SF, where the opening keynote referenced the Google BigQuery demo of a 1PB database.

    Here was the slide shown later in the talk.

    2024-12_0187

    The thing the demo didn’t explain was that query cost $5,580. The conclusion, big data is just too expensive to query often.

  • Big Data Downsides

    Companies often want more data to help them make decisions on how they run their business. There has been this quest to gather and analyze as much data as possible to increase the efficiency of their operations to help reduce costs or increase profits. This has led to the importance of data as an asset, and the need for more data professionals in many organizations.

    That’s good for many of us that work with data.

    However, using data to try and improve your efficiency has a downside. It can lead you to a very narrow focus in your approach. That can be good in narrow, well-defined areas, such as minimizing the distance driven or packing containers. For less focused tasks, such as telling a story or writing code, this can mean you get stuck in a rut and limit your opportunities to improve.

    There’s an interesting article about big data and Hollywood, specifically looking at the types of products produced. Big data analysis leads companies to aim for the most effective types of movies that make money. Good for a company, not so good for society. Arguably, not even good for a company over time as people will tire of the same story, or type of story over time. Eventually, making simple decisions based on past data will start to fail.

    I can see the same thing in other industries as well. Using Big Data to drive decisions can help, but many of the areas where we use these techniques will evolve and change over time. The way we solve problems with code change over time as we develop new tools, techniques, platforms, languages, etc. There isn’t a perfect way to design a database or write a CRUD app precisely because new capabilities or new possibilities emerge. You could say the same things about marketing, manufacturing, medicine, and many other endeavors.

    This isn’t to imply big data and complex analysis isn’t helpful or useful. It’s just not everything. We need to balance human input, with some creativity, some instinct, some diverse thought, and some guessing. Most importantly, we ought to experiment and learn, not only from what machines might extrapolate, but from how humans change their thinking over time.

    Find a balance, accepting some imperfection in your process and in the world at large. Hopefully that will lead you to some success.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • The Danger of Sharing Data

    In the past, many businesses hired employees whose role was deciding which prices to charge for their goods or services. At one point, organizations largely set prices based on their costs, though over time they tend to look at their competitors and set similar prices. If, however, management from multiple companies went into a room and determined prices, this would be price-fixing.

    Price fixing is illegal. In many countries, we would not allow different companies to work together in a way that might reduce competition or take advantage of consumers. However, that might be a struggle in the future as we find companies using various services to help them manage their systems.

    In this case, a few different Las Vegas hotels used the same company to help them decide how to price their rooms. Rainmaker is a revenue management platform, which uses lots of data to help hotels price their rooms in a way that maximizes their revenue. That sounds great, but if this company is successful and many of their clients are in the same location, this is really a way of sharing data by proxy. The hotels are being sued because of their use of this platform.

    This one of the problems (or advantages) of lots of data. It allows information to be drawn out of data that wouldn’t otherwise be obvious. Certainly, lots of companies look at their competitors and make decisions based on what they see. I’m certain there are lots of people inside airlines constantly checking the prices of their competition. However, they are gathering this data independently and making their own decisions. If Rainmaker were used by American, United, and Delta to set prices for flights, I imagine many would see this as an anti-trust violation.

    Big data is powerful. It can help give an organization an advantage over its competition. This is one reason lots of companies hire data professionals like us; they see data as a very valuable asset. However, in this case, I feel that one company selling this data, or rather the conclusions, to competitors is a problem.

    I expect more problems like this in the future as smart people look to harness the power of data and sell their services to competitive companies in many industries.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.