Companies often want more data to help them make decisions on how they run their business. There has been this quest to gather and analyze as much data as possible to increase the efficiency of their operations to help reduce costs or increase profits. This has led to the importance of data as an asset, and the need for more data professionals in many organizations.
That’s good for many of us that work with data.
However, using data to try and improve your efficiency has a downside. It can lead you to a very narrow focus in your approach. That can be good in narrow, well-defined areas, such as minimizing the distance driven or packing containers. For less focused tasks, such as telling a story or writing code, this can mean you get stuck in a rut and limit your opportunities to improve.
There’s an interesting article about big data and Hollywood, specifically looking at the types of products produced. Big data analysis leads companies to aim for the most effective types of movies that make money. Good for a company, not so good for society. Arguably, not even good for a company over time as people will tire of the same story, or type of story over time. Eventually, making simple decisions based on past data will start to fail.
I can see the same thing in other industries as well. Using Big Data to drive decisions can help, but many of the areas where we use these techniques will evolve and change over time. The way we solve problems with code change over time as we develop new tools, techniques, platforms, languages, etc. There isn’t a perfect way to design a database or write a CRUD app precisely because new capabilities or new possibilities emerge. You could say the same things about marketing, manufacturing, medicine, and many other endeavors.
This isn’t to imply big data and complex analysis isn’t helpful or useful. It’s just not everything. We need to balance human input, with some creativity, some instinct, some diverse thought, and some guessing. Most importantly, we ought to experiment and learn, not only from what machines might extrapolate, but from how humans change their thinking over time.
Find a balance, accepting some imperfection in your process and in the world at large. Hopefully that will lead you to some success.
Steve Jones
Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.
Voice of the DBA 
I have to disagree with the statement that ” people will tire of the same story, or type of story over time”. Storytelling is a passion/hobby for me, something I share with my youngest daughter; something I never expected I would be able to do since males and females tend to have very different interest when it comes to fiction.
All stories are built upon 1 of 7 story archetypes; Rags to riches, Rebirth, The Quest (aka the heroes’ journey) , Tragedy, Voyage & Return Home, Comedy and Overcoming the beast/monster. All works of fiction can be tied back to 1 of these 7 archetypes. What people get tired of is when the exact same outline is duplicated where only the names and places change and even then it may take a few in a row to do that if it’s good writing and directing/acting. People also are more easily exhausted by bad writing which has become a problem in Hollywood the last 10 years with the last few being the worst. The 4th film in THE MATRIX series should have been an easy win but it was filled with bad, juvenile like writing. I’d say the writing is the most important piece of any story. If you have excellent writing but an OK director and actors it can still do well. If you have the best Director in the biz with the most wanted actors in it but the writing is bad it’s gonna flop.
LikeLike
Fair points. There are only a few stories, but the way they are told does change. However, it also takes place across time, and as you noted, writing is key.
I do think that aiming to take shortcuts and only find the easy type of repeat story doesn’t work well, but again, maybe it is down to the writing and acting.
I definitely appreciate writing more as I get older. As I rewatch some series, it’s obvious when writers change, or they are struggling. I’m amazed looking back how often successful series can struggle in the first 1-2 episodes of a new year as they tackle some new storyline.
LikeLike