Voice of the DBA

Tag: big data

  • Big Data Problems

    Nate Silver is a “data guy.” He has made a career of analyzing data, and doing it well. His site, FiveThirtyEight, has become very popular in the last few years after some of their political predictions were surprisingly accurate. Now the site publishes analyses of sports, politics, economics, and more.

    Mr. Silver works with a lot of data, as does the rest of his company, and arguably, he’s one of the people that understands just what “big data” is and isn’t. We could argue about what the term means, but no matter what “big” is to you, Mr. Silver has some thoughts about the challenges we all face with larger data sets.

    In an interview, he noted that storage and access are always issues. Many of are us know this because we are charged with managing data. However he also talks about bias, false positives, and complexity being problems with an analysis that isn’t performed well. It’s a short piece, and I wish there were more depth since many of these topics could be the subject of a book all by themselves.

    While many of us aren’t in charge of performing the analysis, we do often work with those that do. We assist others in writing the queries or assembling the data. Our skill in helping perform analysis, understanding the meaning of data stored in our systems, and learning how to separate the signal from the noise will be talents that your employer will appreciate.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.0MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Problems with Big Data

    Big Data is constantly in the news. We’ve been asked at SQLserverCentral to try and develop some articles, perhaps even a stairway to explain what Big Data is and how we might use it. I’m still trying to grasp the concepts myself, and unlike the amorphous cloud, I’m still looking for some good examples of what Big Data really is.

    When I ran across this piece warning that Big Data isn’t the final solution to all our questions in the world, I wasn’t surprised. The piece notes that Google Flu hasn’t been very accurate in its predictions of outbreaks. At first glance, this gives lots of credence to the idea that the good, solid data analysis and mining techniques we’ve used for years are just as good as any new Big Data fad.

    However as I read more about the piece, it’s not that big data and the analysis of large quantities of information is flawed, it’s that a solid hypothesis matters. Researchers need to be willing to evolve their algorithms as they learn more about a problem. Probably they should also assume their algorithms are not correct until they’ve proven their ability to predict actual trends for some period of time.

    We’ll constantly be searching for ways to better interpret information and make better decisions. No new technology or product is going to magically solve our problems. Good solid understanding of the problem domain will continue to matter as much as the data itself.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 1.8MB) podcast or subscribe to the feed at iTunes and Mevio . feed

    The Voice of the DBA podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.

  • Big Data Comes to Hiring

    I remember first seeing kiosks a few years back in some stores such as Best Buy, Target, Barnes and Nobles, etc. The devices had banners above them that invited people to apply for a job using the computerized system. This was just before my oldest son started applying for jobs, and shortly thereafter I found a few articles that stated quite a few of these large corporations were using the same company to manage the kiosks and pre-screen candidates. Not only were they considering the applications, but they were examining employment records, noting which candidates might have been terminated or left another job quickly. They were trying to use analytics to decide which applicants to interview.

    That entire process reminded me of the “black mark” on your record that I heard about as a kid. It was a reminder to not make too many mistakes, at least not serious ones, that might follow you forever. It appears now that it’s not just entry level jobs that are starting to use data and analysis for hiring decisions. It seems as though some companies are starting to evaluate candidates with different types of algorithms and applications for a wide range of positions. This piece from the Atlantic talks about the history of hiring and a few ways in which data is being gathered about workers and used to analyze their performance. Even more interesting the section about halfway down that talks about how they analyze developer’s work from afar.

    It’s expensive, time consuming, and stressful to hire people. We’ve proven in technology over and over that many of the hiring decisions made, aren’t made that well. We find talented people being underpaid and quick to leave, while other employees are overpaid and are happy to toil away in their positions for years. Neither situation is good for the employer, and often not good for the employee either. It’s less than motivating for any of us when we don’t find some challenge and enjoyment in our jobs.

    Ultimately I know more “people analytics” will be used in the hiring process by some companies, but I find that even with the data, there’s one thing that seems to matter more: personal referrals. Networking matters, and there is no shortage of companies that are placing lots of value on their employees’ networks. Some of them are even looking to conduct the majority of their hires through networking. So make sure you’re networking with others in technology. You never know when it will be the way you get your next job.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.8MB) podcast or subscribe to the feed at iTunes and Mevio . feed

    The Voice of the DBA podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.

  • Is Big Data Good for Data Professionals?

    Big Data isn't well defined, but it makes a lot of people sit up and pay attention.
    Big Data isn’t well defined, but it makes a lot of people sit up and pay attention.

    There’s a lot of hype around “big data”, a term being thrown around so much in the media that I’m not really sure what it means anymore. Is 1TB “big data”? Is 10,000 transactions/sec big data? Or does it mean that you have more data than your systems can handle, causing queries and reports to run slow?

    I almost hope it’s the latter. I hope that our managers start to think that when our systems run slower that we’re dealing with big data, and we need more resources. The whole big data phenomenon could be a way for data professionals to start a new hardware renaissance, where hardware budgets grow and we begin to replace our current systems with bigger, faster servers.

    Or perhaps it’s a way to offload some of the system administration for individual servers and move to cloud services. I don’t think that’s necessarily a bad move for many DBAs as it would allow them to focus on data management, and information extraction rather than dealing with storage and hardware management.

    Big Data is in the news, and it’s being used by vendors to sell new products and services. From Hadoop to new SANs to BI interfaces, there’s no shortage of places where the term “Big Data” might be used to try and shorten the sales cycle. As a data professional, it’s important that you understand what your needs really are, and if the term is being in a way that actually provides some value to your company for the money spent.

    Steve Jones

    The Voice of the DBA Podcasts

    We publish three versions of the podcast each day for you to enjoy.