Voice of the DBA

Tag: NoSQL

  • Pre-built or Ad Hoc

    One of the advantages of NoSQL databases id that the schema and organization of data is very flexible. The various types of databases usually allow the schema or organization of data to vary across entries. I hesitate to call them rows, but essentially each time you add data to a store, you can alter the format of the data inserted.

    For relational database professionals, this seems to be a recipe for disaster, with entirely too much data being captured in an un-organized fashion. At some point a user will want this data to be returned in a report format, which almost always seems to be rows and column related data, even behind the scenes of the incredible visualizations that appear in modern dashboards.

    I had someone recently note that their users don’t want to write ad hoc queries or try and discern the meaning of varying structures of information. They want pre-built structures they can count on and use reliably to answer questions. I suspect many users don’t want to decode the meaning of structures that change, despite the fact that so many users want to reformat and change the shape of data in Excel. Those of you that have to re-import some of these spreadsheets know just how unstructured a set of rows and columns can become.

    I really think that it is important that structures of data be decided upon and ordered in a known way so that users can easily understand the meaning behind the data. However we are gathering more and more data in new ways, from new sources, and we don’t have consistent ways of recording that information. That will continue in the future, and I do think that learning how to access new sources, like Hadoop, and present that data back to users in a familiar format will become a way to show you are a valuable resource for your organization.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.3MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

  • You Better Learn to Work at Scale

    Early in my career, I worked on a large Novell Netware installation. We bought a server that was about 1/4 the size of a modern rack, which contained a number of blade slots each holding some type of hardware: CPUs, disk adapters, network adapters, and even disk drives. The disk slots held ~30MB drives, which were large for that time. Both in capacity, and physical size. Each of these monsters weighed a pound or more, and felt like it was built to withstand a fall from a table top.

    That’s one reason that I’m amazed by the 64GB storage in my phone and stunned by the 500GB mSata card I recently bought. The storage capacities we have available to us today, in small companies and startups, are truly stunning to us as data professionals, but not to the people that have the space available. It seems on a regular basis people easily fill up the space they have and clamor for more. Whether that’s valuable data or not, I do know that much of that data still requires management and takes effort from us as data professionals.

    We’re not going to get more help, so as our databases grow, we need to find ways to manage more data, of disparate forms. Not just rows in tables, but Excel files, binary files of all sorts, XML extracts, Office documents, Hadoop hives and who knows what else. We need to be better at working with varying patterns of data, and perhaps files, at scale in order to continue to efficiently manage our workloads.

    Just like system administrators that are starting to deal with hundreds or thousands of virtual instances, we have to be able to write code that helps us manage data in large quantities. I like PowerShell for more and more tasks, but without a doubt, writing efficient T-SQL that works with sets, learning SSIS, package design patterns, and more will become the skills that differentiate the valuable and capable DBAs from those that just keep the lights on.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.5MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

  • No Handwaving Away the DBA

    There’s a great quote I read, at the end of this article. It says: “…if you think that switching to NoSQL will just let you hand-wave away all of the challenges of running a database, you are terribly misguided.” The context is that all too often people looking to move away from some of the hassles of working with RDBMS platforms, which includes working with the DBA, haven’t completely thought through the issues.

    I do think NoSQL has a place in the world. There are domains of problems that I’m sure Riak, MongoDB, and others, solve in a more efficient way than SQL Server, Oracle, MySQL, and other relational systems. I’m not sure what they are, and to some extent, I haven’t seen good guidance on where particular platforms excel. Most of the articles and pieces on choosing NoSQL seem to be trying to sell me “why a particular platform can replace my other one”, and telling me to add in things like transactions, but not explaining the drawbacks.

    However in all platforms, we often forget that there are really two frames of reference that matter. We need quick ways to work with data, insert it, update it, query it, etc. This is the development frame of reference, and it often dominates discussions of platforms. For good reasons, as developers are expensive, but that’s only part of the system. We also need to consider the operational portion of managing data and applications. When I have those needs to rebuild indexes in relational platforms, or the requirement to periodically merge/remove old versions of documents, or even manage clustered, horizontally scaled resources, we need operational maturity.

    In some sense the DevOps movement is built around merging these two frames of reference into the minds of all those involved. I hope that movement continues to grow and mature, and we learn that developers and operational staff are both necessary, and both need to function in a symbiotic, harmonious fashion.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.6MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

    The Voice of the DBA podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.

  • Moving from NoSQL to an RDBMS

    I saw this link about a company that moved (Goodbye, CouchDB) from a NoSQL database (CouchDB) to a relational one (MySQL) because of some problems they experienced. Sauce Labs provides testing services in the cloud for developers using the Selenium framework.

    The first link is a blog post that talks about some of the things they initially liked about CouchDB and then the problems they experienced. Their move to MySQL showed better performance and stability over the CouchDB they were using. It’s an interesting read, not too long, and it seems well thought out.

    I think that this isn’t a knock on NoSQL databases, and I do think there are problem sets they are well suited to solve. I don’t know that I think because your developers don’t like SQL is a good reason, but there are problems with scale and size that are better handled with some NoSQL solutions. I’m an RDBMS person, but I don’t expect everyone to be that way.

    However this does say a few things about NoSQL maturity. Many of these products have gotten popular quickly, with some success stories, but that doesn’t meant they are necessarily ready for your application unless you have staff that have lots of experience with the tools. These are young platforms, they have problems inside the code, especially if you deviate from the way the authors used the platform, and you may have issues.

    Experience matters. Just as many people have issues with SQL Server scaling when they don’t know how to build a good data model or write efficient queries, I suspect that picking a NoSQL database because it’s cool, or seems easy to implement, or it worked well at some other company. I would almost always suggest that you stick with platforms that your staff is very experienced with, since they are likely going to have issues at scale on every platform, and their experience can make a difference.

    Note that this assumes you have good people. Just because you have a DBA with 10 years of experience doesn’t mean he or she is necessarily good. They could easily have 1 year of experience 10 times. You want people that have worked on a variety of systems in different areas.