Voice of the DBA

Author: way0utwest

  • You Better Learn to Work at Scale

    Early in my career, I worked on a large Novell Netware installation. We bought a server that was about 1/4 the size of a modern rack, which contained a number of blade slots each holding some type of hardware: CPUs, disk adapters, network adapters, and even disk drives. The disk slots held ~30MB drives, which were large for that time. Both in capacity, and physical size. Each of these monsters weighed a pound or more, and felt like it was built to withstand a fall from a table top.

    That’s one reason that I’m amazed by the 64GB storage in my phone and stunned by the 500GB mSata card I recently bought. The storage capacities we have available to us today, in small companies and startups, are truly stunning to us as data professionals, but not to the people that have the space available. It seems on a regular basis people easily fill up the space they have and clamor for more. Whether that’s valuable data or not, I do know that much of that data still requires management and takes effort from us as data professionals.

    We’re not going to get more help, so as our databases grow, we need to find ways to manage more data, of disparate forms. Not just rows in tables, but Excel files, binary files of all sorts, XML extracts, Office documents, Hadoop hives and who knows what else. We need to be better at working with varying patterns of data, and perhaps files, at scale in order to continue to efficiently manage our workloads.

    Just like system administrators that are starting to deal with hundreds or thousands of virtual instances, we have to be able to write code that helps us manage data in large quantities. I like PowerShell for more and more tasks, but without a doubt, writing efficient T-SQL that works with sets, learning SSIS, package design patterns, and more will become the skills that differentiate the valuable and capable DBAs from those that just keep the lights on.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.5MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

  • Out of Space

    I’ve used placeholders to ensure I don’t run out of space on my machines. That’s worked well, and usually I can delete a placeholder when I have an emergency, and then clean up the drive.

    However I got a low warning today when trying to do an iOS upgrade.

    space_a

    I’d already removed placeholders, moved all my libraries and downloads to other drives, and installed most programs in other places. However my profiles are still here and they had grown large. I couldn’t really find anything else to delete here that might clean up space.

    I thought about moving profiles, but I saw this post that noted we shouldn’t move profiles. I’m a power user, but I don’t want to debug or rebuild things on this machine, and especially not now. I want it to work.

    That left me with one real choice: upgrade the hardware.

    I asked on Twitter what people liked in SSDs, and got a number of recommendations, but it seemed that Samsung got the most votes in my informal survey. I even got a link to Tech Bargains for drives, but a few of the links I checked went to EBay or similar discounters. I don’t think there’s necessarily anything wrong with EBay merchants, but in this case I want the drive to be new, working, warrantied, and shipped quickly.

    Amazon is my go-to place for lots of stuff, partially as a Prime member, and partially because they are so easy to work with. In this case, I saw the Samsung 240GB 840 series for about $135 and decided that was a good move. I don’t expect to put much more on this C: drive, but it might grow a bit. I ordered this, and it should arrive by Friday.

    I also decided since I was upgrading things, and my wife’s birthday is coming, I grabbed an SSD for her laptop along with a 4 -> 8GB memory upgrade. Hopefully that will be a nice surprise for her.

  • Normal Data

    I thought this was an interesting post: Your data is not normal. It talks about the fact that we often assume some normal distribution when working with data. We seem to think we can extrapolate, often in a linear fashion, based on past experience. We often use functions and write queries that tend to think of normality as having some standard distribution.

    However that’s not often the case. It’s not necessarily even the best way to begin looking at data for patterns. We might need to apply a number of different types of algorithms to a set of data to see what types of analysis might be appropriate. Not sure what I mean? Let me give you a few examples.

    As a speaker, I often get feedback from my sessions as a series of forms where people have rated the session on a scale and possibly included comments. Many speakers receive the data in the same format, but I’ve often seen speakers get one form that rates them poorly, or has a complaint. The speaker will then view that one item as somehow representative of the session. This despite the fact that they might have ten forms with high ratings. They’re somehow assigning more weight to the negative rating then the all the others. There’s perhaps some value here in doing so, since there might be a legitimate complaint, but often we need to discard this one form as an outlier.

    As another example, I worked with an import firm that was trying to estimate future purchases based on past ones for ordering purposes. Initially they looked at an average of the last three months (rolling) to place future orders. However I pointed out that we had some seasonal fluctuations in our business. There were a few times a year that the rolling average would dramatically under or over estimate the product we needed. As we dug deeper into our historical data, we learned that a better estimate required some level of complexity in our formulas, but we also realized we needed to allow for human overrides when other information was available. That was a fun project, but one that was quite complex when we analyzed the data.

    There are numerous other examples I could give, but the point is this. Learn about your data. Dig into patterns, and don’t assume that normal for one set of data looks anything like normal for other sets.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.0MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

  • Time for Learning

    I was chatting with a friend recently about our careers and how much we have learned over time. We are both interested in continuing our education in technology and debated how we might proceed. I mentioned that I was thinking to spend some time following Colleen Morrow’s journey to pass the 70-463 exam, which is one exam I need to complete a certification. I think that it is hard to work through a topic without having some plan for doing so, and I prefer to follow a book or another person’s plan to guide my learning.

    My friend talked about their experience with Pluralsight, which I’ve also used. They mentioned that it’s exciting to get started, and you certainly can work through a number of courses over there, but at some point you’re just going through courses. You really need to stop and practice the skills you’ve listened and to and been quizzed on, otherwise much of that knowledge won’t be retained.

    I’ve found that to be true as well with classroom training. A week’s worth of knowledge is too much to learn at once, and only with a lot of practice, and substantial documentation available after the training, does the class really do you a lot of good. However, that’s the way that so much training is organized in the technology space.

    Ideally, I think that the best way to actually teach someone a skill is the way that many college classes are organized. Learn over time, slowly, with a few skills taught and then practiced. Repeat that for months and you’ll gain knowledge. However who has time to spend six months learning something like SSIS, a skill at a time.

    You do, and I do. In one sense, our Stairway Series provide this, as does Pluralsight. These methods give you the ability to work through a subject slowly, in a guided way, at your pace. What these methods don’t do is push you to keep moving, which is perhaps the most difficult part of learning a new skill for most people.

    Email reminders, pings, and all the possibilities available with technology likely won’t work for most of us. However I might suggest the best way would be to find a few friends or co-workers looking to acquire the same skills and press each other to stick to a schedule. It’s worked for exam study, and I bet it would work here, applying some pressure for you to keep up with your friends, and continue your learning.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.9MB) podcast or subscribe to the feed at iTunes and LibSyn.