Voice of the DBA

Tag: databases

  • Serious Storage

    This was our quarter million dollar server years ago.
    This was our quarter million dollar server years ago.

    Years ago I worked for a company that had a Novell network. We had a multi-server environment with lots of users and were having issues with both space and users. We bought a Netframe server, packed with 350MB drives and a limited edition 1000 user version of Netware v3.11. This was also the time when I got to use my C-language experience, writing a login utility that would handle our user IDs above 250 on Netware since that was the limit for all of our other servers.

    That server, which cost something like $280,000 in 1991 was the biggest one on our network, with something like 8GB of storage. That seems like a pittance today, especially compared with the sale EMC just made. The Vatican is getting 2.8PB of storage from EMC for its library. EMC is also providing consulting services to digitize some of the historic manuscripts and documents that have been deteriorating from user and handling.  It’s an ambitious 9-year project, of which this is just the first 3 years.

    That’s a serious amount of storage, and while most of it will be used for raw, unstructured storage of images, some will have to house a database. There will be the equally critical part of cataloging and organizing the meta data about these documents into some type of database. I don’t know if this will be a relational or some other store, but without some database that keeps track of what each image represents and how to retrieve it, it’s entirely possible that these documents might get lost, in the same manner they may be lost in physical storage today.

    The scale of this project somewhat astounds me. Going from MB and GB to thinking about PB and EB is something many of us will deal with over the next decade as our organizations gather, store, and manage more and more data. Perhaps a few of us will get to work on some interesting projects like this one that look to preserve valuable knowledge from our past.

    Steve Jones

    The Voice of the DBA Podcasts

    We publish three versions of the podcast each day for you to enjoy.

  • Self Describing Databases

    Self describing databases would be very, very handy.
    Self describing databases would be very, very handy.

    When contained databases were introduced in SQL Server 2012, I thought they were a very nice enhancement to the platform. The idea of having security managed inside the database, thereby removing the problem of orphaned users, was something I looked forward to. Preventing collation conflicts is also a nice benefit, though I’ve never had a problem with this issue. However I was, and still am, hoping that this was just the beginning of what a contained database would be. I had a vision of a self-describing database.

    There are many things that we need to manage with databases: maintenance, backups, auditing, performance statistics, alerts, and more. Most of these are set up at the instance, but I’ve never quite understood why. Software should help us work more efficiently, and work with us, not against us. So many of these administrative items are dependent on the needs of a particular database or application, but we perform all these actions at the instance level.

    Backups are an example of a database dependent item, as we constantly teach new DBAs. However the backup plans are set at the instance level, and if you have disparate requirements for different databases, these result in different backup plans. The MSDN documentation says we should even have a manual for each database. If that’s the case, why is the information about a database backup stored outside of the database? I think it would make more sense for the backup plan to be stored in the database, and when the database is attached, or the plan created, the instance would read this plan and create the necessary jobs. If we detached the database, the jobs would be removed. We run recovery in each database, couldn’t that serve to enable or disable any alerts, jobs, etc. for any database that wasn’t accessible?

    I envision much more than just backups as being a part of the database. Every action we need to manage in a database from alerts to auditing, should be stored there. Let the Agent on the instance read the database and automatically set up the jobs needed. If there are conflicts with scheduling, alert the DBA at the instance level, but keep the metadata about the needs of the database in the database. It would make consolidation, or movement of databases much simpler. It would also fit with the idea of a database service, rather than a database server.

    Steve Jones

    The Voice of the DBA Podcasts

    We publish three versions of the podcast each day for you to enjoy.

  • Look Beyond the First Result

     

    I'm not sure Orbitz did anything wrong, but this certainly made me re-examine how I look at results when searching.
    I’m not sure Orbitz did anything wrong, but this certainly made me re-examine how I look at results when searching.

    This story about Orbitz was very interesting in the way metadata about users was affects the results of queries on the Orbitz site. It’s slightly skewed to imply Orbitz was charging OSX users more than Windows users, but that doesn’t seem to be the case. Instead, if an OSX user was detected, the order of search results was changed to show more expensive options first. Users were understandably upset, as some of them ended up paying more for services than they might have otherwise.

    However is this a valid way to present the data? From the business side of things, this makes sense. If you identify a trend, you may look to exploit it or incorporate the potential results into your decision making. I could even see some sort of business intelligence system making this decisions automatically, without human intervention. As we build more sophisticated software, using a large set of inputs, I could see exactly this type of thing occurring more frequently.

    Is it fair? Is it moral? Those are tough questions. We often want to go where the data leads us, and while many of our human tendencies don’t make any sense, the data sometimes shows patterns in our behavior. In the end, I’d argue this was a very poor model of the data. All Mac users aren’t looking to spend more money on goods or services just because they might have purchased a more expensive computer. There are likely quite a few other qualifications inputs that should be considered as well.

    Steve Jones

    The Voice of the DBA Podcasts

    We publish three versions of the podcast each day for you to enjoy.

  • A Computer Goes to College

    The incoming freshman at RPI.
    The incoming freshman at RPI.

    A college education is something many parents desire for their children. It’s almost expected that all school children in the US will grow up aiming to attend some university. Not all kids will follow this path, and whether it’s the best choice for most of them is debatable. Apparently IBM thinks it’s little one is ready to go as well.

    A Watson supercomputer is being sent to Rensselaer Polytechnic Institute in New York to go to school. In it’s quest to defeat the Jeopardy champions, this type of computer was fed a tremendous amount of information from general sources. A clone of Watson received specialized medical data to help doctors diagnose patients better. This last incarnation of the computer system will be fed information from various researchers in different fields at RPI. What will come out of this, experiment nobody knows.

    The Watson system going to RPI seems modest in size, with 15TB of storage reported. That seems small. The specs posted for the Jeopardy system include 2880 cores and 16TB of RAM. That’s not the size of system many of us every get to work with. The software, called DeepQA, is the core of the system and allows it to process vast amounts of data and make connections between the data to answer questions.

    2880 cores is a lot, as is 16TB of RAM, but there are SQL Server installations that are getting close, with 2TB of RAM. Some Parallel Data Warehouse installations with hundreds of cores. I can foresee a day when Watson-sized systems are more common. Perhaps we’ll even get a Watson version that helps us better analyze the data we work with today, a version that works closely with the DBA-turned-data-analyst to extract information from the petabytes of storage we’ll have accumulated.

    Steve Jones

    The Voice of the DBA Podcasts

    We publish three versions of the podcast each day for you to enjoy.