Tag: databases

  • Mining and Algorithms

    Amazon
    The recommendation engine is the result of some complex algorithms

    When we publish articles that look at the performance of T-SQL code, the pieces are very popular. It seems that people are always willing to learn how to write better code, and there is no shortage of people willing to discuss the methods, the results, and the implications of various techniques. It also seems there is no shortage of argument and debate that take place in the discussions as to the best way to solve problems.

    However we don’t really understand what will resonate with people, or what things they’d like to learn. What we would like is a system like Amazon’s recommendation engine that might help us improve traffic, and also bring you information that is more relevant and interesting to your career. If you’ve ever purchased something from Amazon and then browsed their site, you might notice the recommendations appearing all over the site. If you’ve received an email from them, you might see some interesting items suggested to you.

    Amazon knows their engine is valuable, and it’s constantly being refined. We don’t know exactly what they use to build the recommendations, but they are constantly analyzing the results and tweaking their system. That makes sense for their commerce oriented business, since they can easily measure the results by looking at sales, but it might also make sense for other businesses. Many of us have reports, and other queries used to perform analysis on our data. Do we regularly analyze the performance of those algorithms and look to tweak them?

    I’ve found many employers in the past were content to develop a system and leave it alone until some new requirement arose. One company didn’t like that approach, and regularly looked to tweak our systems to try and get more performance out of an inventory analysis application. It was frustrating to regularly rewrite a series of reports, but also challenging to try and find ways to solve a complex problem.

    As more and more data professionals begin to work in the analysis and business intelligence areas, learning to build algorithms, and tweak them over time, will be more and more important. It will be challenging, but it should be fun if you are excited by the challenge of improving your algorithms over time.

    Steve Jones


    The Voice of the DBA Podcasts

    We publish three versions of the podcast each day for you to enjoy.

  • Use Tools

    Fortunately building tools for IT is easier than this.

    As someone that works with Information Technology, I usually work on applications designed to make work easier. For example, we have accounting systems that ease the handling of debits and credits, they detect mistakes in data entry, and in general require less people to handle the accounting needs of many firms. Do we have less accountants?

    I suppose we might have less, but it seems that the finance departments, handling AP and AR in many companies is as large as I remember from my younger days. Perhaps the department is smaller than it might otherwise be, but it’s usually not small.

    In IT, we have all kinds of tools available to us that can help with our jobs. What’s more, we can create new tools as needed to do our jobs. Powershell or Perl scripting, Codeplex projects, and more are available to most of us to help us better manage systems. In SQL Server, we even have a built in framework, Policy Based Management, that helps us prevent changes or problematic configurations. If you find a third party tool that can help, you can make a case for its purchase, using the time savings translated to actual costs. One of those tools might really make your job more enjoyable..

    In theory, we ought to be able to manage many more systems per person than we used to. Does that many that we need less people? Perhaps, but it seems we grow systems fast enough that we still need to hire more staff in many environments.

    I so often see people working in IT fail to take advantage of all the tools we have to automate much of their jobs. They often tell me if they automate too much of their jobs, they won’t be needed and may get let go. Personally I think that’s an excuse not to exercise your skills, challenge your mind, and get rid of tedious work There’s no shortage of work to be done in most companies. If that’s true, why not use your tools to get rid of the tedious work and spend your time on something more interesting?

    Steve Jones


    The Voice of the DBA Podcasts

    We publish three versions of the podcast each day for you to enjoy.

  • What is Big Data?

    big data visualization
    Big Data visualization of “water” being spoken to a child.

    This industry is full of some very smart people. We tend to get some very strong opinions, and some rather snarky comments at times about the way the technologies that we use are presented. Karen Lopez (@datachick | blog) recently made headlines with her comments on big data. “What the heck kind of definition is that?” she asked when looking at the Wikipedia definition of Big Data. If “big data” is data that is “awkward to work with” then I know lots of people that have had to deal with big data anytime they had to work with outer joins.

    While I’m not sure big data is well defined, I do know that there are large data sets which can overwhelm relational databases, both in scale and the speed at which the data is collected. Microsoft added StreamInsight to SQL Server to help deal with the speed problem, but I’m not sure that relational databases can handle the size issues at the same time. Some large computing and analysis problems don’t lend themselves well to non-relational soluttions, and other technologies are needed.

    I do agree with Karen that big data, whatever that means to you, is not really in conflict with relational databases like SQL Server. There are lots of data sets that work well inside of a relational database, especially when you have built a solid data model and referential integrity. Many problems with relational databases come about because of poor code added to the platform, not a problem with relational platforms in general.

    Relational database systems are not “obsolete”, despite what some vendors of other databases think. I have complete confidence that SQL Server as a relational platform will be used for years to come.

    Steve Jones


    The Voice of the DBA Podcasts

    We publish three versions of the podcast each day for you to enjoy.

  • Self Service SQL

    self service help and advice
    Self service isn’t always better.

    I was looking over the Microsoft Virtual Academy recently, and went through one of the courses they have on SQL Server for the private cloud. I was curious how this solution is being both implemented and presented. It was interesting, and I learned a few things, including the fact that Microsoft is looking to build self-service into this idea of a private cloud. Users can build a virtual machine from a template, with Windows, SQL Server, and more already setup. They receive a SQL Server they can use as long as they like, and can delete it when they are done with it.

    There are a few problems here, and they are similar problems that we’ve seen in the past as we pushed out more capabilities to individual users. The first one is that allowing people to create their own SQL Server instances can be problematic as they don’t do a good job of setting up maintenance. SQL Server is easy to set up, but a little more complicated to manage over time. I suppose templates could potentially have backups and maintenance built in, but if you do this with any scale, you might end up with storage and space issues as well.  There’s only so much automation can do with limited resources, and I could see this ending up causing more infrastructure problems over time for IT staffs.

    The second issue is that people just don’t manage data well. How many people would delete a virtual machine without a final backup? How many people would create machines, load data, and then forget they had data out there? Apart from security concerns, there are likely to be efficiency concerns. How many people might create their own database server for financial data, even though the person in the next office might have already done the same thing? How many reconciliation problems will be brought to the DBA when people load different data sets into their servers, run reports, and then try to compare the results?

    I think IT self-service is a good idea in many ways, but it’s not the answer for all problems, and without some controls, I can see it easily causing as many issues as it solves.

    Steve Jones


    The Voice of the DBA Podcasts

    We publish three versions of the podcast each day for you to enjoy.