Voice of the DBA

Category: Editorial

  • Learning from Microsoft

    I know that many data professionals out there are skeptical of the cloud. Certainly there are problems with reliability, security, and more that prevent many of you from migrating to some online database. However that doesn’t mean that everything being done in platforms like the Azure SQL Database is a bad idea.

    One of the interesting things that has occurred at Microsoft is a completely new engineering effort for SQL Server. It seems there is one SQL Server development team that works on the Azure SQL Database. This is (I guess) the main branch in version control, and what we see with SQL Server 2016 is a subset of the code being built for the on-premises, boxed product.

    Certainly there are some features that exist in the box that aren’t in Azure, but relatively few of them. Those features, and many new ones, have to be included in the version control system, but they can’t be activated. Microsoft must be making extensive use of feature flags, allowing their developers to build and test features in the Azure cloud, but not make them available to customers, or potentially even other parts of Microsoft.

    If you really think about the effort to build a platform like Azure SQL Database and allow it to be a development environment as well as a production environment, that’s an amazing engineering effort. From the outside, I’m amazed at how far the SQL Server team has come in the last few years. I’ve seen a number of features released in the cloud first, and then appearing in the SQL Server 2016 CTPs.

    I think there are some amazing lessons here for software development. Certainly I think many of us could learn from how feature flags and multiple versions of software can co-exist in systems. That would certainly help many of us deploy new software in a way that minimizes interruptions to customers and clients. Or manage those multiple clients with different requirements from the same codebase.

    I hope that Microsoft will continue to evolve, as well as share their knowledge with the world. There are other companies that seem to be accomplishing amazing things at scale, Netflix, Amazon, Google, and more. Some of them share their techniques, but I truly hope that the methods they all use for building software become spread throughout the industry and help all of us build better, more secure applications.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.3MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • The Dangers of Algorithms

    I ran across an interesting piece in Wired recently that talks about algorithms and the power they hold in the modern world. I don’t think it’s a great piece, and it starts to mix code, data, algorithms and other aspects of modern software in a way that seems disjointed to someone that works on software. I think there is come confusion about how data and software differ.

    I don’t deny that there are potential negative impacts of the ways in which we process data. Certainly we have bugs in software that cause issues, but we also end up with implementations of specifications that may not quite match up with the expectations of our clients. Even in simple applications, if I’m not constantly interacting with a client in a detailed way, I find that the software can operate in a subtlety different manner than others expect. Of course, there are certainly people that deliberately program systems to intentionally help, or hurt, others, but I think we have just as many issues from small communication mistakes. That’s one reason why DevOps, CI/CD, and frequent releases are valuable. We get feedback quicker on the implementaiton of the design.

    Does it make sense to disclose algorithms? Make source code available for examination? I definitely think there are intellectual property issues to consider, but when there are legal challenges or arbitration, I would expect that the actual code of a system can be examined. We definitely need to ensure that code examined in these cases is not part of the public record. However there needs to be open examination in such a way that the groups involved, and perhaps even other interested parties, have confidence that the algorithm performs as expected.

    This becomes easier in databases, since our code is often easily accessible, but more complex in applications. Many of us realize that subtle changes in application code can cause issues for data, which is part of the reason we like declared referential integrity in our databases. That way we’re not affected if an application makes a mistake in enforcing our data integrity rules.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.7MB) podcast or subscribe to the feed at iTunes and LibSyn. 

  • Getting Big Data

    Many of us work with data every day. Often we do so at work, but plenty of us also look at data outside of our jobs. The recent Power BI Report contest shows that our colleagues look at all sorts of different kinds of data. What is more interesting to me is that there is a huge variety of data that people choose to analyze, from sports to goverment to healthcare to airlines.

    I encourage you to get a set of data and work with it. I know that I’ve often just created my own sets when building demos, but there are so many interesting sets out there that I would think many of you could find one that might be of interest to you in building experiments. After all, knowing something about the data is important in deciding how to analyze or visualize the information contained within.

    One of the things that I’ve had people ask about is where can we find data. How can you get a set of data that you might be interested in. What’s amazing is that there are all sorts of sources out there, many of them free. I ran across a list of data sets at Forbes that contains quite a bit that many of you might want to play with. I know I’ve used a few of these in the past and am currently downloading various machine learning sets from UCI with which to experiment.

    Certainly some of these data sets cost money if you want specific information, which makes sense. It takes resources to compile data, and a nominal charge makes sense. However most of these sources will let you download large sets to play with if you choose. You might need some PowerShell, Python, or other scripting skill to get lots of files, but that’s a good excuse to learn another skill.

    Be aware that many of these sets are CSV files, so you’ll need to work on your ETL skills as well to load these into a SQL Server database for fun. Yet another excuse to learn. And if not, you could just take them as is for your very own Power BI dashboard. And to help you along, and provide a resource, I’ve started my own list of sets at SQLServerCentral.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.9MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • A Good Use for Hadoop

    Netflix is an amazing company. I’ve been a customer for over a decade, starting with the well known red envelopes that moved DVDs through the mail. Since then I’ve continued to subscribe as a streaming company and have enjoyed quite a few of their original programming efforts.

    However I’ve also been fascinated by the way in which Netflix has made use of technology to grow a unique kind of company. They own very little, having used Amazon’s AWS to host their infrastructure and mostly serve content they acquire from contracts with various media companies. They are famous for their Chaos Monkey approach, having machines randomly fail to test the fault tolerance of their entire infrastructure. I read recently about the closing of their last, small data center, so outside of employees’ laptops, they keep all systems in the cloud.

    However I ran across a post on the scale of their data flow, and it’s amazing. Apparently their events are generating 1.6PB a day of data. That’s incredible, and a scale at which very, very few of us will ever work. Personally, I think that’s interesting, but I’d prefer not to be working on a PB of changing data a day, but I might feel differently if I worked at a company like Netflix that has obviously been successful with dealing with data at that scale.

    The post notes that they put this data into Hadoop, where it can then be queried. I’ve often wondered what the domains are for using Hadoop. Most of the people I know that have tried it aren’t really working at large scales. I think that a well built ETL process would allow their data to easily work in a SQL Server data warehouse, and possibly even the Azure SQL Data Warehouse. However at 1.6PM a day, Hadoop seems like a much better choice.

    I’m curious how often that data is queried. Is most of that 1.6PB actually used in reporting, or is much of it lost in the shuffle and ignore? Is there a process to aggregate some of this raw data and then delete the older values? Can 1.6PBx365 actually be useful in analyzing your business? I’m sure some is, but wouldn’t a lot of those events lose their value over time?

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.7MB) podcast or subscribe to the feed at iTunes and LibSyn.