Voice of the DBA

Category: Editorial

  • Big Power Issues

    Recently British Airways had a massive computer failure and had to cancel hundreds of flights across a couple days. This was a large disruption for travel as tens of thousands of passengers were stranded around the world. In addition to the concerns many of us have about security, I also worry about large scale systems that continue to grow in scale, with more users depending on them, and not necessarily getting updated with modern technologies. While I know many of the customer facing systems have been enhanced for companies like airlines, I’m not sure the core infrastructure that backs these systems has changed.

    British Airways has blamed the cause of the outage on a power failure, with some finger pointing between the airline and their outsourcing company. BA appears to say a power surge caused problems with it’s UPS’s and batteries, which resulted in a situation where “the controlled contingency migration to other facilities could not be applied.”

    I’m sure some of the information being published is carefully vetted by lawyers and doesn’t necessarily reflect the actual technical issue, but it doesn’t matter. A well designed HA solution shouldn’t care if a primary system drops off line, falters, or anything else. In the worst case, any IT system failover can be forced, understanding a potential loss of data. In the SQL world, we could always remove a primary and just go with whatever data is on the secondary. Certainly within a few hours we could be up and running.

    I suspect that the architecture of the BA system is not well designed for large failures. I know that dealing with power is tricky. While working for a large company (10k+ employees), our data center went down one day, which affected our public presence, customer support, and plenty of internal systems. We had UPS service underway, which had disconnected main power for some reason and we couldn’t quickly switch to the public grid when a system failed. It took a few hours to reroute power, during which our CTO paced the data center floor in an irate manner. Certainly that didn’t speed things along.

    As noted in the piece I’ve linked, there are some strange inconsistencies with the explanation. It’s entirely likely that a data quality issue caused problems that took hours to rectify. I’ve been on the wrong side of those, working through queries and attempting to piece together some version of the correct data from various sources. If that’s the case, then I’d really like to know what the faliure of their systems allowed bad data to disrupt operations. There are learning opportunities here for us data professionals.

    I do hope that some of the details of how BA architected their system gets shared among technical staff. At the least, I’d like to have large companies like UPS, Wal-Mart, British Airways, and other companies publish and share information about technical details and architectures. Some of the tech companies do this already (Amazon, Google, etc.), but it’s good for other industries to share their information as well. We owe it to the industry to learn what works well and what fails as we grow systems to larger scales and more complex interactions. Certainly I’m proud that many SQL Server experts share their experiences, helping others learn and make fewer mistakes in the future.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 5.4MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Presentations that are Easy on the Eyes

    Recently I was at a conference and was struggling to see some Visual Studio code on the large screen. As I age, I find myself trying to sit in the front of most talks if I need to see the screen. In this talk, the speaker was using the default dark theme in VS, and it was hard to read the C# code. It often doesn’t matter if I can’t read code, after all, I’m not coding during the talk, but I do like to try and follow along as the person is talking.

    I’ve had the same issues with presentation templates. Some companies seem determined to stand out with a colored template for their event, and as they try to differentiate themselves, some will invariably use darker colors for a background. I’ve run into more than a few of these where they look fantastic on a monitor but are barely readable on a projector.

    A few days later I ran across a post from Aaron Bertrand on using dark settings for SSMS. At first I thought that was a bad idea, and I debated this with a few people. I even asked a quick Twitter poll, with most people responding they preferred a light template, but only a few people responded. I decided to seek a little wider audience and see what others think.

    What colors are easier for you to see when someone presents? Do you prefer lighter backgrounds or darker ones? I know we all want higher contrast between the foreground and background, but as we add in code coloring and other elements, what do you prefer when you’re watching something on a screen larger than a monitor?

    I do need to experiment a bit, and as Aaron suggested, blog about my settings. Mostly so that I have a copy of them that I can easily find on new VMs and machines. I don’t think dark backgrounds are better on projectors, but I haven’t really experimented. Let me know what you think, and perhaps I’ll have a whole new look for you later in the year as I present in the fall.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.1MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Consider the Downside

    Technologists are often seen as cynical and disappointed by how poorly much code is written or maybe because an application architecture isn’t well designed. I hear no shortage of complaints over other people’s code, one of the reasons I think we have so many new projects started on Github (or other pubilc repos). Everyone thinks they can build better software than others.

    However, we are often quite optimistic when we write our own code. We throw together quick apps or scripts that solve a problem, expecting the best conditions. In general, we don’t think widely about the possible issues with our code. We build and test for the happy path.

    I thought about this as I saw this post on twitter. It shows a number of Trello boards that contain passwords. To be fair, as I researched this, most of these items are private boards, though seeing names and passwords in a Google search is worrisome. I suspect most of these boards are places where less than trustworthy individuals are storing data, but I did see no shortage of public Trello boards for various software development teams. No passwords were shown, but plenty of notes, and I didn’t spend a lot of time digging through the various cards, something I suspect a hacker would do.

    I think many cloud services are very useful and they help us work in new ways. The thing we should all keep in mind is that the search engines are always probing data, even if hackers aren’t. Perhaps there isn’t a concerted effort by anyone to look for our particular set of data, but you never know. Someone may stumble upon our data and if it’s publicly visible, that’s an issue. Even server names, IPs, and more can provide information that we’d rather not disclose.

    If for no other reason, consider that employment has changed in the world. More and more of us change jobs often. Today’s co-worker might be tomorrow’s disgruntled ex-employee that discloses information to anyone, or worse, to specific hackers that might make use of it. I’m not sure how much data from our companies is valuable in a general sense to a large group of hackers, but I know there are plenty of technical vandals that would use the information to cause disruption, because they can.

    Whenever you look to setup some new process, especially with secure data, please don’t treat this lightly. Expect that someone will find a way to cause issues. And if you use any service that is connected to the public Internet, I’d be extremely careful of the data that you store there. You might be surprised how much of it is visible in search engines.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.5MB) podcast or subscribe to the feed at iTunes and Libsyn

  • DNA for Data Storage

    In all my years of working with data, there’s one think I’ve learned: it always grow. Over the years, I’ve gone from having to store data in these sizes:

    • A few KB on cassette tapes as a middle school kid
    • Using 360k floppies to 720k for programming, then 1.44MB to 100MB ZIP drives to 600MB CDs to 4GB DVDs
    • 10MB HDDs that grew quickly to be GBs and currently are TB sized in my desktop
    • A flip phone that could handle dozens of contacts with a SIM card to MBs of data in a T-Mobile G1 to a 64GB phone today

    In that time, I’ve often filled up many of those devices and needed more space. I am looking forward to seeing capacities grow again. My 64GB phone is full of pictures and video, and I need to manage the space periodically. The same thing for the 500GB in my laptop and the 2TB on my desktop. I need to add storage, and archive off some older information, but I certainly don’t want to lose any data, especially my pictures. Those might be the most valuable bits I have.

    I am alternately disappointed and amazed by storage technology advances. Capacities have grown tremendously, but speed and latency have had fits and starts. SSDs were a great improvement, and I do look forward to the newer 3D flash for laptops and high performance servers. I am glad there are these large capacity HDDs, but maybe there’s a better way to store lots of data for the long term.

    I ran cross this piece the shows the University of Washington and Microsoft working together to store data in synthetic DNA. They managed to store 200MB, which breaks the old record of 22MB. If they can improve the technology and make a commercial version, the theoretical storage could hold 1EB in a cubic inch. Even if we got 10% of this, 100TB in a little cube would be amazing. You can read more about the project at Microsoft Research.

    I have no idea what the write and read latencies are, but this could provide a nice backup for lots of data that we want to keep around, especially for images, video, perhaps even old auditing or other data that needs cheap, reliable storage that lasts years. I know I’d like to have this capability as my collection of data continues to grow, sometimes at amazing rates. I just hope this is cheap enough for general use, since as a data guy, I’d likely want 3 copies of everything. You know, just in case.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.5MB) podcast or subscribe to the feed at iTunes and Libsyn.