Voice of the DBA

Tag: software development

  • The Impact of Outages

    This is not what any of your clients want to see. So prepare for issues

    Are you going to start seeing more pressure for outages in applications? I suspect many outages are caused more by application issues than database ones, but those two are becoming very tightly linked as we look to more rapidly deploy features and enhancements in our applications, which often include database changes

    This outage from United shows that there can be a huge impact, not only financially, but also an inconvenience to clients and potential lost future business. A company might struggle to with future business after a large outage, especially when there are so many other choices easily available to consumers across the Internet.

    That brings to mind a very interesting problem as companies grow and look to build scalable systems. Large groups of servers require some level of standardization, mostly for the ease of management by IT workers as well as the ability to train future workers to understand the systems. However that standardization becomes a point of failure when there is a problem during an upgrade, or even a hack from some type of malware.

    I saw an interesting piece on how Netflix has tried to expect, and handle failures in the cloud, and a comment from Jeff Atwood that you ought have your own chaos monkey to regularly test your systems. Interesting advice, and in many cases, it’s probably good advice to ensure that both your systems and your people know how to deal with outages.

    You probably cannot eliminate outages, as Netflix and many other companies have learned. However you can work to ensure your people and systems know how to respond. I also wonder if having (at least) two versions of your systems out there at all times that work in a similar way might be a way to provide some tolerance against a single point of failure. I don’t know how you might implement this, but it might provide some protection against a failure in a completely homogeneous environment.

    Steve Jones

    The Voice of the DBA Podcasts

  • Leave Developers Alone

    Working in my own litlte world

    A computer deals with interrupts all the time. They are the mechanism by which it can simulate multi-tasking among many different programs in a modern operating system. However those interruptions have a price, and too many of them can affect performance. As hardware grows larger, we have other issues from interruptions that we try to mitigate with techniques like soft-NUMA affinity or multiple pipelines built into hardware. All of these are designed to prevent a computer from spending any significant time on non-productive tasks because of interruptions.

    In the real world, many of us deal with regular interruptions from work. They might be emails, instant messages, phone calls on a cell phone, or the old fashioned someone-stopping-by-your cube-to-chat. All of these things add up to less productivity, especially for developers. One study finds a 10 point IQ drop from regular email and phone interruptions. I don’t know about you, but I’m not sure I can afford a 10 pt drop in IQ when I’m working.

    Some companies are starting to realize that developer’s brains are a scarce resource, andinterrupting them can dramatically impact productivity. I have found some places, like this one, that are setting aside quiet time for developers to work without being bothered. Similar to the technical debt that Steve McConnell has talked about, there seems to be an interruption tax that some development shops are loathe to pay.

    Even if you don’t gain any productivity, or have fewer bugs, or ship more often, I think that your developers will appreciate it. It could be an easy way to increase happiness, improve retention, and even sell your company as a good place to work. And it’s easy to implement: just leave people alone a few hours a day.

    Steve Jones

    The Voice of the DBA Podcasts

  • Self-aware Instances

    I like the idea of limited self-awareness in our systems.

    I’ve been seeing lots of monitoring solutions written up lately. I saw this blog from Phil Darmanin on SQLServerCentral and we have had numerous articles written over the years on building monitoring solutions. The build vs. buy debate is one that companies regularly have, and if time is an issue, my company, Red Gate, sells a great solution in SQL Monitor that’s worth checking out.

    However most of the solutions I see use a similar architecture. They have a central monitoring server that reaches out to all other servers and gathers information from them, which is rolled up into a database for reporting. That’s fine, but in many cases there isn’t a ton of testing or error handling built into home grown solutions, and when they fail, you lose data for the day. You also either go into fire-fighting mode to get things working, or you just drop data for a few days.

    However I’ve had a different approach in the past. I’ve typically had a set of monitoring scripts that I’ve deployed to each instance, having them run on a schedule that gathers all the data on settings, changes, backups, jobs, etc. for that instance. I then have a central server that just reads that data from each instance, so if I have an issue with my rollup process, I still have the data on each instance. For this Friday, I wanted to ask this question of you:

    Do you like the idea of architecting processes to be more self aware and distributed, or do you prefer some central process that reaches out to control everything?

    It’s an interesting debate, and one that reminds me of an MIT competition years ago. Early in the 90s, there was a competition where different professors were trying to build robots to navigate a space. One team used a central CPU with complex instructions that controlled the legs and eyes and made decisions. The other used smaller, less powerful controllers in each leg and eye and had them “learn” to work together. Both worked well in different ways, but the latter approach always struck me as a more interesting one.

    For your SQL Server, or application code, which approach to you prefer and why? Let us know this Friday.

    Steve Jones

    The Voice of the DBA Podcasts

  • Five Lines of Code

    An interesting story on how a database speeds up by changing some application code. Worth the read for sure.

    I caught this from Andy Warren’s blog, and it shows why we need developers to think for themselves and ask relevant questions, not just meet the spec. If someone had asked if Accuweather needed micro-granularity for location earlier, they might never have had issues.