Early in my career, I worked at a few smaller companies where a problem with the computer systems meant I went into the office and stayed until things were working. As I went to a few larger companies, I wasn’t alone when a system went down, but the process was mostly the same. We figured out what was wrong and found a way to fix or replace things, occasionally with help from a vendor. Those were the (not necessarily) good days before our internal networks were connected to a public Internet.
Companies developed formal incident response plans to deal with various issues, whether these were problems we caused ourselves or failures of an application. I had the fortune, or misfortune, to be involved in more than a few issues and learned a great deal in how to solve problems as well as how to manage the impact to a large number of employees.
As email and Internet use grew, so did the attacks with viruses and other sorts of malware. Antivirus software helped a great deal, but these days ransomware seems to be a common problem that isn’t as preventable as I would have hoped. Quite a few friends have dealt with ransomware issues, most of which have not been widely reported in the news.
I saw an article about a few things that you might want to consider adding to your incident response plan. While some of these items might be unique to the ransomware threat, the thing that struck me was that there is a need to react quickly, in real-time, in response to any detection of an issue. I can only assume this means that there needs to be some advanced monitoring of nodes to detect issues, and I’m not sure how many organizations would adopt this, but in today’s world where we want systems available 24/7, perhaps they will.
Being on call is a part of working in many IT departments. Having a response plan, even the general outline of one, helps to coordinate resources and ensure that we use people effectively. Tools are important, especially in today’s complex world, and it is important that one of those tools is a simple backup, preferably air-gapped from your main systems. If you don’t have these things in place, you might suggest someone start assembling them. These days it seems it’s not if you will get attacked, but when.
Lastly, I don’t often see this addressed in plans, but make sure you have spelled out some guidelines on rotating staff and getting rest. Far too many companies want “all hands on deck” and forget that normal business still needs to occur. Any incident could last longer than a day, and you want to ensure that some of your staff is fresh and ready to take over from those that do need rest. Don’t be afraid to send some people home, or better yet, don’t call them in the first place.
Steve Jones
Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.
Voice of the DBA 