Probably the most often asked question on SSC is about full transaction logs, and often full disks. It happened again recently. I don’t have an exact count of threads, but rough searches show this to have 70,000+ results, when most other searches I run show many, many fewer results in the forums.
I talked about this recently in my backup talk on the SQL in the City tour stops. Back up your logs.
This editorial was originally published on Mar 26, 2009. It is being re-run as Steve is away at DevConnections.
One of our SQLServerCentral authors sent me this post, which I found to be pretty amazing. Apparently hackers broke into a backup server at WebHostingTalk, which is a community for hosting providers to discuss issues. They deleted backups first, and then moved on to deleting tables from the database. It resulted in some downtime, and then a restore of an old copy of their database prior to trying to restore more recent copies. More information from the admin is here.
Now that’s just malicious and nasty.
Hopefully this wasn’t an inside job, though I could see that as a possibility. For a service like this, that just provides a place for a community, let’s people interact and talk, this is just vandalism. It serves no purpose. It’s likely no one even knows it was “M@M@sB0Y” or some other hacker, so there’s no fame, and it disrupts people who just want to converse with colleagues.
I’d hate to think about this happening here. We have lots, and lots of posts from people all over the world, and while it wouldn’t kill us, it would really annoy many people that have volunteered their valuable time to help others by losing their work. We definitely need to make sure we don’t have an issue here.
I see two takeaways from this incident for most DBAs and administrators. First be sure that your backup servers are just as secure as your production ones. There is production data in the forum of backup files here, and you should be providing as much security for them, and perhaps more, as on other servers. Don’t treat these servers lightly.
Second I think this shows that there’s a need for tape backups, or some type of non-linked backup. A tape grabs the files from your server, completely separately from the SQL Server (or other application). It is a pull link, and it’s not obvious from the source server where these files have gone. That’s good in that it prevents some type of attack on the main server from propagating on. Most people use a push from their server to a remote device as part of the backup process. That’s OK, but it provides an easy link for someone to attack the backup server from the main one.
And one more benefit of tape? Usually they’re rotated out, so even if a hacker or insider knows how to get to the backup system, without physical access they can’t touch your tape.
Steve Jones
The podcast feeds are available atsqlservercentral.mevio.com. Comments are definitely appreciated and wanted, and you can get feeds from there.
You can also follow Steve Jones on Twitter:
Overall RSS Feed: 
or now on iTunes! 
Today’s podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.
I really appreciate and value feedback on the podcasts. Let us know what you like, don’t like, or even send in ideas for the show. If you’d like to comment, post something here. The boss will be sure to read it.
One of the directions in which Red Gate Software is to build more tools and utilities that all of you can use for working in cloud environments. We aren’t ignoring the desktop tools like SQL Compare and Prompt (v6, yeah!) that you’ve come to appreciate and depend on, but we are investing in expanding these tools to work with a remote, services style architecture since there are plenty of companies looking at AWS, Azure, or other providers for a portion of their applications.
I’ve been learning a bit more about the way cloud services work and one of the things that becomes important in an IaaS environment is scripting. The environment you get is a base install and if you need more software or processes to run when a new VM starts up, you need to script the various installations or configurations.
There are some great scripting features for SQL Server, like Finebuild for installs and Powershell for most everything else. Scripting is a great way to accomplish tasks, but like most skills, it needs to be used in order to be efficient and practical for most tasks. I’ve struggled with this, and I’m sure many others feel the same way.
However I am starting to think that Powershell might be the best investment in my future skills. The quicker I can write it and build scripts to accomplish tasks, the more I’ll be able to do over time as I build a library of commandlets and scripts. That’s if I remember to document them well so I can easily search for what I need.
If you’ve ever worked in a large environment, where the numbers of nodes number in the thousands, or tens of thousands, then you know many software packages don’t scale to that size very well. I’ve seen quite a few pieces of software that were well written, and ran well with hundreds of items, but choked miserably when the count crept past a few thousand. Typically those are edge-case sized installations, and most applications aren’t designed for, or tested at, those scales. I haven’t registered 1,000 instances in SSMS, but I could guess that it would not necessarily run smoothly with that many instances being examined.
Software designed at scale tends to take the “install another central node” and manage multiple central nodes individually. That quickly becomes a pain point for the humans that administer the systems as they must learn and remember where individual systems are being managed. It’s not much different for hardware as well. When a system exceeds the capacity of a single node, we often find that we are adding multiple modes and managing them individually. Many backup systems work this way, adding new units to handle the additional backup space requirements.
However there’s a staffing cost as well, and it’s one that can be overlooked as a company grows. More systems mean more backups, more maintenance, and more failures. Even with automation and standardization, there’s a physical workload increase on your staff, and also a stress level increase from managing more systems. It can be easy to overwhelm your staff, assuming they can easily handle the additional load because they have plenty of tools to do so.
Tools are required, and whether you build or buy them, you’ll make use of them if you want to minimize your salary costs. However you still need to make sure you grow your staff, train them, and give them time off. The last thing you want is to overload and burn out a small staff of one or two. I’d bet that the week your staff quits or takes ill, your systems will choose to fail.
Steve Jones
Video and Audio versionsToday’s podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.
|
Voice of the DBA 