I wrote recently about a bad first day for an intern. He/she was fired, without cause in my opinion, when a production database was damaged while following a document for developer setup. The situation felt like a mistake, and one that wasn’t necessarily the fault of the individual. To me, this was extremely poor handling of the situation from a CTO.
In the discussion for the piece, someone pointed out that it might not just be a new employee that makes a mistake that causes downtime. Certainly an unexperienced employee could have caused the issue, but I know there are people with lots of time in a position that make similar mistakes. It could be that someone that has been there a long time followed a poorly documented procedure, or applied the procedure to the wrong situation.
In sports and the performance arts, I’ve heard a saying that says: amateurs practice until they get it right, but professionals practice until they don’t get it wrong. It’s the difference between just gaining a skill that you complete and continuing your learning to ensure that the execution of that skill is ingrained into your being. You internally build the muscle memory and skill to complete a task correctly over and over with practice.
In the database world, this might be the difference between someone that has restored a database (the amateur) and someone that regularly practices restores of not only full database backups, but differentials and logs, tail log backups, filegroup restores, and more (the professional). The former has a skill, but the latter is confident and likely to complete the skill in a variety of situations without error.
Practice won’t prevent mistakes, especially under pressure, but lots of practice does help us learn how to perform better. We learn what options are available, what issues might arise, and which choices we should be making. Just as professional athletes and musicians can make mistakes, we can as well. However, those mistakes should be rare. We should get most things right as the product of numerous hours of working deeply in, and understanding, our craft.