I’ve started quite a few jobs throughout my career. In college, I worked in a variety of restaurants, changing employers at the start of every summer and again at the start of the school year. At some of those positions, I had a hectic beginning, with the staff shorthanded days and no one to train me, but they needed me to jump in. More than a few bartender jobs started like that, with me jumping in, making drinks, and teaching myself where things were stored. Luckily there weren’t a lot of mistakes I could make that might severely damage the business.
Fortunately the frequency of new jobs has declined over time, though if I changed positions at this point, I’d go through much of the confusion that many new starters have. Where are things, how do I get work done, what credentials and procedures do I use, and more would be the questions on my mind for a few days. Perhaps a DevOps shop might smooth the process out and prevent issues, but I could see how this type of mistake could be made: a new intern used a document to setup a database, and ended up crashing the production database.
There are lots of comments on that post, but essentially a setup document contained production values (url, user, pwd) and a junior developer used those values instead of the output from their local laptop. I common mistake. A document should have no ambiguity, and certainly should never have production passwords inside of it. Individual credentials should always be used to connect to sensitive systems, with some auditing of who connected (and hopefully what they did).
I’ve had developers crash databases before and delete data. If they’re databases I manage, there’s a backup. I’ve never been in the situation where a database I was managing didn’t have some backup. However, I’ve had clients and coworkers managing databases with no backups. Perhaps the situation I felt lots of empathy for was a client that had developers making backups before they deployed updates to a client database. One day an update didn’t work correctly, and it asn’t until a day later they realized. At that point, they couldn’t restore the backup. They called me, only to have me inform them that they were actually making striped backups. Unfortunately, they had deleted one of the stripes, and we couldn’t find it. Data loss, that fortunately didn’t kill the company, but it did cost quite a bit.
There are any number of ways that a person can make a mistake on their first day. To prevent this, a good process should limit the damage that an individual can cause because of ignorance. This might be especially important for anyone that might have access to production data.
The Voice of the DBA Podcast
Listen to the MP3 Audio ( 4.1MB) podcast or subscribe to the feed at iTunes and Libsyn.