So picture this. It’s first thing in the morning at work. You receive a request to update some values in a table in production. It’s a fairly static look up table. It seems easy enough. You write the script to update the values and then without pausing to think if it’s a good idea… you run it… in production… without testing it first or asking someone to look at it… only to then realize you forgot… *dun dun dun*… the WHERE clause… You just updated Every. Single. Row. In. Production. Yep! This was me one recent morning at work before I had inhaled my morning dose of wake-up-juice-so-I-don’t-screw-it-all-up (a.k.a. caffeine).
Fighting the urge to descend into sheer panic mode, I
awkwardly quickly told my co-worker that I just f’d up our main production database what had happened and begged / pleaded / groveled asked for help in fixing it fast (note: he’s super fast at fixing oops)! I also asked our resident bad ass Business Analyst if any users had so happened to have made any more changes this morning ‘cuz I mayhavetorestorethetablebecauseIjustscrewedup. Thankfully, she didn’t bat an eye and confirmed back within what felt like an eternity a few minutes that no one had made any more changes. We managed to fix it in what felt like an infinite pause in space and time record time! Whew! I then explained to our oh-so wonderful and patient Business Analyst (BA = bad ass) what happened and we all had a good chuckle. Thankfully I had realized what I had done fairly quickly before it became that much more complicated to fix.
The lesson here, folks, is to:
- never run a change in production without asking someone to verify your code first OR at least test it in non-production BEFORE running it in production
- make sure you’re sufficiently caffeinated before you even
breathe near or look at or sneeze aroundtouch production
- we all make mistakes.. senior / lead DBAs are *gasp* human too!
Despite the fact that I have over twenty years in IT (ten plus as a DBA), we all make mistakes from time to time. It’s nothing to be ashamed of as long as we learn from it and hopefully never make the same mistake twice… or thrice… or whatever. We are not alone! 🙂