Something you never want to see in your Inbox

For the latest in a (hopefully short) series of Burns screw-ups, I just need to cast my mind back to last Saturday. I’d finished my final on-call weekend at Pythian the weekend before but, in a bid to give my team lead a night’s sleep and take advantage of our different time-zones, I grabbed the pager when I woke up stupidly early on Saturday morning – about 5am, I think.

Pythian have a pretty robust system for handling pages which, for each team, includes the on-call DBA for that team; followed by the on-call DBA for another team that is our backup; followed by our manager; followed by everyone in the company. Each stage lasts 15 minutes and we get new pages every 5.

So, anyway … while I was working on some personal stuff, the on-call DBA for our backup team came on-line, fixing a problem. We had an IM chat when he was done and I suggested I pick up his pager too, so he could get some sleep. (If you didn’t notice the first stupid mistake at this point, you’re not concentrating.) He was very grateful, but he’s done us plenty of favours in the past.

I carried on fixing a couple of minor problems, working on a few personal things and when I was finished, I decided to lie down on the settee to watch some TV, knowing I only had about an hour more on the pager, then I’d hand it back to the guys in Ottawa. (Surely by now you can spot the second big mistake?)

A couple of moments later (because that’s how it seemed) my mobile phone woke me up with a text message alert – a page. Mmmm, except, when I went to look at my monitor, this was what was sitting in my Inbox.

!Fully! Escalated Page for Client X

Well, the text wasn’t really that big in Thunderbird, but it seemed that big! Oh, and there were 45 minutes worth of other pages, building up to the big moment! I think my first two words when I spoke to my (now-awake) team lead on IM were – ‘Arrrgghhh!’ and ‘Sorry!’. Why was he awake? Well, it wasn’t because he’d been paged – I’d been paged. It wasn’t because the backup DBA had called him to wake him up – I was the backup DBA (twit!). No, it was because one of the other on-call DBAs, on being woken up, had called the guy who was supposed to be on-call – my team lead.

That !Fully! Escalated means everyone in the company 🙁 You can only imagine the apology email …

Actually, people were OK about it and it wasn’t a serious problem we were being paged for, but lessons learned,

• Don’t be your own backup.
• Don’t try to be helpful if you’re tired yourself.
• Make sure your pager or text alert is so loud it wakes the neighbours. (Who can then wake you, because it’s annoying them.)

Looking on the bright side? Well,

• Alex got a night’s sleep, but probably woke up a bit earlier on Saturday than he was expecting to.
• I owe Raj Thukral 12 pints, but he lives on a different continent so probably can’t call in the debt.
• It won’t happen again – Friday is my last day 😉