What do you do when something goes wrong; when something goes wrong in an obvious and public way?
Should you tell your users that you’re dealing with it, even if at that point you have no idea what’s causing the problem? Or do you just buckle down and try to fix it as soon as possible, hoping that the outage has affected as few people as possible and that nobody really noticed anyway?
Personally, I’m a fan of getting the bad news out there and owning it. It’s something that was confirmed to me recently, once at work and again while trying to use the internet at home.
Case study #1: me
I started writing this post last week, but I ran out of time and saved it to draft to complete later. This week I was upgrading a few WordPress plugins and themes only to discover that the latest update (v.1.2.1) to the Blackbird theme broke four site homepages.
What to do: fix it in the hope that nobody noticed or publicize my foul-up and let everyone know there was an issue. It was at that point I remembered this post and took my own advice:
Twenty minutes later I tweeted the good news that I’d fixed the issue and a link to the solution in case other people encountered the same thing.
This latter post was retweeted once, by our own IT Service Desk.
Case study #2: BT
A couple of Saturdays ago, while I was sitting at my desk at home, one of my five year old boys popped his head into the study and said “something has happened to the big telly”. I hurried downstairs and was rather relived to discover that he simply meant they couldn’t access Netflix. Panic over.
After a bit of investigation I discovered that while there was a broadband connection, there were certain sites that our smart TV couldn’t access, including Netflix and the manufacturer’s site (to check for a firmware upgrade). I retired to my study to investigate further.
On my PC I discovered that there too I was only able to access certain sites; my mobile phone was doing the same, plus my wife’s tablet. Google was fine, Amazon wasn’t; I could access Facebook, but not Twitter. This was when I began to suspect the issue was to do with DNS, the system that maps domain names to the servers they are stored on (e.g. www.example.com maps to the IP address 188.8.131.52).
I tried to access a few sites directly by their IP address, and I could. Both Amazon and Twitter were available that way, so I simply changed my DNS settings to use Google Public DNS (184.108.40.206 and 220.127.116.11) and suddenly everything returned.
My internet service provider (ISP) is British Telecom (BT), so I checked their business service status updates page: there was nothing about this. I contacted @BTCare, the support channel that BT runs on Twitter. I heard nothing back.
I searched Twitter and there “BT DNS problem” was all but trending on the social network. So I put a query out on Facebook too and very quickly friends in Devon and East Anglia also reported that they couldn’t connect via BT (but could via their mobile network).
It was beginning to look like a UK-wide incident, but still there was no word from BT about it on any of the channels that I checked. It wasn’t until a couple of hours later that I checked the BBC website and read BT apologises for broadband problem.
@BTCare eventually posted an apology too
The feedback was interesting and not altogether unsurprising with many users saying that they had spent an hour or more trying to diagnose the incident, and others simply saying that they wished that BT had let people know sooner that there even was a problem.
In their first two books 37Signals (now Basecamp) advise that you own your bad news:
When something goes wrong, someone is going to tell the story. You’ll be better off if it’s you. Otherwise, you create an opportunity for rumours, hearsay, and false information to spread. (Rework, Vermillion, 2010, p.231)
They advocate openness, honesty and transparency. Don’t keep secrets, they say, or hide behind spin. “Customers are usually happy to give you a little bit of breathing room as long as they know you’re being honest with them.”
I think that’s pretty solid advice, to be honest.