Matt Gurney: The Rogers failure wasn't what you thought it was
Too much of our analysis and commentary about the major outage has missed the mark — this was bigger than one telecom's bad day.
A note from the editors: After a two-week vacation, The Line is resuming operations. It’s great to be back. We hope you all had a chance to get rested up, as well. Our summer publication schedule, barring news events that demand coverage, will be a bit lighter than normal, but we are accepting pitches for freelance articles, and plan to resume our weekly dispatches. Thanks for being part of our efforts here at The Line.
By: Matt Gurney
Most of the conclusions reached after the Rogers telecommunication outage two weeks ago are wrong. Millions of people lost home internet, television and cellphone service for the better part of a day (some for much longer). For those who had all their services bundled with Rogers, this meant being entirely cut off, including from access to emergency services. It was a big deal, both in terms of lost economic productivity and for those Canadians who needed help and could not access it.
The problem isn’t with Rogers, though. The problem is with everyone else.
I don’t want to be misunderstood. Rogers is bad. It did have a big problem. I am not a fan. Their customer service is generally awful. Their reliability and performance is decidedly meh and the meh costs a fortune. So don’t take this column as some sort of apologia for Rogers. I am one of their customers, but I’m only one of their customers because none of the other options are much better.
But still. The lesson of two Fridays ago shouldn’t be that Rogers is bad. It also shouldn’t be that the CRTC is bad or that our politicians are spineless and that our regulators are thoroughly captured. All of those things are true, but they’re not the lesson. That wasn’t the failure of two Fridays ago. The failure of two Fridays ago was that when one of our telecom companies went down, a pretty horrifying cross-section of Canadian society had no back-up plan.
Let’s imagine an alternate universe where things in Canada simply functioned better. Close your eyes and just dream it up. You're in a different Canada now. The CRTC is awesome. Our politicians are terrific. Rogers is an incredibly good company that is masterful at delivering services that are overwhelmingly reliable and affordably priced. Even in this increasingly far-fetched parallel timeline, no telecom company is going to bat a thousand. You will never have 100 per cent service reliability. This alternate Canada still has outages — maybe they're rare and brief, but they're not unheard of or impossible.
And that’s why we can’t look at what happened two weeks ago as a failure at Rogers. Obviously Rogers failed. But the real failure was a failure of imagination and planning on the behalf of millions of individuals, and a worryingly diverse set of institutions, that did not have a back-up plan.
I host a radio show for two hours every weekday. Even before the pandemic I was operating out of a home studio. Since the pandemic hit, everyone else has basically caught up with me in that regard — some studios are beginning to re-open now, but a big, big part of broadcasting in Canada has shifted to remote work, and may not ever shift fully back. Getting my show on the air every day requires at least five successful links in a telecommunication chain that brings my voice from the microphone on my desk to the listener's ear. I also do irregular but semi-frequent television broadcasts from my home. Because of this, having redundant communication backups is just (literally) a cost of doing business. I have very deliberately set up my communication services to involve a variety of service providers. My home internet is on Bell, and functioned throughout the recent outage. My cellphone (and my wife's) are on Rogers, and we both lost service. I also have a Rogers mobile hub that uses the cellular network to provide internet connectivity if ever my main Bell connection goes down, which happens from time to time.
This degree of redundancy probably costs me about $150 a month on top of what I'd pay for the services. Do I enjoy spending that extra $150? Of course not. But I take my work seriously, and in my line of work, being reliably online, every day, isn't optional. It's the table stakes. So the extra bucks are a business expense, and I just have to eat the cost. (As I eat the expense of having enough backup power to do my show even with the electricity out, and that's happened more often than I'd care to recount.)
Can someone please tell me why I apparently have put more thought into keeping my daily two-hour current events chat operating and online than Interac has given to keeping the Canadian economy online?
Interac, of course, is the electronic payments processor that powers Canadian ATMs and debit terminals, as well as online e-transfers of money. Or, as they humbly describe themselves on their own website, they are the “foundation of Canada's payments system." This may be boastful, but it's not inaccurate: Interac handles tens of millions of financial transactions each day.
You know, when it’s working.
Interac relies on Rogers. When Rogers went down, Interac went down, too. I was out and about running errands that day and came across numerous handwritten signs tacked up on doors and windows of businesses, warning customers that only cash could be accepted. (Some businesses were still able to take credit card payments, depending on which service they were using. I don’t honestly have a sense of how well the credit card networks held up, but it seemed to me to be about a 50-50 thing, for whatever that’s worth.) This wasn’t a problem for me. I almost never use cash, but I do keep some handy exactly for emergencies like this. (A holdover from my 2003 blackout experience.) Most people don’t these days. Two Fridays ago, that showed.
I honestly don’t know how much business was lost due to the literal inability of customers to use Interac to pay for goods and services and for businesses to accept the payment. Indeed, with the ATMs down across much of the country, you couldn’t even pull out some cash to pay the old-fashioned way, unless you had a bank branch nearby you could go to directly in person. (I’ve heard from friends that the lineups at some of them in my neck of the woods were quite long.)
The point here isn’t to pick on Interac, although, to be blunt, they deserve it. This company runs a critical component of the Canadian economy. They are absolutely essential to our national economic infrastructure. Did it occur to no one to put together some sort of emergency plan to temporarily switch over to other providers? Apparently not! The company has already said in a statement that its executive team is immediately working to "[add] supplier diversity to strengthen our existing network redundancy so Canadians can continue to rely on our services daily" and "[strengthen] our emergency business continuity plans." That's the fancy 21st-century version of fixing the barn door after the horses have escaped, of course, but there you go.
Interac's failure was, first and foremost, a failure of imagination. Interac either hadn't thought about this kind of outage, or hadn't thought about what that kind of outage would mean. The only thing Interac has going for it is that it has plenty of company. On Friday, as I was driving around doing my errands, I was listening to some of the local AM radio stations for news updates, and I heard messages from local hospitals and health-care systems putting out emergency recall notices to their staff. Everyone was being asked to come in to work, and they were relying on radio stations because their own communication networks were down. Hospitals might not be running the national economy, but I suspect most of you won’t fault me for going out on a limb and suggesting they’re kind of important. Why don’t they have back up communication plans? Come to think of it, maybe appealing to radio stations and hoping your staff are tuned-in was the back-up plan.
It’s fun to beat up on the big, bad telecom company. God knows I indulge in a bit of that myself from time to time. The CRTC is an easy and deserving target, too. Our feckless politicians deserve all the legitimate criticism we can hurl at them.
But I think we’re missing the mark on this one. The Rogers failure exposed a whole other layer of failure, and I would argue a more worrying one. A frightening amount of this country had no back-up plan in the event of a communications disruption that was entirely foreseeable. Indeed, these things have happened before.
But our hospitals didn’t have plans. Our first-response agencies didn’t have plans. The electronic payments processor that is the backbone of our economy didn’t have plans. And millions of Canadians didn’t, either.
This can’t surprise you. The last two years have certainly been a reminder that Canadians, as individuals and inside institutions, tend to look at emergency planning as something to be ignored, and that only weirdos worry about. When forced to explain what their emergency plan is, even for fairly routine and mundane events, the answer seems to be, more often than not, a helpless shrug and maybe the raising of crossed fingers. Perhaps followed by some snide wisecrack about not being worried about zombies.
Forget zombies. I’m not asking for miracles, or some top-to-bottom overhaul of how we live and work in this country. I’m just asking for vital local and national institutions to work as hard and think as deeply about keeping their services running in the event of telecommunication outages as I have worked and thought about keeping my daily gabfest up and running even when landscaping guys accidentally destroy the fibre-optic cable that brings my Bell internet service into my house, again.
I understand that not every individual can afford the kind of redundant links that I can. All of us have to do what we can to assess our needs and address any possible shortcomings. It is good advice, in general terms, to always take reasonable steps to be prepared for reasonably foreseeable emergencies and inconveniences, but not everyone can spend the time and money, and that’s life. It is what it is. But even individual examples of foresight and resiliency don’t matter in the face of institution-level failures. There is only so much each of us can do for ourselves.
It seems to me that the challenge of the 2020s in this country is simply getting Canadians to begin expecting, and demanding fairly low levels of basic competency. Excellence can wait. The goal right now is much more modest than that: let's achieve basic levels of competency, and take basic steps to provide basic levels of system redundancy in our vital services, especially as regards fairly low-level and foreseeable threats. This is not too much to ask for.
Or it shouldn't be. But it is, somehow, isn't it?
The Line is Canada’s last, best hope for irreverent commentary. We reject bullshit. We love lively writing. Please consider supporting us by subscribing. Follow us on Twitter @the_lineca. Fight with us on Facebook. Pitch us something: lineeditor@protonmail.com
I think what a lot of this comes down to is cost. We've spent a generation 'optimizing' every system to squeeze out cost and emergency planning is inherently a cost with a once in a blue moon payback. We've similarly optimized supply chains and healthcare so we can continue to enjoy lower prices and lower taxes.
I agree that we should put some of that cost back in the system, but history seems to prove that Canadians are cheap and don't like paying for things that don't provide immediate gratification. Moreover, most (but not all -- see the healthcare system) of the failures have been inconvenient to most people which doesn't provide much motivation to change behavior.
The problem is us and our sense of entitlement to new stuff for cheap!
"Can someone please tell me why I apparently have put more thought into keeping my daily two-hour current events chat operating and online than Interac...?"
Because you have no Accounting department. Large companies have specialists that cut expenses, professionally. Their pressure is relentless. Where professional engineers are able to intercede, state that there are standards, unacceptable risks, Accounting can be fought off. In a large decision, involving many millions, the engineer MUST to have government laws/regulations to point to, or you will be overruled. Often they'll spend years overruling you, chopping away at the defenses bit by bit, but they get there.
"A failure of imagination, not of calculation", oddly, was my much-repeated line at the military conference I gave my "Titanic" talk at. They just didn't imagine the need for so many watertight compartments, for a double-hull - as had been on the earliest liners. Every generation of ships had fewer redundancies, less resiliency, more economic efficiency. From the "Great Eastern", with 50 watertight subdivisions, until you got the Titanic, with six. (Took sixty years. Accountants never sleep.) Then, the Titanic stimulated new REGULATIONS for watertight compartments, ships had to be forced to those designs, or economic pressures would relentlessly have made them more efficient, and less safe, again, in a few generations.
It's always that way: name a car safety feature that became universal because of free-market competition where customers always chose safety over savings. Never happened. They all had to be mandated. Computer network safety features are the same way: an efficient company will always beat a resilient company, except on that five-thousandth day; by then, the resilient companies are broke.
Matt's $150/month doesn't render him unable to out-compete other 'casters; but in a neck-and-neck competition for investors, a company with 7% profits, pays investors 14% more for their investment than one that makes 6%. If they can do that legally, then anything they can legally cut, will be, no matter how "prudent" your risk-managers say the extra network is.
http://www.cuug.ab.ca/branderr/titanic/