I think what a lot of this comes down to is cost. We've spent a generation 'optimizing' every system to squeeze out cost and emergency planning is inherently a cost with a once in a blue moon payback. We've similarly optimized supply chains and healthcare so we can continue to enjoy lower prices and lower taxes.
I agree that we should put some of that cost back in the system, but history seems to prove that Canadians are cheap and don't like paying for things that don't provide immediate gratification. Moreover, most (but not all -- see the healthcare system) of the failures have been inconvenient to most people which doesn't provide much motivation to change behavior.
The problem is us and our sense of entitlement to new stuff for cheap!
There's an opportunity cost to security, which people - not just Canadians - fail to realize. Hence supply chain breakdowns because nobody ever checked to make sure supply chains did not converge on one country, etc., etc.
Let's not give too much credit to these organizations. It's not like they all deliver basic and not fully reliable service at low cost - that are often expensive by global standards. It's just that cutting margins of safety is much easier than reorganizing to improve productivity.
Many years ago we *had* a medium sized trucking company as a customer for Internet. It was about an $90 a month service. We suggested to them getting some sort of backup in case a poll got hit / the line cut etc. They declined. Sure enough, line got cut in a traffic accident and the customer after being down for 8hrs was yelling and screaming how of the 7 ISPs she had in the past (always a telling sign), we were the worst! They lost thousands of dollars blah blah blah... They probably spent more a month of their door mat cleaning service than protecting their business critical connection. Some people really are penny wise, dollar foolish, aka cheap.
You are correct but it's not just Canadian companies, after the 2001 crisis communications countries around North America, where the regulatory foot print is pretty light, dramatically reduced spending on network integrity. They have started to invest more in their networks but the underspending is going to take a lot more investment.
"Can someone please tell me why I apparently have put more thought into keeping my daily two-hour current events chat operating and online than Interac...?"
Because you have no Accounting department. Large companies have specialists that cut expenses, professionally. Their pressure is relentless. Where professional engineers are able to intercede, state that there are standards, unacceptable risks, Accounting can be fought off. In a large decision, involving many millions, the engineer MUST to have government laws/regulations to point to, or you will be overruled. Often they'll spend years overruling you, chopping away at the defenses bit by bit, but they get there.
"A failure of imagination, not of calculation", oddly, was my much-repeated line at the military conference I gave my "Titanic" talk at. They just didn't imagine the need for so many watertight compartments, for a double-hull - as had been on the earliest liners. Every generation of ships had fewer redundancies, less resiliency, more economic efficiency. From the "Great Eastern", with 50 watertight subdivisions, until you got the Titanic, with six. (Took sixty years. Accountants never sleep.) Then, the Titanic stimulated new REGULATIONS for watertight compartments, ships had to be forced to those designs, or economic pressures would relentlessly have made them more efficient, and less safe, again, in a few generations.
It's always that way: name a car safety feature that became universal because of free-market competition where customers always chose safety over savings. Never happened. They all had to be mandated. Computer network safety features are the same way: an efficient company will always beat a resilient company, except on that five-thousandth day; by then, the resilient companies are broke.
Matt's $150/month doesn't render him unable to out-compete other 'casters; but in a neck-and-neck competition for investors, a company with 7% profits, pays investors 14% more for their investment than one that makes 6%. If they can do that legally, then anything they can legally cut, will be, no matter how "prudent" your risk-managers say the extra network is.
Forgot to add: did you see any water or sewer outages? Electrical power? No.
All well - those industries that have long learned the hard way about resiliency, have tough government regulations, and voluntary industry standards (the latter only possible for monopolies; with competition, "voluntary" is cut), so they kept up service.
Hospitals are still learning that IT is now as essential as power, that they can't "go back to paper" like they could in 2005. But crap like this, and ransomware, are rapidly teaching them about backup IT. Alas, they have been absolutely hammered by resource issues for two years, and they'll be another two, fighting alligators of staff-shortages, before they can turn to draining the IT swamp.
I read your Titanic essay - it's very good and thought-provoking. We need more red teams in safety related design, or anything that needs to stay running.
I've been saying this since Friday. Rogers has SLAs that they violated and they'll pay the penalties and the market should take care of it. Bell and Telus reps will displace a lot of rogers volume over the next few months and then we'll all forget. We should be truly appalled that there are critical organizations who have carefully laid all of their eggs in one basket which is, frankly, full of egg sized holes.
Interac is a key operator in one of the most regulated industries in Canada and they don't have a redundant internet provider? That's Obscene. Health Services losing connectivity from a single outage? Criminal negligence. Even if it was difficult to imagine a systemwide failure from rogers prior to last year, this failure was a repeat performance. Anyone who was down as a result of the rogers failure is down because they evaluated their options and cut a corner.
NO ONE planning a critical service that requires connectivity just skips the discussion of redundant systems. These conversations weren't skipped, they were had, and they decided it wasn't important enough for redundant connections.
Rogers will probably lose my personal business, mostly because they seem to think it's "fixed" but i still lose connection for a minute at least every half hour or so. So their definition of fixed and mine are not the same. But they will get punished by the market. The market doesn't currently have a way to punish Interac, or the various critical government services that didn't appropriately plan. We need to deal with THOSE too or we're just rewarding the incompetence that got us here.
25 years in IT, a good chunk of which was at various small data centres. Which, BTW had better redundancy than interac it seems. Each one of my bosses had to fight like hell for that redundancy, it really is all about the bottom line, to me, interac should be under the same regulations as any Canadian financial institution, and government need to focus on ensuring a solid, secure infrastructure, not in bailing out failing broadcast and print companies
Totally agree. I'm surprised actually that the banks didn't require the redundancies from Interac. They might have violated their own requirements by not investigating the emergency plans of their providers.
Redundancy isn't cheap - especially in some of the more remote areas, duplicating your connectivity (and like truly without a single point of failure) can really be a huge burden. But for some businesses and organizations, it's critical.
What if it's a hack next time and it's not a day to get back up? what if it were a few days?
Maybe if we had a proper Canadian Communications Commission reporting to Parliament through ISED that focused on maintaining a dependable, accessible, affordable $70 billion communications system . . . But no, we have the CRTC, reporting through Heritage that only cares about Cancon in a $19 billion broadcasting industry and only appoints leaders with broadcasting experience. Not that there’s anything wrong with broadcasters, of course.
Well they are supposed to report through ISED on the telecom Act and Heritage on the broadcasting Act but Heritage always seems to take the lead on selecting the chair (and Pablo is certainly doing that) and Commissioners. Even the Broadcasting the Telecom Legislature Review panel reported to Heritage which, of course, picked up on a couple of the most problematic Broadcasting suggestions by declaring the Internet to be broadcasting and completely ignoring telecom because , I assume, it's hard and not a whole lot of fun
I particularly appreciated this commentary. Having worked in public service for many years, working on redundancy is a key part of risk management. Unfortunately redundancy costs something as your personal example shows. In business it is the cost of doing business.
In a hospital that is trying to deal with staff shortages and overcrowding, redundancy starts to slip down the cost priorities. Lord help anyone who suggest increasing revenues to address the issues. The cacophony of voices that arise to cut this service or that service at the mere suggestion of increased revenues makes it difficult since most suggestions are for services that the individual or business is asking to be cut is not a service that they use or access.
I also agree that we all need to build redundancy where we can, individually, locally, nationally in business or public sector. We need to recognize at the same time that there are many Canadians that can't afford to. And so the redundancy has to be seen as a public good and not necessarily an individual one. A public policy problem if I've ever seen one.
The idea of a public good seems to have been replaced with the idea that, if something isn’t economically viable, now, it’s struck off the list, so to speak. Focus is on spending, in the public sphere, because investment is not politically attractive to the voters, who vote for what’s in it for themselves, by and large. Or, perhaps, the concept of investing is not understood, if I am being kind. ‘Pay now or pay later’ is a fact of life.
I would be interested to know about what tens of billions are being wasted? Deciding what money to spend on public services is about choices, priorities and values. I would honestly like to know what tens of billions we as a country should stop spending on.
I agree that governments don't know best and I certainly have no reason to think you're evil. What I am interested in is what can we suggest to government is a good way to change what they spend on, keeping in mind that generally the reasons things get funded probably made sense at a time and a place. I really don't understand how the US Government works so I can't really comment on what they spent the resources on.
I didn't realize you were speaking about the money spent these past two years to help individuals and businesses survive the effects of the economic disruption of the pandemic. As I understand it, back in March 2020, as you know, the governments (federal and provincial) decided to shut down the country given the unknowns of the COVID-19 virus and the reports of fatality rates such as was happening near the end of February in Northern Italy and New York City at the beginning of March.
I think that the government decided that Canadians were going to be drastically affected by the sudden loss of income to individuals as the entire economy shutdown and introduced in period of about three or four weeks a massive program called the CERB (the Canadian Emergency Response Benefit). They also heard the call of other affected groups who would not qualify for CERB such as students and pensioners. They also introduced a couple of months later a program to have businesses retain employees with a wage subsidy. As I recall the idea was that keeping people employed was going to make it easier to get people back to work once the economy restarted.
The programs were certainly imperfect and many individuals that got the CERB and who in fact were not eligible are now seeing those monies being clawed back which of course is causing news. I have not heard much on the business side other some clearly undeserving businesses being revealed by media.
The Public Accounts of Canada reveal the actual spend in 2020-21 for the CERB and I attach the web link here for your reference. When I looked this up I found it interesting to see all the regular programs that Employment and Skills Development Canada, there are a lot of them. Although I also get the purpose behind virtually all of them even though I would likely never access a number of them. https://www.tpsgc-pwgsc.gc.ca/recgen/cpc-pac/2021/vol2/edip-ewddi/pt-tp-eng.html
Of course the Public Accounts for 2021-22 should be out in the fall as the books are closed off and audited for the fiscal year that ended on March 31, 2022.
Anyway, I guess my point is that the emergency programs were massive in scale but they are time limited. The government will need to find permanent monies to reduce. and it would be great if we could explore that.
I spent much of my career providing critical systems to major communications companies around the world including most of the Canadian ones. What happened at Rogers (a failed upgrade of their core network) shouldn't have happened but it wasn't a surprise. These companies have drifted away from the core mission, providing essential communication solutions to Canada, to value added content (i.e. the NHL) in order to drive more revenue. As a consequence, comms companies have been underspending on their networks since 2001 and have been driving more spending on revenue generation than network integrity. Like it or not they are more like Electric & water companies (which take delivering their services as an existential one) than entertainment companies and the CRTC should take it's regulatory role a lot more seriously. I'm not sure if this means they have to start divesting these assets/deals but the CRTC should demand a minimum up time rate and take customer sat a lot more seriously. This is going to take a lot more than a 5 day credit to fix this issue.
I am a retired CA, part of the audit process every year is critical risk assessment. I think the risk assessments were obviously wrong when it came to Rogers, Interac and a few other places. Clearly this should have been addressed much better, BUT the auditors should have also flagged this...You can be sure NEXT year, after it has occurred, it will be better addressed! Something that caused this much hardship to in one day clearly needs to be addressed from the auditors standpoint too.
The auditors might well have done that but there’s no reason to truly believe that companies will follow through if they think there’s ‘no today’ benefit and their Board is negligent. Short term gain, long term pain.
Rogers went dark for a day April 2021. It was an Ericsson software update that affected 11 million users/customers. This time it was a router? Our telecoms are too big, too thin, and too profitable.
We are a penny wise and pound foolish people. We just are, it's cultural. We hate opening our wallets to pay for nicer things, look at our health care system for instance, perhaps because this country is a series of protected guilds and interest groups squeezing people. We still have a colonial economic culture in many ways. I sometimes joke we have such high immigrant numbers because we need more suckers for our protected economy to rip off.
This has been going on too long, it's embarrassing. No, the Americans aren't going to take us over and Quebec won't seperate if we as Canadians push back against our mediocre elite and their featherbedding. But first we need to address our culture and stop behaving like colonials with such low expectations.
I agree with the article but still am interested in what brought both the wireless and wired service down at Rogers and what Interac is doing to make sure they stay online in the future. With all the recent turmoil at Rogers and some rumours their technical people have been leaving I would like to know what exactly happened. As someone who worked in IT in the past, I know that often there will be a small number of people who REALLY know the system. Most technical people are competent but when something really weird knocks the whole system offline there will generally be that one person you need to have look at it. If you can’t get a hold of them or they have left the company you then have to muddle through until you find the problem.
Rogers just finished pushing Joe Natale out as CEO, an Electrical Engineer, to be replaced by Tony Staffieri, an Accountant. Perhaps many senior technical people saw the writing on the wall and left. Company culture comes from the top down after all.
Rogers needs to be held accountable. In the last 12 months there has been nothing but drama at Board level. Edward Rogers replaced an engineer with former Head of Accounting with goal to save money. There are a whole lot of other companies who need to do better - but 1st let’s acknowledge that Rogers did an unbelievably poor job. They should not be allowed to control more of the Canadian telecommunications market through purchase of Shaw.
Accountable for the drama? We can expect any "improvements" to cost us and the other guys will up their convoluted plans too.
I doubt that Telus, Bell, and Shaw and the little guys are any more prepared than Rogers. Rogers was out for day in April 2021. Hardly a boo to be heard.
Natale and Staffieri are more than an engineer and the head of accounting. What money was saved by putting Staffieri in place? You're simplification makes it sound like they actually get their hands dirty, or at least clutch a pencil with an eraser to chew on. Staffieri was CFO, Ex VP before taking on the Pres. & CEO job. He's also on a mess of boards; Chair at Ryerson, Maple Leaf Sports, and Mackenzie Health Foundation.
Natale is much the same. Pres. & CEO of Telus, on the board of Sick Kids & Home Capital Group (mortgages), senior leadership roles within KPMG Consulting.
So replacing an engineer with an accountant is a silly statement.
What they do now is apologise. The engineer left Rogers with over $14 million severance. I agree, Rogers should not get Shaw.
At this point, I think you have enough material for a lengthy book (seriously) at what an utter basket case Canada has become, and the potential (probable) implications of the myriad of sheer ineptitude on display. I'll even provide you a title - Woe, Canada - We Lean, Off Guard, For Me.
One asks (as you repeatedly have) what productive, tangible things CAN Canada actually do at a world class level at this stage of our history?
The list of answers is embarrasingly brief - in fact, I can't think of a single thing.
In my half century living in Canada, I've watched public and private standards slowly but very steadily slipping away in every aspect of Canadian life. Mediocrity has become so acceptable, risk aversion so ingrained, entitlement so galling, that the majority of Canadians are now conditioned to accept that This Is Just The Way Things Are - meaning, they shrug their shoulders - and that's it! No outrage, no demands for better - just bland, indifferent, depressing acceptance for the declining state of affairs.
Evidence is everywhere - in tangible things like decaying roads and bridges, unreliable and overpriced utility and communication networks, and even overflowing garbages in public and private spaces. Canadian tolerance of galling, lazy ineptitude and outright, obvious corruption in public politicians and bureaucrats from the municipal through provincial to the federal levels is, actually, incredible.
We have so many in public and private life (elected, appointed, or 'employed') that have made careers out of accomplishing absolutely nothing. Instead of making things better, they have succeeded at failure so consistently that their legacy is making almost every aspect of Canadian life worse.
I continue to search in vain for a way out of this quicksand Canadians are all caught in.
There ARE good people in this country ready to change our national character from one I describe above to one Canadians can actually be proud of. The problem is, many of them are getting discouraged, or (most distressingly) are making active plans to leave Canada and never return.
If you do write a book, Matt - perhaps that can be the second part of it - after you detail the overwhelming rot, perhaps you can offer a plan, a prescription, a panacea for what ails Canada?
my company does work with many payment processors as well as loyalty card processors. Even the smaller ones try to have some sort of carrier independence. I really, really want to hear how Interac justified NOT bringing in multiple physical carriers. I have no inside knowledge on it so this is all pure speculation.
Worst case, total cheap incompetence on the banks who oversea it.
Best case, their data center (why just one?!?) was in a place where Rogers was the only company actually with fiber. Interac could have gone to Bell and actually purchased "Bell fiber" but it was actually delivered over Rogers infrastructure and not dark fiber but handed off at some CNI and Interact was none the wiser. We ( I work at an independent ISP) have ckts like this so its not unheard of, but generally someone that large would ask to see how much physical path redundancy there is as part of their due diligence.
As for hospitals and such, my guess is cost drivers and not having the technical staff. Its easier now, to provide "point and click" redundancy through technologies like SDWAN and more flexible VPN tecjnologies), but not that long ago having true carrier redundancy meant some somewhat specific technical knowledge (e.g. running your own ASN) which most orgs like hospitals would not have. Its one thing to have your home on multiple connections, its another where you are a large org with services (e.g. VPNs) tied to static IP addresses that people / other orgs connect TO. To have your own IP space (i.e. independent of the carriers you connect to) requires fairly esoteric skill sets. A hospital would be too small to have such technical skill among their staff so its gonna be harder. Like I said, this is getting a LOT easier but I suspect they are not there yet.
The cost benefit analysis of redundancy is completely different for people and companies with competition and those without. If you are offline enough, your customers will go elsewhere. If a hospital is offline, do they suffer? No.
If there wasn’t a fire ban on, I would start using smoke signals to contact my family. Trying to find out how to send a middle finger using smoke. Tricky stuff.
To paraphrase a joke from Bill Maher: I don't mind Canada, its just the Canadians I can't stand....
We, Canadians, collectively, are naive, complacent, lazy, and docile. Our national motto should be changed from 'A Mari usque ad Mare' to 'Oh Well, What Can You Do?'
I have some mixed feelings about the Truckers Convoy, but one thing that was extraordinary was the fact that Canadians actually got pissed enough to *disrupt* something, cause a scene, and be....not very nice. THE HORROR! But a nation-wide, commerce-disrupting, emergency services-threatening outage from a oligopoly telecom player, regulated by the government?
Riggghht. The convoy was a clown line, filled with determined stupidity, shouted slogans from Q blogs, Plandemic and various fascist racist twits, all while claiming they were doing it all for our freedoms. Some of these these dipshits would label me a marxist. Oh right, add some fundy Christian blather to tie it together and the future looks dim.
I think what a lot of this comes down to is cost. We've spent a generation 'optimizing' every system to squeeze out cost and emergency planning is inherently a cost with a once in a blue moon payback. We've similarly optimized supply chains and healthcare so we can continue to enjoy lower prices and lower taxes.
I agree that we should put some of that cost back in the system, but history seems to prove that Canadians are cheap and don't like paying for things that don't provide immediate gratification. Moreover, most (but not all -- see the healthcare system) of the failures have been inconvenient to most people which doesn't provide much motivation to change behavior.
The problem is us and our sense of entitlement to new stuff for cheap!
Agree with every word of this.
There's an opportunity cost to security, which people - not just Canadians - fail to realize. Hence supply chain breakdowns because nobody ever checked to make sure supply chains did not converge on one country, etc., etc.
Let's not give too much credit to these organizations. It's not like they all deliver basic and not fully reliable service at low cost - that are often expensive by global standards. It's just that cutting margins of safety is much easier than reorganizing to improve productivity.
And "stakeholders" complain less.
Many years ago we *had* a medium sized trucking company as a customer for Internet. It was about an $90 a month service. We suggested to them getting some sort of backup in case a poll got hit / the line cut etc. They declined. Sure enough, line got cut in a traffic accident and the customer after being down for 8hrs was yelling and screaming how of the 7 ISPs she had in the past (always a telling sign), we were the worst! They lost thousands of dollars blah blah blah... They probably spent more a month of their door mat cleaning service than protecting their business critical connection. Some people really are penny wise, dollar foolish, aka cheap.
You are correct but it's not just Canadian companies, after the 2001 crisis communications countries around North America, where the regulatory foot print is pretty light, dramatically reduced spending on network integrity. They have started to invest more in their networks but the underspending is going to take a lot more investment.
"Can someone please tell me why I apparently have put more thought into keeping my daily two-hour current events chat operating and online than Interac...?"
Because you have no Accounting department. Large companies have specialists that cut expenses, professionally. Their pressure is relentless. Where professional engineers are able to intercede, state that there are standards, unacceptable risks, Accounting can be fought off. In a large decision, involving many millions, the engineer MUST to have government laws/regulations to point to, or you will be overruled. Often they'll spend years overruling you, chopping away at the defenses bit by bit, but they get there.
"A failure of imagination, not of calculation", oddly, was my much-repeated line at the military conference I gave my "Titanic" talk at. They just didn't imagine the need for so many watertight compartments, for a double-hull - as had been on the earliest liners. Every generation of ships had fewer redundancies, less resiliency, more economic efficiency. From the "Great Eastern", with 50 watertight subdivisions, until you got the Titanic, with six. (Took sixty years. Accountants never sleep.) Then, the Titanic stimulated new REGULATIONS for watertight compartments, ships had to be forced to those designs, or economic pressures would relentlessly have made them more efficient, and less safe, again, in a few generations.
It's always that way: name a car safety feature that became universal because of free-market competition where customers always chose safety over savings. Never happened. They all had to be mandated. Computer network safety features are the same way: an efficient company will always beat a resilient company, except on that five-thousandth day; by then, the resilient companies are broke.
Matt's $150/month doesn't render him unable to out-compete other 'casters; but in a neck-and-neck competition for investors, a company with 7% profits, pays investors 14% more for their investment than one that makes 6%. If they can do that legally, then anything they can legally cut, will be, no matter how "prudent" your risk-managers say the extra network is.
http://www.cuug.ab.ca/branderr/titanic/
The electrical engineer Joe Natale was just pushed out earlier this year from Rogers as CEO and replaced by Tony Staffieri, an accountant.
Pushed out with a $14 million golden parachute. Things are tough all over.
Would have been great if he was around the last few weeks.
Forgot to add: did you see any water or sewer outages? Electrical power? No.
All well - those industries that have long learned the hard way about resiliency, have tough government regulations, and voluntary industry standards (the latter only possible for monopolies; with competition, "voluntary" is cut), so they kept up service.
Hospitals are still learning that IT is now as essential as power, that they can't "go back to paper" like they could in 2005. But crap like this, and ransomware, are rapidly teaching them about backup IT. Alas, they have been absolutely hammered by resource issues for two years, and they'll be another two, fighting alligators of staff-shortages, before they can turn to draining the IT swamp.
A thoughtful and thought-provoking commentary. Thank you.
I read your Titanic essay - it's very good and thought-provoking. We need more red teams in safety related design, or anything that needs to stay running.
I've been saying this since Friday. Rogers has SLAs that they violated and they'll pay the penalties and the market should take care of it. Bell and Telus reps will displace a lot of rogers volume over the next few months and then we'll all forget. We should be truly appalled that there are critical organizations who have carefully laid all of their eggs in one basket which is, frankly, full of egg sized holes.
Interac is a key operator in one of the most regulated industries in Canada and they don't have a redundant internet provider? That's Obscene. Health Services losing connectivity from a single outage? Criminal negligence. Even if it was difficult to imagine a systemwide failure from rogers prior to last year, this failure was a repeat performance. Anyone who was down as a result of the rogers failure is down because they evaluated their options and cut a corner.
NO ONE planning a critical service that requires connectivity just skips the discussion of redundant systems. These conversations weren't skipped, they were had, and they decided it wasn't important enough for redundant connections.
Rogers will probably lose my personal business, mostly because they seem to think it's "fixed" but i still lose connection for a minute at least every half hour or so. So their definition of fixed and mine are not the same. But they will get punished by the market. The market doesn't currently have a way to punish Interac, or the various critical government services that didn't appropriately plan. We need to deal with THOSE too or we're just rewarding the incompetence that got us here.
25 years in IT, a good chunk of which was at various small data centres. Which, BTW had better redundancy than interac it seems. Each one of my bosses had to fight like hell for that redundancy, it really is all about the bottom line, to me, interac should be under the same regulations as any Canadian financial institution, and government need to focus on ensuring a solid, secure infrastructure, not in bailing out failing broadcast and print companies
Totally agree. I'm surprised actually that the banks didn't require the redundancies from Interac. They might have violated their own requirements by not investigating the emergency plans of their providers.
Redundancy isn't cheap - especially in some of the more remote areas, duplicating your connectivity (and like truly without a single point of failure) can really be a huge burden. But for some businesses and organizations, it's critical.
What if it's a hack next time and it's not a day to get back up? what if it were a few days?
Maybe if we had a proper Canadian Communications Commission reporting to Parliament through ISED that focused on maintaining a dependable, accessible, affordable $70 billion communications system . . . But no, we have the CRTC, reporting through Heritage that only cares about Cancon in a $19 billion broadcasting industry and only appoints leaders with broadcasting experience. Not that there’s anything wrong with broadcasters, of course.
Best concrete suggestion so far. Separate bodies, to regulate the infrastructure that carries bits, and to regulate the bits themselves.
Why do they report through heritage? Illogical.
Well they are supposed to report through ISED on the telecom Act and Heritage on the broadcasting Act but Heritage always seems to take the lead on selecting the chair (and Pablo is certainly doing that) and Commissioners. Even the Broadcasting the Telecom Legislature Review panel reported to Heritage which, of course, picked up on a couple of the most problematic Broadcasting suggestions by declaring the Internet to be broadcasting and completely ignoring telecom because , I assume, it's hard and not a whole lot of fun
I particularly appreciated this commentary. Having worked in public service for many years, working on redundancy is a key part of risk management. Unfortunately redundancy costs something as your personal example shows. In business it is the cost of doing business.
In a hospital that is trying to deal with staff shortages and overcrowding, redundancy starts to slip down the cost priorities. Lord help anyone who suggest increasing revenues to address the issues. The cacophony of voices that arise to cut this service or that service at the mere suggestion of increased revenues makes it difficult since most suggestions are for services that the individual or business is asking to be cut is not a service that they use or access.
I also agree that we all need to build redundancy where we can, individually, locally, nationally in business or public sector. We need to recognize at the same time that there are many Canadians that can't afford to. And so the redundancy has to be seen as a public good and not necessarily an individual one. A public policy problem if I've ever seen one.
The idea of a public good seems to have been replaced with the idea that, if something isn’t economically viable, now, it’s struck off the list, so to speak. Focus is on spending, in the public sphere, because investment is not politically attractive to the voters, who vote for what’s in it for themselves, by and large. Or, perhaps, the concept of investing is not understood, if I am being kind. ‘Pay now or pay later’ is a fact of life.
I would be interested to know about what tens of billions are being wasted? Deciding what money to spend on public services is about choices, priorities and values. I would honestly like to know what tens of billions we as a country should stop spending on.
I agree that governments don't know best and I certainly have no reason to think you're evil. What I am interested in is what can we suggest to government is a good way to change what they spend on, keeping in mind that generally the reasons things get funded probably made sense at a time and a place. I really don't understand how the US Government works so I can't really comment on what they spent the resources on.
I didn't realize you were speaking about the money spent these past two years to help individuals and businesses survive the effects of the economic disruption of the pandemic. As I understand it, back in March 2020, as you know, the governments (federal and provincial) decided to shut down the country given the unknowns of the COVID-19 virus and the reports of fatality rates such as was happening near the end of February in Northern Italy and New York City at the beginning of March.
I think that the government decided that Canadians were going to be drastically affected by the sudden loss of income to individuals as the entire economy shutdown and introduced in period of about three or four weeks a massive program called the CERB (the Canadian Emergency Response Benefit). They also heard the call of other affected groups who would not qualify for CERB such as students and pensioners. They also introduced a couple of months later a program to have businesses retain employees with a wage subsidy. As I recall the idea was that keeping people employed was going to make it easier to get people back to work once the economy restarted.
The programs were certainly imperfect and many individuals that got the CERB and who in fact were not eligible are now seeing those monies being clawed back which of course is causing news. I have not heard much on the business side other some clearly undeserving businesses being revealed by media.
The Public Accounts of Canada reveal the actual spend in 2020-21 for the CERB and I attach the web link here for your reference. When I looked this up I found it interesting to see all the regular programs that Employment and Skills Development Canada, there are a lot of them. Although I also get the purpose behind virtually all of them even though I would likely never access a number of them. https://www.tpsgc-pwgsc.gc.ca/recgen/cpc-pac/2021/vol2/edip-ewddi/pt-tp-eng.html
Of course the Public Accounts for 2021-22 should be out in the fall as the books are closed off and audited for the fiscal year that ended on March 31, 2022.
Another source for data on the Wage Subsidy is https://www.canada.ca/en/revenue-agency/services/wage-rent-subsidies/emergency-wage-subsidy/cews-statistics.html
Anyway, I guess my point is that the emergency programs were massive in scale but they are time limited. The government will need to find permanent monies to reduce. and it would be great if we could explore that.
I spent much of my career providing critical systems to major communications companies around the world including most of the Canadian ones. What happened at Rogers (a failed upgrade of their core network) shouldn't have happened but it wasn't a surprise. These companies have drifted away from the core mission, providing essential communication solutions to Canada, to value added content (i.e. the NHL) in order to drive more revenue. As a consequence, comms companies have been underspending on their networks since 2001 and have been driving more spending on revenue generation than network integrity. Like it or not they are more like Electric & water companies (which take delivering their services as an existential one) than entertainment companies and the CRTC should take it's regulatory role a lot more seriously. I'm not sure if this means they have to start divesting these assets/deals but the CRTC should demand a minimum up time rate and take customer sat a lot more seriously. This is going to take a lot more than a 5 day credit to fix this issue.
I am a retired CA, part of the audit process every year is critical risk assessment. I think the risk assessments were obviously wrong when it came to Rogers, Interac and a few other places. Clearly this should have been addressed much better, BUT the auditors should have also flagged this...You can be sure NEXT year, after it has occurred, it will be better addressed! Something that caused this much hardship to in one day clearly needs to be addressed from the auditors standpoint too.
The auditors might well have done that but there’s no reason to truly believe that companies will follow through if they think there’s ‘no today’ benefit and their Board is negligent. Short term gain, long term pain.
Rogers went dark for a day April 2021. It was an Ericsson software update that affected 11 million users/customers. This time it was a router? Our telecoms are too big, too thin, and too profitable.
We are a penny wise and pound foolish people. We just are, it's cultural. We hate opening our wallets to pay for nicer things, look at our health care system for instance, perhaps because this country is a series of protected guilds and interest groups squeezing people. We still have a colonial economic culture in many ways. I sometimes joke we have such high immigrant numbers because we need more suckers for our protected economy to rip off.
This has been going on too long, it's embarrassing. No, the Americans aren't going to take us over and Quebec won't seperate if we as Canadians push back against our mediocre elite and their featherbedding. But first we need to address our culture and stop behaving like colonials with such low expectations.
I agree with the article but still am interested in what brought both the wireless and wired service down at Rogers and what Interac is doing to make sure they stay online in the future. With all the recent turmoil at Rogers and some rumours their technical people have been leaving I would like to know what exactly happened. As someone who worked in IT in the past, I know that often there will be a small number of people who REALLY know the system. Most technical people are competent but when something really weird knocks the whole system offline there will generally be that one person you need to have look at it. If you can’t get a hold of them or they have left the company you then have to muddle through until you find the problem.
Rogers just finished pushing Joe Natale out as CEO, an Electrical Engineer, to be replaced by Tony Staffieri, an Accountant. Perhaps many senior technical people saw the writing on the wall and left. Company culture comes from the top down after all.
Welcome back. "Always have a plan B" should be engraved in the eyeline of everyone involved in infrastructure training.
Rogers needs to be held accountable. In the last 12 months there has been nothing but drama at Board level. Edward Rogers replaced an engineer with former Head of Accounting with goal to save money. There are a whole lot of other companies who need to do better - but 1st let’s acknowledge that Rogers did an unbelievably poor job. They should not be allowed to control more of the Canadian telecommunications market through purchase of Shaw.
Accountable for the drama? We can expect any "improvements" to cost us and the other guys will up their convoluted plans too.
I doubt that Telus, Bell, and Shaw and the little guys are any more prepared than Rogers. Rogers was out for day in April 2021. Hardly a boo to be heard.
Natale and Staffieri are more than an engineer and the head of accounting. What money was saved by putting Staffieri in place? You're simplification makes it sound like they actually get their hands dirty, or at least clutch a pencil with an eraser to chew on. Staffieri was CFO, Ex VP before taking on the Pres. & CEO job. He's also on a mess of boards; Chair at Ryerson, Maple Leaf Sports, and Mackenzie Health Foundation.
Natale is much the same. Pres. & CEO of Telus, on the board of Sick Kids & Home Capital Group (mortgages), senior leadership roles within KPMG Consulting.
So replacing an engineer with an accountant is a silly statement.
What they do now is apologise. The engineer left Rogers with over $14 million severance. I agree, Rogers should not get Shaw.
Always appreciate your comments Lou. Glad you agree with comment re takeover of Shaw. That is the most salient point.
Good piece, Matt. Hope you had a good break.
At this point, I think you have enough material for a lengthy book (seriously) at what an utter basket case Canada has become, and the potential (probable) implications of the myriad of sheer ineptitude on display. I'll even provide you a title - Woe, Canada - We Lean, Off Guard, For Me.
One asks (as you repeatedly have) what productive, tangible things CAN Canada actually do at a world class level at this stage of our history?
The list of answers is embarrasingly brief - in fact, I can't think of a single thing.
In my half century living in Canada, I've watched public and private standards slowly but very steadily slipping away in every aspect of Canadian life. Mediocrity has become so acceptable, risk aversion so ingrained, entitlement so galling, that the majority of Canadians are now conditioned to accept that This Is Just The Way Things Are - meaning, they shrug their shoulders - and that's it! No outrage, no demands for better - just bland, indifferent, depressing acceptance for the declining state of affairs.
Evidence is everywhere - in tangible things like decaying roads and bridges, unreliable and overpriced utility and communication networks, and even overflowing garbages in public and private spaces. Canadian tolerance of galling, lazy ineptitude and outright, obvious corruption in public politicians and bureaucrats from the municipal through provincial to the federal levels is, actually, incredible.
We have so many in public and private life (elected, appointed, or 'employed') that have made careers out of accomplishing absolutely nothing. Instead of making things better, they have succeeded at failure so consistently that their legacy is making almost every aspect of Canadian life worse.
I continue to search in vain for a way out of this quicksand Canadians are all caught in.
There ARE good people in this country ready to change our national character from one I describe above to one Canadians can actually be proud of. The problem is, many of them are getting discouraged, or (most distressingly) are making active plans to leave Canada and never return.
If you do write a book, Matt - perhaps that can be the second part of it - after you detail the overwhelming rot, perhaps you can offer a plan, a prescription, a panacea for what ails Canada?
If so, I'll buy it. Book it.
my company does work with many payment processors as well as loyalty card processors. Even the smaller ones try to have some sort of carrier independence. I really, really want to hear how Interac justified NOT bringing in multiple physical carriers. I have no inside knowledge on it so this is all pure speculation.
Worst case, total cheap incompetence on the banks who oversea it.
Best case, their data center (why just one?!?) was in a place where Rogers was the only company actually with fiber. Interac could have gone to Bell and actually purchased "Bell fiber" but it was actually delivered over Rogers infrastructure and not dark fiber but handed off at some CNI and Interact was none the wiser. We ( I work at an independent ISP) have ckts like this so its not unheard of, but generally someone that large would ask to see how much physical path redundancy there is as part of their due diligence.
As for hospitals and such, my guess is cost drivers and not having the technical staff. Its easier now, to provide "point and click" redundancy through technologies like SDWAN and more flexible VPN tecjnologies), but not that long ago having true carrier redundancy meant some somewhat specific technical knowledge (e.g. running your own ASN) which most orgs like hospitals would not have. Its one thing to have your home on multiple connections, its another where you are a large org with services (e.g. VPNs) tied to static IP addresses that people / other orgs connect TO. To have your own IP space (i.e. independent of the carriers you connect to) requires fairly esoteric skill sets. A hospital would be too small to have such technical skill among their staff so its gonna be harder. Like I said, this is getting a LOT easier but I suspect they are not there yet.
The cost benefit analysis of redundancy is completely different for people and companies with competition and those without. If you are offline enough, your customers will go elsewhere. If a hospital is offline, do they suffer? No.
If there wasn’t a fire ban on, I would start using smoke signals to contact my family. Trying to find out how to send a middle finger using smoke. Tricky stuff.
Let us know how you go. There’s got to be a way.
To paraphrase a joke from Bill Maher: I don't mind Canada, its just the Canadians I can't stand....
We, Canadians, collectively, are naive, complacent, lazy, and docile. Our national motto should be changed from 'A Mari usque ad Mare' to 'Oh Well, What Can You Do?'
I have some mixed feelings about the Truckers Convoy, but one thing that was extraordinary was the fact that Canadians actually got pissed enough to *disrupt* something, cause a scene, and be....not very nice. THE HORROR! But a nation-wide, commerce-disrupting, emergency services-threatening outage from a oligopoly telecom player, regulated by the government?
'Oh well, what can you do?'
Riggghht. The convoy was a clown line, filled with determined stupidity, shouted slogans from Q blogs, Plandemic and various fascist racist twits, all while claiming they were doing it all for our freedoms. Some of these these dipshits would label me a marxist. Oh right, add some fundy Christian blather to tie it together and the future looks dim.