Rogers, one of Canada’s major telcos / ISP went down early this morning. It’s having a big impact on internet access, including credit card / Interac payments, phone service for Rogers customers, banking, emergency services …
Not personally much affected. Am puzzled as to why so many systems are involved. No doubt the answers will eventually appear.
Our network and phone lines at the office are down. I am, of course, on vacation in Croatia and there’s not a damn thing I can do about it.
I cant believe this is a measly 3 posts.
This is major stuff. Banking and emergency services are affected throughout the country.
Interac is down throughout Calgary. On the first day of Stampede. I cant be bothered myself, but restaurants and bars across the city rely on these ten days for a major amount of business. This is Canada. Interac (point of sale debit) is the dominant form of payment by a long way. I personally haven’t used cash in years.
All three of us senior managers at work happen to have our personal phones on Rogers. We have a 24/7 business to run. We went out and bought a phone from Telus for the weekend.
One of my clients is a network engineer and suspects this is a result of an attack. No idea how much water that holds.
It’s definitely annoying. My internet (which is Telus, not Rogers or Shaw) has been extra slow, and of course it’s a cash-only day.
More troubling is that there is no sense of why this happened, nor when it will get sorted.
Is the US press covering it at all?
I don’t see it on CNN or Washington Post, at least on a quick look. New York Times:
ETA: gift link
Luckily I’m with Telus for cable, internet and cell phone. Other than having to pay cash at the store we really haven’t been impacted. But it sounds like a real mess for many. I’m amazed that it has been out all day. Anyone have any idea what can cause an outage this serious?
My job noticed the outage, I think it may have been the cause of one of my colleagues being on a call for a large part of the day due to a bunch of DNS queries timing out, earlier - but they were largely working now, and we could explain any that were still a problem.
The low level of news on the subject is pretty surprising. I don’t know if it was an attack or not, but I’d be even more surprised if we don’t get at least a plausible explanation for the outage.
My money is on a botched update or a cyber attack.
I’m in Toronto right now, staying with relatives whose provider is Rogers. My AT&T (US) phone connects to Bell Canada while I’m here, but service is spotty too. More details later when the connection is more reliable!
We went to a restaurant tonight (in Québec), Interac (direct debit) was unusable but credit cards worked, using the same wireless point-of-sale terminals. Later we went shopping at Bureau en Gros (aka Staples), same thing. We saw a couple buying school supplies (in July ??) and paying something like 250$ in cash. 
I hope the banks filled their ATMs’ cash drawers to the brim on Friday; otherwise it’s going to be a tedious weekend for many.
Credit cards do work, apparently. I went down to the corner store for a few things today, and a notice on the door said “No Interac, but Cash, Visa, and Mastercard accepted.” I had my Visa card, so that was no problem. Still, I hope the problem clears up soon, and we find out what happened.
I’d been up in the middle of the night doing some online banking, buying train tickets for an upcoming vacation, etc, and around 4:30 I noticed that I couldn’t send an Interac email transfer from my online banking.
I’d also made a Visa Debit purchase from Amazon; usually those go through immediately and I see it leave my bank account right away. This time, the payment was accepted by Amazon but nothing reached my bank account. I’d bought the train tickets the same way earlier and that had gone through immediately.
Figuring it was scheduled maintenance, I just logged out and went to sleep. (Working evening shift leads to odd hours sometimes.)
I woke up to the outage. None of our cellphones were working; they are all on Rogers. However, our Internet is not on Rogers, and it was fine. Our news came via that internet service.
I was scheduled to go into work later than usual because reasons, so I went out to do some shopping. My first stop, no Interac debit. So I went to my bank’s in-branch ATM and withdrew cash. The rest of my stops were all in cash. All the stores had signs up: no Interac debit; cash or credit (Visa/MC) only.
Interac debit is processed ‘online’ immediately; if there is no communication, it just does not work. Unlike Visa/MasterCard cards, there is no ‘offline’ option to record the transaction and submit it later.
I have gotten so used to just ‘tapping the card’ that it felt odd to turn back to cash. I went to work, turning on my phone at breaks to see whether there was service.
When I returned home, I checked my accounts and the Amazon transaction had appeared. After some hesitation while my phone displayed ‘Rogers-EXT’, my cellphone service reappeared sometime after midnight.
This whole thing shows the importance of having backups and alternate methods to do things. This wasn’t just a merchant’s payment terminal not working because the line was down. Apparently quite major pieces of infrastructure were affected (the connection between Amazon and my bank?), not just personal end-user services.
Yes, service seems to be coming back online, but Rogers doesn’t know why it happened:
We don’t understand how the different levels of redundancy that we build across the network coast to coast have not worked," said Kye Prigg, Rogers’ senior vice-president of access networks and operations, on CBC’s Power & Politics.
The boffins don’t think it was a hacker or an attack:
I have been puzzled for some time that such outages occur and they just don’t hit the news.
Last month there were two big outages that affected the store I work at (along with lots of other people - one of them extended into Canada, too), rendering us cash-or-check only (yes, we still accept personal checks). Hardly a blip in the media even if you looked for it. That was just the POS functions of a major network (cellphones unaffected), but it was still massively disruptive to business and to people attempting to buy stuff.
I don’t get why these outages aren’t bigger news.
It’s also why I keep a cash reserve on hand. Also found out a certain number of our customers carry a check or two for “emergencies”. Backups are a good thing, even if one is accustomed to spending money with technology.
I’m glad I switched from cable internet to fiber optic a couple of weeks ago. But lots of people trying to work remotely at our company were forced to come in to the office or take the day off.