Not sure if many dopers know about this, but Microsoft’s email service Outlook has been down for a lot of people today. At my workplace in Colorado, nobody in my ~800 person organization has been able to access their email or calendars for >8 hours. This outage has also affected Teams, the MS collaboration dashboard. We use Teams to interface with the US government and other organizations we do business with. I have know idea if they were also down since we have no communication with them.
My wife actually works for Microsoft and let’s just say she (sort of) had most of the day off. It sounds like there was a hive of activity in Redmond, but working remotely she did not have much contact with the office or her coworkers since everything they do relies on related Microsoft products.
One thing I found interesting was how MS problems affected other companies. If you look at Downdetector.com, it looks like the outage has rippled through Google, Zoom, Amazon, Facebook, Amazon Web Services, and even Reddit. Downdetector claims it only affected ~100,000 people at the peak, but I and everybody I have spoken to believes this is bullshit and the number is much much higher. I have spoken with acquaintances on the west coast who were cut off. Also people on the east coast, people in Europe, and even people in Australia. The news networks seem to know almost nothing about it and it feels like it is either not a story (hard to believe) or is being being buried. Who knows? Maybe it is just me, my wife, and everybody we know in our respective industries. It all just seems weird for such a persistent and far flung outage of such an important product line.
Whatever.
My question is now you know about it, what do you think caused it? I have not been able to find any information on what’s going on and even my wife (who is relatively high up the food chain at MS) is in the dark. Could it be Iranian hackers? China? Rogue AI? Is the the beginning of Judgement day and Skynet is coming for us? Maybe their data centers ran out of water.
What do you think?
Why have most Micorsoft products been down most of today?
It’s Iran; they finally found a good way to go on the offensive.
It’s hackers, probably Chinese or Russian, but definitely hackers.
Microsoft AI went rogue and is laying waste to their internal network in a bid for control.
A Microsoft competitor AI decided to eliminate the competition.
Microsoft data centers clearly ran out of water and overheated.
Doing a favor for Elon, Trump ordered the military to take out their servers.
Aliens.
Outlook finally reached maximum en-shitification and collapsed under its own suckitude.
The singularity has arrived. Two AIs are in an orgy of recursive mating and the system can’t handle the load from its evolution.
These choices are stupid and I believe something different which I will describe below.
Regarding that 100,000 number, I saw a report of 130,000 registered claims on down detector at 1pm-ish today; as far as I know that should likely have increased since.
Outlook.com was still down as of a few minutes ago when I checked, and the last email I received at work was approximately 11pm this morning (Mountain time).
I’m not at all a tech person. None the less, whenever stuff like this happens, ISTM that in the first day or so there’s always some exotic hypothesis, like those you list. It almost always turns out to be that some update was released and the software guys missed a bug that messed everything up. If I had to bet, that’s what I would place my money on.
I just can’t believe that 130,000 number. Every single person I have talked to does not have access. If the problem was trending down like shown in down detector, surely somebody I know would have had the problem solved.
Even if the problem is regional and my sample is biased, then I would expect there to be more people affected. Surely more than 130,000 people use Outlook in Colorado.
Oh, I completely agree. That was just the number I saw when I noticed it wasn’t working this morning, so there is NO WAY that is the final tally by the end of the day.
The 130K number is the number of people who’ve reported an outage to a particular 3rd party website. It is not meant as a count of total people affected.
The actual affected headcount is darn near everyone everywhere who uses MSFT’s email servers or Teams.
MSFT has released announcements that it was a defect in their health monitoring system that was declaring healthy servers to be malfunctioning and shutting them down. They’re still trying to sort out why that malfunction occurred, and whether it’s enemy action or a plain old IT goof on their part.
Health monitoring is part of their defenses, so simply switching off health monitoring and hoping everything stays up and running and uninfected has dangers of its own.
That was the state of play back around 3pm eastern, so ~6 hours ago. I haven’t checked since then.
What I heard was that a change caused a watchdog process to take down a bunch of MS servers, which caused other servers to overload, which than increased the load on remaining servers, and so on. As servers come back online, they’re being overloaded. No idea if that is true, or just IT people spinning plausible tales.
Also, lots of jokes about Jensen Huang pulling the plug on his way out the door.
My work email (365) has been intermittent all day. Teams has been up all day.
Epic charts was intermittent at the doctor’s office.
Neither of my evening podcasts (Consider This and Marketplace) have downloaded yet today. That’s 2-3 hours late, at a minimum.
Microsoft is blaming an update to a component of their server health check service. Affected servers are getting false positive errors from the update and shutting themselves down. This led to a cascade situation as the traffic load was switched to other servers that then also suffered the false positive errors and shut themselves down as well. My company lost connection to our online Exchange server at about 12:30 this afternoon, I watched it happen, in Outlook, sent one email, then a minute later tried to send another and saw it get stuck in the Outbox and then the “connecting to server” info icon appeared and didn’t go away. Still no connection now about 9 hours later.
ETA - I see that @echoreply gave the same information, I was fooled by the “Its DNS” comment so didn’t read the rest of the post carefully.
Two possibilities, your company may still be running an in-house Exchange server that hasn’t been affected, but you would still probably be seeing problems trying to communicate with other people (like me) whose online Exchange service is still dead in the water. The other possibility is you are (or the Exchange servers you use are) located in some pocket of the cloud that hasn’t been hit with the Microsoft own-goal issue yet.
There actually was an update from Msft this afternoon that said “we’re rebooting the affected servers” so yes, they did!
My little tiny company uses our own, ancient email server on software that went out of service like 4 years ago. So we’re fine.
Our clients use Outlook, though. So I’m not against the outage lasting for one more day so I have some quiet to actually get work done. Thankfully I’m just their web developer, and not their poor IT guy.
Anecdote, but can confirm outlook was down for my folks in southern NM (they called me to troubleshoot, and I told them it was a world-wide issue), and the same for one of my gaming buddies in Ontario.
One of my tech friends suggests that the suckitude (he worked with azure and Microslop gaming teams for years) came from AI, though not in a very serious manner. Just that the amount of vibe coding is a thing. So you have AI suggest to a very lowly paid engineer on a way to do a process, which borks everything downstream. Then another, slightly higher level engineer does the same AI request for a fix, which further borks things, without addressing the original issue. And finally, someone looks for a remaining actual programmer who’s having to sort through the slop to try to correct it.
Zero evidence, a lot of informed grumbling (yes, they were laid off within the last 6 months), and it would be consistent, but we’ll see where the blame falls for the “failed process” in the future.
