READ. SCROLL. LISTEN.

Unbiased headlines. Facts, not spin.

Unbiased Headlines is an independently owned news organization. We bring you the facts, not opinions, written fresh from 110+ named sources. Sources linked so you can verify it yourself.

← Back to headlines

Microsoft's Preliminary Report Blames July Bug, Teams Traffic Shift and Lost Server Capacity for 365 Outage

Microsoft's Preliminary Report Blames July Bug, Teams Traffic Shift and Lost Server Capacity for 365 Outage
Four days after the Microsoft 365 outage began, Microsoft's preliminary report says a bug shipped in July sat dormant for more than three months until extra Teams traffic and reduced server capacity set it off. More than 80% of its networking gateway servers stopped responding, and the safeguards meant to contain the damage failed. Microsoft has not yet explained several pieces of the chain, and a final report is still to come.

Four days after the Microsoft 365 outage began on Thursday, Nov. 12, Microsoft has put a technical explanation on the table. The company's preliminary post-incident report, posted to customers in the Microsoft 365 Admin Center, and a Monday, Nov. 16, post on X by executive vice president Charles Lamanna lay out three problems that landed at the same time.

None of them was an attack. Microsoft has said from the start that the outage was not a cybersecurity breach.

A bug that shipped in July

The root defect came from an update to the software on Microsoft 365's networking gateway servers, the machines that sit in front of Outlook, Exchange Online and other services. The update began rolling out in July and spread worldwide over several weeks. It ran for more than three months with no visible problems.

The flaw was a lock contention problem. Under heavy load, parts of the software could get stuck waiting on one another, eating up processing power and memory until the server froze.

Microsoft says its load testing missed it. The company's labs leaned on weaker hardware with fewer processor cores and a different traffic mix. The report says the contention gets worse on machines with more cores and newer hardware, which is what the production servers use.

Two triggers, same morning

About 15 hours before the outage, Microsoft Teams failed over from its alternate networking gateway to the primary one that supports Exchange Online. That added traffic. Microsoft called it a normal operation and has not said what prompted it.

At the same time, nearly 14% of the gateway's server capacity was out of service because of an unrelated issue from the previous day. Microsoft says the remaining capacity was enough to handle the load on its own.

The two together were not. "Taken together, the increased load and the reduced server capacity, caused the network gateway to encounter the latent bug from July," Lamanna wrote. "This caused requests to build up until servers ultimately stopped responding. This occurred during the peak window of traffic in the United States."

The outage began at 11:20 AM ET on Thursday. Microsoft says the defect hit its server fleet in under two hours. At the low point, more than 80% of the gateway servers were unresponsive.

Safeguards that spread the problem

The design meant to keep failures contained did the opposite. When servers stopped responding, traffic was redirected to healthy servers running the same flawed code. Those buckled too. That is a cascading failure.

Microsoft also says protections meant to stop problems from crossing between regions failed because of what the report calls a separate issue. It has not explained that issue. The result was a global outage that, by Microsoft's count, reached users in more than 70 countries.

Recovery was slow for two reasons, according to Lamanna. Servers that came back were immediately flooded by queued traffic and failed again, a pattern engineers call a "thundering herd." And Microsoft's automated recovery tools "were not designed to restore service from this broad unresponsive state."

Because the bug was months old, the usual fix of rolling back a recent change was not available. "Our normal rollback, flighting, and feature flags were unable to quickly revert the change," Lamanna wrote. Teams, OneDrive and SharePoint could shift to alternative infrastructure and recovered earlier. Exchange Online could not, and delayed email kept arriving after service returned Friday morning.

Microsoft's position

Microsoft's framing is that this was a convergence, not a single failure. Executives say each of the three factors could have been handled by itself. The company also points out that the faulty update passed its standard staged rollout and ran cleanly for months.

The damage was not abstract. Jails could not release prisoners, hospitals fell back to paper records, schools lost contact with parents and police departments could not receive crime reports, according to the Guardian's account of the outage. Microsoft 365 has more than 430 million users, according to figures cited by TradingView. Microsoft shares barely moved during the disruption, TradingView reported.

Open questions for the final report

GeekWire flagged two areas of likely scrutiny: why the testing did not use the hardware Microsoft actually runs in production, and why a fix that passed staged rollout still carried the flaw. GeekWire also noted that Microsoft spent $45.2 billion on capital expenditures in its most recent quarter, much of it on AI infrastructure.

Several links in the chain remain unexplained by Microsoft: what the July update was meant to fix, what caused the 14% capacity loss, why Teams failed over, and what the separate failure was that let the outage cross regional boundaries. It is also unclear whether a separate Teams outage earlier on Nov. 12 was related.

Microsoft has said it will publish a final report with more detail. That document is where the company is expected to describe what it will change in testing, rollout and recovery tooling.

Sources used for this briefing

8 links from 8 independent newsrooms (syndicated copies of the same report count once)

These are the sources it draws on, linked so you can verify.

center
LiveMintMicrosoft explains global outage: How a four-month-old software bug triggered disruption and delayed recovery | Mint
center-left
The IndependentWhat caused the global Microsoft outage? Why Outlook went down
center-left
MashableMicrosoft outage cause revealed: It was a perfect storm
left
The GuardianMicrosoft identifies months-old bug as a root cause of global outage
unknown
BleepingComputerMicrosoft says months-old bug caused global Microsoft 365 outage
unknown
finwire.ioMicrosoft identifies months-old bug as a root cause of global outage
unknown
GeekWireWhat went wrong with Outlook: Microsoft blames ‘thundering herds’ and a bug unnoticed for months
unknown
TradingViewMicrosoft Reveals Hidden Bug Behind Massive Global Outage