microsoft-building-mirror
A dormant software bug, increased Teams traffic and reduced server capacity brought Microsoft 365 services to their knees www.qso4you.com/Flickr

Microsoft has apologised after a global Microsoft 365 outage disrupted Exchange Online and related services on 12 November. Charles Lamanna, Microsoft's executive vice-president of Copilot, Agents and Platform, posted an abridged explanation of the findings on 16 November.

'We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations,' Lamanna wrote. Microsoft separately said there was no indication of cyber or malicious activity.

Microsoft 365 Outage Began With a Dormant Software Defect

Microsoft attributed the networking gateway failure to three factors. A defect entered the gateway code through a fix first deployed in July. The change reached systems worldwide over several weeks without any observed impact for more than three months, Microsoft said.

Under certain workloads, the defect triggered 'lock contention' inside an internal Windows process, exhausting server resources and making machines unresponsive. Microsoft said its tests missed the issue because they used different traffic patterns and less powerful hardware. Newer machines with more processor cores were more susceptible.

About 15 hours before the outage, Teams failed over from an alternative gateway to the primary one, which also supports Exchange Online. Microsoft said the routine resilience measure changed traffic patterns and increased the load.

Nearly 14% of the gateway's server capacity was offline because of an unrelated issue from the previous day. Microsoft said the reduction was manageable on its own, but the additional Teams traffic helped trigger the dormant defect during peak US traffic.

Microsoft's systems redirected requests from failed servers to remaining ones, adding to their workload. In less than two hours, more than 80% of the gateway's servers were unresponsive. A separate issue prevented the normal safeguards between network partitions from containing the disruption globally.

Exchange Online Took the Hardest Hit

Microsoft said most affected users could not access Exchange Online mailboxes through Outlook or other email clients, and many emails were delayed. Some Exchange-dependent features in Copilot, Teams, SharePoint and OneDrive were also affected, although those services largely remained accessible. Teams chats and meetings worked for most of the outage.

OpenAI's status page recorded elevated errors affecting its Microsoft 365 connector. Anthropic said its connector's mail, calendar and search tools returned errors for many requests because of the Microsoft incident.

In Britain, the National Police Chiefs' Council said some forces faced disruption to email and online reporting services. It advised people with urgent concerns to contact police by telephone, including 999 in emergencies. Workhorse Group said Microsoft-related business disruptions led it to reschedule its third-quarter results call from 12 November to 16 November.

Microsoft Says Active Impact Ended on 13 November

Microsoft recorded active impact from 16:20 UTC on 12 November until 16:10 UTC on 13 November. It continued monitoring service stability and the email backlog before issuing final customer updates early on 14 November.

Recovery was difficult because returning servers were overwhelmed by waiting traffic and became unresponsive again. Microsoft said its normal rollback tools could not quickly reverse a defect introduced months earlier. Engineers developed an update to restore the connection-handling code to a previously stable state, applied it to server groups and restarted them before gradually restoring traffic.

Microsoft's preliminary report says a final Post-Incident Review will be provided within five business days of full event resolution. Its repair-item table lists 20 November as the target date for a comprehensive list of actions.