A bug left undetected for at least three months was the key cause of a major Microsoft 365 outage which began on Friday, Australian time, the technology giant says.
Outages involving dozens of Microsoft services, including email application Outlook and communications platform Teams, saw millions of accounts, including government agencies, affected for around 24 hours.
Microsoft published its preliminary post-incident report on Monday within its Microsoft 365 Admin Center, which is only accessible by account administrators.
The outage was caused by "three main factors", Microsoft stated in its report, which Information Age has obtained.
The first and seemingly most crucial one was "a latent defect" caused by a software update the company made to its Microsoft 365 networking gateway servers in July, when it implemented a fix for a previous issue.
"The update went through safe deployment and saturated globally over several weeks, during which time no issues were observed," Microsoft said.
"For more than three months, the change was deployed worldwide without impact."
Microsoft's testing "did not detect the latent issue", the company said, because it had carried out network load tests using only older and weaker hardware which was less susceptible to the problem.
The bug eventually "caused servers to become overwhelmed and unresponsive under certain load conditions", Microsoft added.
Affected servers became unavailable due to what is known as "lock contention", which occurs when multiple computing processes try to access the same locked resources at the same time, causing stalls and backlogs.
Servers 'ultimately stopped responding'
Two secondary factors appeared to contribute to the outage.
The first was caused by an issue with Microsoft's communications app Teams, approximately 15 hours before the wider Microsoft 365 outage.
This failure saw Teams jump between Microsoft 365's "two independent and heterogenous network gateway systems", the company said, in typical behaviour to avoid further disruptions.
However, Teams jumped to the "primary gateway", which also supports Exchange Online – the backbone of Microsoft's mammoth email and calendar engines.
"This increased the load on the networking gateway and changed its traffic patterns," said Microsoft, which suggested this surge in load "increased the likelihood" that its servers would experience lock contention.
"Isolated server failures began in some regions during this period."
Soon after, at around 3.20am AEDT on Friday, Microsoft engineers "received a high-priority alert indicating impact to Exchange Online services".
The other secondary factor outlined in the report was that "nearly 14 per cent of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day", Microsoft said.
While this drop in server capacity "had no impact on its own", the increased load "caused the network gateway to encounter the latent bug from July".
"This caused requests to build up until servers ultimately stopped responding.
"This occurred during the peak window of traffic in the United States."
'Cascading failure from server to server'
During the increased network load caused by the latent bug, the Teams issue, and the reduced server capacity, Microsoft's network traffic management systems redirected users to other healthy servers, which is typical behaviour designed to keep networks afloat, the company said.
"However, the redirected traffic increased the likelihood for the same problem to occur on the remaining endpoints," Microsoft said.
"This caused a cascading failure from server to server as the same amount of load was spread across fewer servers."
In under two hours, the peak of the outage saw more than 80 per cent of Microsoft 365 networking gateway servers rendered unresponsive.
By 6.20am AEDT, Microsoft had "determined that the elevated load we were observing was not driven by an overall increase in traffic, but by many servers being unexpectedly unavailable, which was shifting load onto and overwhelming the remaining infrastructure".
What's more, the outage spread globally because gaps between so-called 'capacity units' of infrastructure in different regions "did not stop the cascading failure" due to "a separate issue", Microsoft said.
Despite restarting many servers, attempting to bypass problematic components, and making changes to a subset of its servers, Microsoft's actions "had not been effective" by 9am AEDT, the report found.
Attempts to shift network traffic and return affected servers to service saw "limited recovery" in the following hours, but these servers soon "[became] unresponsive again upon receiving traffic".
By 11.30am AEDT, Microsoft engineers had "determined that competing processes were preventing connections from being handled successfully".
After more hours of work, including disabling some automated traffic redirection and reverting the changes which introduced the latent bug, Microsoft 365 "began to stabilise" by around 9pm AEDT, but "capacity had not returned to normal levels".
While some services like Teams, Office productivity apps, and file services OneDrive and SharePoint were restored more quickly, Microsoft admitted "recovery took longer than usual" for some other services, including Exchange Online.
Around 24 hours after outages began, the issue was fully resolved by 3.10am AEDT on Saturday, 14 November.
Microsoft is expected to release its final post-incident review later this week.