A bug in a program used to test software updates was the cause of Friday’s mass global IT outage, cyber security company CrowdStrike has said in its first review of the incident, offering $US10 Uber Eats gift cards to “teammates and partners” who assisted the US firm.
The company’s software update crashed around 8.5 million computers running Microsoft’s Windows — disrupting everything from airlines to banks, supermarkets and news broadcasters — has promised to improve its systems, as financial losses for affected organisations continue to mount.
CrowdStrike’s quality-control software did not properly validate the contents of the software update it pushed out on Friday, the company said in its report.
The software update which caused Windows machines to crash, triggering the so-called Blue Screen Of Death (BSOD), was an updated threat sensor for CrowdStrike’s cyber security software Falcon.
The crashes were “due to a defect” in the software update “which went undetected during validation checks”, the company said.
When the faulty update was loaded by Falcon, it caused what is known as an “out-of-bounds memory read”, which occurs when a program reads data from a place in the computer’s memory which it is not supposed to, sometimes causing crashes.
The report is broadly in line with what cyber security experts believed caused the mass outage.
CrowdStrike released a fix for affected systems last week, but experts have stated it will take some time to get all 8.5 million affected computers back online.
CrowdStrike promises to improve
To prevent such an incident from happening again, CrowdStrike said it would improve its software testing procedures and Falcon’s error-handling capabilities “to ensure errors from problematic content are managed gracefully”.
The company said it would change its software deployment strategy by using a so-called “canary deployment” to roll out changes to a small number of systems first, before a wider rollout is initiated.
It also promised to “conduct multiple independent third-party security code reviews” and “independent reviews of end-to-end quality processes from development through deployment”.
Friday's outage saw 8.5 million Windows computers crash and display the so-called Blue Screen Of Death. Photo: Shutterstock
CrowdStrike said it would develop better monitoring systems for updates, and provide customers with more controls and notifications about changes.
Security researcher Kevin Beaumont described CrowdStrike’s report as “good and really honest”, but said the company “deploying rapid content updates globally simultaneously” was a tactic “waiting to go wrong”.
“None of this is Microsoft’s fault. CrowdStrike made a boo boo, it happens,” he said on social media platform X.
“Ultimately it’s a collective error. CrowdStrike’s response has been really good post error.
“They clearly realise they need to prioritise safety now.”
CrowdStrike said its full investigation would be released publicly in a forthcoming Root Cause Analysis.
CrowdStrike offers $10 vouchers
A CrowdStrike spokesperson reportedly confirmed to TechCrunch that the company had offered a $US10 Uber Eats gift card to some of its “teammates and partners” who assisted in the wake of the global outage.
A copy of an email reportedly sent to voucher recipients said CrowdStrike recognised “the additional work that the July 19 incident has caused”.
“And for that, we send our heartfelt thanks and apologies for the inconvenience,” it said.
“To express our gratitude, your next cup of coffee or late night snack is on us!”
However, some recipients reported receiving error messages when they attempted to redeem their voucher.
CrowdStrike spokesperson Kevin Benacci told TechCrunch:
“We did send these to our teammates and partners who have been helping customers through this situation.
“Uber flagged it as fraud because of high usage rates."
The damage bill from the outage is expected to run into the billions of dollars, amid calls for CrowdStrike and Microsoft to compensate some affected companies.
Australia’s Minister for Home Affairs, Clare O’Neil, said on Saturday, “There is no question that there is going to be an extensive conversation from here about the cost to companies, to consumers, about CrowdStrike and how it has handled this matter.”
CrowdStrike CEO George Kurtz, who has already been criticised over his initial public response to the outage, has been called to answer questions about the incident by the US House of Representatives Homeland Security Committee.