Telstra technicians repeatedly decided not to fix the problem that ultimately triggered a network outage affecting 8.8 million people, executives conceded during Senate testimony that also found 8,000 customers have so far been reimbursed an average of just $12.50.
The carrier has so far paid around $100,000 to those affected by the outage Telstra CEO Vicki Brady said as she testified to the Senate Environment and Communications References Committee’s Triple Zero service outage inquiry on Friday.
Brady provided a detailed timeline of the events leading up to the day-long, nationwide outage – which began at 3.38am after a power supply failure prompted the replacement of the chassis of a Microchip Technologies SSU 2000 network time protocol (NTP) server.
Hundreds of NTP servers of different brands synchronise time across Telstra’s network, but the company uses three of the SSU 2000 units – worth around $30,000 each – and in this case, when the Melbourne unit rebooted it adjusted its date back by 1,024 weeks, to 2006.
The problem – which was caused by a well-known ‘GPS rollback’ flaw in the long obsolete NTP server – was detected at 4.20am and continued until 4pm, ultimately affecting 8.8 million customers as a ‘secondary issue’ extended interruptions into the next day.
Investigations found technicians had previously reconfigured the Melbourne server to fix a bug but had failed to document those “design changes” – meaning that while the server replacement was later performed to protocol, it did not respond as expected.
The vendor had issued advisories about the bug as early as November 2000 and Brady admitted that Telstra technicians had logged the issue as early as September 2022 – but that a risk assessment concluded it wasn’t relevant for their network.
Telstra technicians again decided not to fix the bug in January this year, when Brady conceded “there was definitely a reminder that this update needed to be done.”
“It’s not just a case of them ignoring that update,” she said, adding that “they did consider it but the way this was designed, and the way we were using this particular server, the update was affecting a feature that we were not using at that time.”
The undocumented design change, however, “meant that [issue] became relevant” – ultimately leading to the massive network outage.
Telstra executive Gerard Tracey conceded a newer server would not have caused an outage. Photo: YouTube
Under intense grilling by committee chair Sarah Hanson-Young, Gerard Tracey, Telstra’s group owner for service resilience and critical communications, admitted: “[if we had] a newer piece of hardware, operating in the same design that we intended to,” he said, “the issue wouldn’t have happened.”
Hanson-Young described Telstra’s failure a result of “incompetence”.
Charting the path to recovery
The hastily called Senate hearing was Brady’s first opportunity to publicly share details of the outage, which was first detected at 4.20am on July 8 and snowballed as customers woke up, got ready for work, and began encountering phone and data connection problems.
The incorrect date spread across Telstra’s network, creating a cascade of failures that caused the network to reject 40 to 50 per cent of calls since, from the network’s perspective, the phones were effectively calling from 20 years in the future.
By 6.33am, the company was handling media reports about the outage; at 7.11 am, technicians identified which device had caused the problem.
Most calls and data services “were working correctly” by 10.00am, Brady said, and the initial NTP server error had been addressed by 4pm.
“Modern mobile networks are complex, but complexity does not reduce our commitment to prevent failures, detect them quickly, and ensure the right safeguards are in place,” she said.
“We invest significantly in the resilience of our network and in modernising it as technology evolves – but this incident shows we have more work to do.”
ACMA is undertaking an investigation, and Telstra has engaged Technology Audit Partners to conduct a detailed investigation into the outage – with Brady endeavouring to share the results of that investigation and use them to inform future procedural changes.
Senate inquiry chair Sarah-Hanson Young said Telstra's failure sounded like "incompetence". Photo: YouTube
“Networks are not infallible, but our job is to make sure we are taking the steps to mitigate that as best we can,” she said, “and we accept here that it wasn’t good enough; our controls and our processes definitely let us down, and that impacted a lot of people.”
“I’m incredibly disappointed, and I can tell you that everyone across Telstra is disappointed,” Brady said.
“We didn’t intend this to happen, and we need to get to the bottom to understand exactly what happened here and what we need to change.”
A field test for Triple Zero protections
Compounding its massive service disruptions, the outage was a live test of protections designed to reduce the impact of such outages on critical Triple Zero services – including ‘camp on’ provisions that automatically shunt calls to other mobile networks.
Some 58,835 callers successfully connected to Triple Zero during the outage but around 3,200 Triple Zero calls ‘camped on’ to TPG and Optus networks, Tracy said, citing a “significant increase in camp on calls” from around 6.15am.
“At that time,” he said, “we knew the Telstra devices were struggling to connect to the Telstra network and register on our network.”
While the primary fault was fixed by day’s end, a second issue with the same software configuration extended recovery, causing 604 Triple Zero calls to fail through 2pm the next day – including 172 that camped onto other networks and 432 that weren’t connected.
Telstra followed up all calls, conducting 604 welfare checks in total: “the backup process, which is to camp on, worked,” Brady said, “and our welfare check process worked.”
The support of the new Triple Zero Custodian – which Telstra notified of the outage at 7.14am – “was a helpful coordination mechanism” during the outage, Brady said, adding that “Triple Zero sits at the heart of public trust in Australia’s communication system.”
“Telstra has a critical role in that system, and we take that responsibility extremely seriously: no one should be left wondering whether a call for emergency help will get through.”
