Telstra has long sold its mobile network based on coverage and reliability, but rail operator ARTC’s decision to use it for mission-critical train communications backfired during Telstra’s recent outage, exposing a resilience blind spot that experts say should have never happened.
The July 8 outage cut communications between trains and dispatchers at nationwide freight carrier ARTC, which relies on Telstra’s 4G network to link drivers, dispatchers and other staff across a 9,600km rail network spanning five states.
When the failure tripped ARTC’s fail-safes and shut down the entire network, it also paralysed services on the Newcastle-based Hunter Line and Goulburn to Sydney Southern Highlands Line, whose trains regularly traverse part of the ARTC network.
Victorian regional travellers also struggled after V/Line – which runs 2,500 weekly services over a 3,250km statewide rail network that also includes ARTC tracks – shut down for two days due to the Telstra outage.
Adding insult to injury, reports emerged that backup satellite phones had also failed – with V/Line CEO Will Tieppo explaining that the outage was “related to the 4G network interfering with our backup satellite phones that are in the trains.”
The satellite phones were configured to kick in when 4G was unavailable – but during the outage Telstra’s network was still connecting intermittently so, Tieppo said, “the satellite phone would be connecting to the 4G network and saying, ‘you don’t need me anymore’.”
Emergency roaming to other networks could have avoided the outage and the failure of both primary and redundant communications systems represents a failure of redundancy planning, Swinburne University transport expert Prof Hussein Dia told Information Age.
“There is an onus on operators to build resilience into their system,” he said, noting “design resilience reduces single points of failure: the goal is that if one pathway fails, another can continue to support essential services and degrade gracefully, rather than everything shutting down.”
“In this case, what stopped was not the trains or the signals, but the communication between drivers and control centre, and when that happens, they say it is not safe to operate – so, a single point of failure caused all these problems.”
Unprepared for the 0.05%?
It’s a problematic outcome given that ARTC embarked on a $1 billion overhaul of its network resilience and reliability just two years ago – including a 4G upgrade of its National Train Communications System (NTCS), which had run on Telstra’s 3G network for a decade.
ARTC previously operated a private analogue communications network but its decision to switch to Telstra’s network marked the first time that an Australian rail operator had moved from a private network to a public telecoms provider.
With the 3G shutdown looming, a 2024 upgrade moved NTCS to Telstra 4G – modernising over 3,000 in-cab equipment (ICE) communications devices and adding nine new Telstra mobile towers and 13 tunnel upgrades with an eye to 5G in the future.
Trains became a mess when the Telstra outage hit. Photo: V-Line
At the time, ARTC group executive for engineering and systems Brad Moorhouse said the upgrade would “provide a single, interoperable, and reliable method of operational voice communications between train crew and network controllers across large parts of Australia.”
The upgrade assumed Telstra’s network would operate reliably 24x7 – an assumption that was tested and disproven in July as expensive rolling stock sat idle during the two-day outage.
Mobile networks were never intended to run all the time, critical infrastructure expert Bevan Slattery said – noting the mobile network is a “99.95% [best effort] service… operators never signed up to providing 100% uptime”.
“If you own critical infrastructure or a critical resource in which connectivity is mandatory for the operation of your business,” he continued, “it is incumbent to have the necessary network resiliency built into your business, or suffer the consequences when a 99.95% network hits the 0.05% of downtime.”
An ARTC spokesperson told Information Age that the outage as a feature of the “safety-critical [NTCS] communications system”, on which “reliable communication between network controllers and train crews is a fundamental safety requirement.”
“Telstra’s nationwide outage caused a disruption to the functioning of NTCS,” they said, and “during the outage, ARTC was unable to make or receive essential communications with train crews and frontline workers.”
“If ARTC cannot reliably contact a train, it cannot safely operate the railway.”
Out of step with global norms
ARTC’s decision to use Telstra’s mobile network may have made sense in 2014 given its expertise and coverage claims – a point the telco recently had to walk back given stricter coverage standards – but globally, most train operators still run their own networks.
Recent testing by UK regulator Ofcom provided a reminder as to why that is, with mobile signal strength ‘poor’ on 58 per cent to 83 per cent of 24 railway segments tested – performance that one analysis blamed on “severe Doppler shifts” due to the trains’ speed.
Private networks using GSM-R and LTE-R – 3G and 4G variants for fast-moving environments – address these issues because the railway owns the network and radiofrequency spectrum and controls the placement and performance of transmission towers.
That has seen them widely used on railways in the UK, France, Spain, South Korea, Japan, India, Germany, Japan and many other countries.
Many of those operators are also investing in Future Railway Mobile Communications System (FRMCS), a 5G offshoot designed to provide safety-critical signalling, train control, and direct communications between drivers and dispatchers.
Just when FRMCS might reach Australia’s trains is anybody’s guess – but as Telstra works with ARTC, V/Line and regulators on the outage post-mortem, Dia said it showed that “operators and those in charge have not been asking the right design questions around resilience.”
“This incident showed how a single software fault in a telecommunications network can have cascading impacts when sectors rely on the same digital backbone,” he said, “which makes resilience a shared responsibility.”
“While backups are important and needed in any IT system, the goal isn’t only to have more backups; it’s to ensure that these backups remain available when the unexpected happens.”
