Having borne the brunt of criticism over the eCensus failure in August in silence, IBM Australia is now trying to foist blame on everyone from the Australian Signals Directorate to the service providers that carried internet traffic to the website.
Big Blue broke its silence on the eCensus this week with a contentious submission to a Senate inquiry examining the failure.
The company said it still “deeply regrets the inconvenience” caused to Australians on Census night, and that it “accepts its responsibility as head contractor” for the project.
But it used its submission to blame Nextgen Networks and Nextgen’s then upstream provider Vocus for allegedly failing to properly implement a geoblocking strategy that IBM had in place to counter the threat of a distributed denial of service (DDoS) attack.
Nextgen and Vocus countered that they had been kept in the dark about the geoblocking strategy until just days before the eCensus site went live, and that IBM ignored their advice to buy proper DDoS protection.
IBM also attempted to blame its customer, the Australian Bureau of Statistics (ABS), for the majority of the downtime that the eCensus site experienced.
“Public access to the site was temporarily suspended - initially for approximately three hours at the direction of IBM, and immediately thereafter for approximately an additional 40 hours at the direction of the ABS,” the company said.
And IBM also implicated the Australian Signals Directorate (ASD), whom it alleged had “declined to undertake a detailed review” of Census security, unlike in past years.
“IBM understands that the ASD was asked by the ABS to review the security arrangements for the 2016 eCensus site, but the ASD declined to undertake a detailed review,” IBM alleged.
“IBM understands that the ASD had reviewed arrangements for the 2011 Census, which also employed geo-blocking arrangements” to counter DDoS attacks.
Continued blame-shifting surrounding the high-profile IT failure has not been well-received by the IT community, but the stakes are high.
@FakeDanTosello A lot of blame shifting about ability between various submissions to the enquiry, specifically ABS<->IBM<->NextGen<->Vocus
— Name Withheld (@p_ansell) October 20, 2016
Prime Minister Malcolm Turnbull promised “very serious consequences” for both IBM and ABS in the wake of the failure, and it seems any companies involved in some way aren’t immune from being dragged into the fallout.
However, after being publicly dragged in, Nextgen and Vocus both put forward significant counterclaims to IBM’s submission, along with contractual evidence (redacted from the public versions of their submissions) to support their assertions.
And things are likely to get even uglier in the short term as the two main players – ABS and IBM – front a public hearing on Tuesday October 25.
The Senate committee examining the failure must report by November 24.
Network errors
Much of the new allegations centre on ‘Island Australia’, the internal codename that IBM uses to describe a geoblocking service that it hoped to use in the event of a DDoS to block traffic coming from international sources.
It believed it would be effective as Census traffic was meant to come from Australians who were in Australia at the time, and therefore its mitigation strategy was to shut off non-domestic traffic sources.
Island Australia would be implemented by the two telcos contracted to carry internet traffic to the Census website – Telstra and Nextgen.
IBM said that “Telstra and NextGen agreed to block international traffic as a DDoS mitigation strategy” but did not say when the agreement with either was reached.
Generally, information about Census site security “was treated as confidential and generally shared only on a need-to-know basis” both in the lead-up and during the project, IBM said. While ABS and the ASD were informed well ahead of time, it appeared the telcos found out about the mitigation strategy very late.
Nextgen said it “was not privy to the IBM ‘Island Australia’ strategy until 20 July 2016, just 6 days before the eCensus site went live on 26 July 2016”, but still went ahead to meet IBM’s requests as best it could.
Nextgen was using capacity on international links owned by other companies, including Vocus. Vocus said it its own submission that “it advised Nextgen that it did not provide geoblocking”.
A Vocus link in Asia would later carry some of the DDoS traffic to the Census site; Vocus and Nextgen dispute the impact of that occurrence.
But IBM sees it as a central reason for the site being pulled offline. “Had Nextgen (and through it Vocus) properly implemented Island Australia, it would have been effective to prevent this DDoS attack and the effects it had on the eCensus,” IBM said.
Vocus instead latched onto a brief admission by IBM of human error, where employees acted on a site performance monitoring system that “miscarried”.
“Some IBM employees who were observing the monitor mistakenly formed the view that there was a risk that data was being exfiltrated from the website and that the risk needed to be further investigated,” IBM said.
“Out of an abundance of caution, IBM shut down access to the site and assessed the situation. The cause of the problem was identified. No data exfiltration occurred.”
“The cause of the census website being unreachable was IBM employee’s falsely identifying normal traffic patterns as data exfiltration, and manually turning off their Internet gateway routers which IBM took approximately three hours to configure and bring the website back up again,” Vocus said.