The federal government will move to close a “back door” in the metadata retention regime that has allowed a range of groups including the RSPCA and local councils to access the controversial scheme.

The government’s response to a Parliamentary Joint Committee on Intelligence and Security (PJCIS) report into the mandatory data retention regime was tabled this week, with all but two of the 22 recommendations accepted fully.

The PJCIS report was handed down in October 2020, but the former Coalition government sat on it and did not respond to it, despite its bipartisan and unanimous nature.

The metadata retention scheme was launched by the former Coalition government in early 2015, requiring telecommunications firms to keep customer metadata for two years in order to assist law enforcement and security agencies with serious criminal and national security investigations.

There were concerns from the outset, however, that this data would be misused and not handled properly, and that scope creep would see a much wider range of organisations accessing the personal information of everyday Australians.

These concerns were realised in the years since the scheme launched.

The PJCIS report from nearly 18 months ago found that a loophole in the metadata legislation was allowing a range of bodies to access metadata far beyond the 22 law enforcement bodies prescribed by the government.

More than 80 other agencies were found to have accessed the metadata retention scheme, including the RSPCA, Victorian Institute of Education, Taxi Services Commission and local councils.

This is far beyond the 20 law enforcement agencies the government said the metadata scheme would apply to when it was introduced, and means that these organisations are accessing personal data of Australians for reasons other than criminal investigations.

This “back door” was found in part 280(1)(b) of the Telecommunications Act 1997, and the PJCIS recommended that it be scrapped.

The federal government has now agreed to do this, saying it has provided an “inappropriate means to access telecommunications data without appropriate oversight and safeguards”.

“The government will introduce legislation to repeal this provision and replace it with one that limits access to data (including personal information of subscribers) to specified entities in situations where that access is necessary and proportionate to achieving an allowable purpose,” the government’s response to the PJCIS report said.

Attorney-General Mark Dreyfus said the government would make a number of other reforms to the metadata scheme to provide for clear guidelines for the access and management of the data, improvised required keeping and better training for officers.

“The government is committed to ensuring the Mandatory Data Retention Regime continues to support the work of law enforcement and national security agencies while also ensuring that these powers are subject to appropriate safeguards,” Dreyfus said.

“The government will now work to implement the committee’s recommendations as soon as practicable.”

The government will move to launch national guidelines on the operation of the metadata scheme by enforcement agencies to provide greater clarity, consistency and security, and will also amend the Act to clearly define the term “content or substance of a communication”.

The PJCIS recommendation to create better consolidated data on the operation of the regime was also accepted.

“The government agrees with the committee that collecting more information about the current functioning of the data retention regime will assist oversight and review bodies in undertaking their work, provide a higher degree of transparency and give the Parliament and the Australian community greater assurance about the use of these powers,” the government’s response said.

The government was less willing to accept a PJCIS recommendation to increase the threshold for ASIO to authorise the disclosure of metadata.

“Restricting access to telecommunications data as recommended by the committee would be a fundamental change to ASIO’s operations and would significantly constrain ASIO’s capability to protect Australia and Australians from threats to their security,” the government said.

The government will also make reforms to reduce the number of officers who are able to authorise the release of personal metadata.