Hundreds of automated decision-making systems have been found in the first mapping of the technologies across the NSW public sector, with most being developed without a specific regulatory framework.
The state’s Ombudsman — who uncovered unlawful automated revenue recovery technology in 2021 — on Friday released the mapping and warned of inconsistencies, missing legal advice and a lack of governance in the use of automated decision-making (ADM) systems.
Departments, agencies and councils are not yet required to disclose the use of ADM generally or when providing reasons to individuals about an administrative decision, but are increasingly turning to the tools to improve services and efficiencies.
The ADM systems range from an agency’s long service leave planning assistant and local council customer chatbots to highly impactful applications like police conducting automated risk assessments on an individual’s risk to reoffend
The new research for the NSW Ombudsman relies on voluntary disclosures by agencies and local councils. It still found 275 ADM systems in use or planned to be in use, including 198 in the state government sector and 77 in the local government sector.
As well as the lack of visibility of the ADM systems, agencies and officials appear to have a lack of appreciation for the existing legal frameworks that should be keeping them in check, NSW Ombudsman Paul Miller said.
“It was almost as if there was a view that because the technological tools were new, or because they were not expressly prohibited or regulated, that their use was unregulated, unless and until new laws were made specifically to deal with them,” he told a NSW parliament inquiry into AI on Monday.
“Of particular concern was the risk of ADM projects being perceived, developed and run primarily through the lens of being an IT project without proper consideration of the fact that they are first and foremost about the lawful performance of an administrative, usually statutory, function.”
The research also revealed that many projects had no legal input at the design phase, while very few ever progressed to a minister or cabinet process.
“It was apparent through the process of surveying agencies, that it’s not the case, for example, that the secretary of the department has visibility, let alone decision-making authority, over all the ADM systems within their their own department,” Mr Miller said.
“So it is certainly the case — and I think the research bears this out — that decisions are being made at a relatively disaggregated and potentially quite low level within an organisation.”
The state Ombudsman has put the spotlight on ADM systems since 2021 after uncovering Revenue NSW had unlawfully used one to recover unpaid debts, in some cases emptying the bank accounts of vulnerable people.
The Ombudsman wanted public servants on the front foot, aware of the risks of automating decisions, including potential maladministration.
The follow up work tabled to Parliament on Friday is based on the mapping and analysis research by the ARC Centre of Excellence on Automated Decision-Making and Society.
It confirms automation is “widespread, varied and increasing” across the public sector, Mr Miller said, and that the systems typically aren’t being used to make a final decision.
But there is only limited oversight, with most of the ADM systems reported not being subject to any ADM (or AI) specific regulatory framework.
While NSW has a country-leading AI Assurance Framework intended to de-risk projects, it is applied only to future projects that meet certain investment thresholds.
A new version of the AI Assurance Framework is being finalised and will be embedded in the state government’s general ICT guidance, but it is unclear if the new version would capture more ADM systems to become a comprehensive regulatory tool.
“The current version of the assurance framework is of limited value in terms of being a comprehensive regulatory tool for controlling the use of AI and automated decision-making,” Mr Miller said.
While many existing technology agnostic regulatory frameworks can be effective checks, the rise of ADM systems is also creating genuine questions about things like a right to reasons, a right to be heard and systemic bias in algorithms, Mr Miller said.
Former Australian Human Rights Commissioner Ed Santow, who led a landmark report on technology and rights and is now a professor at the Human Technology Institute, says huge reforms aren’t necessarily needed.
But ADM systems like AI do require clarification in law to ensure transparency on how decisions are made and explained, and there must be recourse to challenge them in a technical and meaningful way.
“Where you have a right to reasons because of an administrative decision, it would be completely undermining of that right, if all you get instead of reasons is just the ones and zeros [or] the source code,” he told the inquiry.
“You need to be able to, if you’re using AI to make a decision, provide a meaningful, plain English answer.”
At the federal level, the Albanese government has committed $10 million to develop a consistent legal framework for automated decision-making in response to the robodebt scandal — a gross misuse of automated decision-making.
