THE UK Government has been warned that it risks eroding public trust if it continues to push through public-sector collaboration with the US tech firm Palantir.
The company, which has been accused of selling its services to Israel’s IDF and has been identified as “profiting from genocide” in Gaza, has been awarded more than £670m worth of government contracts.
This includes a £330m contract with NHS England relating to patient data, and a £240m contract with the Ministry of Defence (MoD).
In June the firm also won a £9 million contract to supply and manage the software behind licensing for firearms, explosives and poison.
But Mark Wong, an AI ethics expert researching the topic at Glasgow University has warned of "significant risks" if the Government continues with its current AI roll-out plans.
Wong, a senior lecturer in the university's social and urban policy department, said: "There are significant risks of losing public trust and worsening inequalities when AI is implemented from the top down.
"Public trust towards government institutions has already been in decline for years, but rushed adoption of AI systems is breaking that trust further.
He explained that without the input or informed consent of frontline staff in the public sector the Westminster Government is creating a sense of exclusion from key decision-making processes that directly impact their work and day-to-day lives.
"Many people are feeling increasingly excluded from government's decision-making about how AI is used in the public sector," he said.
"Several cases of AI causing wide-spread concerns and harms could have been pre-identified or prevented, had frontline workers and diverse communities been meaningfully consulted prior to deployment, such as healthcare.
"But the concerns from people who are most likely harmed by the integration of AI, such as health and care workers, patients, and marginalised communities, are rarely listened to before an AI is deployed in key public services. "
It comes amid widespread backlash from healthcare workers, protesting the implementation of Palantir software into NHS England's data systems.
More than 80,000 people have now signed a petition started by the Good Law Project calling on the tech firm's contract to be scrapped.
On Wednesday June 3, Westminster's Department of Science, Technology and Innovation (DSIT) Committee unveiled a report warning of a "concerning" reliance the UK Government has on the tech firm.
The report warned that the UK Government "appears to be worryingly comfortable" with how dependent the public sector is on a small number of large digital and technology providers, such as Palantir.
Of all the technology providers that the public sector relies on, the committee ruled that "Palantir concerns us most", saying that its presence "represents an unacceptable point of weakness".
Amnesty International echoed the report, raising concerns about the company "supporting highly controversial policies and activities".
"Palantir has been supplying software to the Israeli military and intelligence services that contribute to Israel’s ongoing genocide, apartheid, and illegal occupation in Gaza," Kristyan Benedict, the organisation's UK crisis response manager, said.
"A company profiting from such grave human rights abuses should have no role in our health system or wider public sector," he added.
Alongside its links to Israel's genocide in Gaza, Palantir has also been used by the US Immigration and Customs Enforcement agency (ICE) to target, identify and locate immigrants.
The firm's co-founder Peter Thiel has also come under increasing fire for his comments around the NHS, which he said needed "ripping from the ground" and starting over, along with a 22-point manifesto published by Palantir that has been condemned as "evil" and "disturbing".
Wong added that the Government has a "duty to ensure" that decisions are made in the public's best interest, but at present this is being disrupted by the influence of tech companies and lobbying from big businesses.
He said: "The government has a duty to ensure that the public and their best interests are at the heart of policymaking processes.
"But, at the moment, only tech companies and big businesses have the closest influence on government ministers and advisors, whilst the public are neglected and have to live through the consequences of poorly-informed decisions made behind closed doors.
"These consequences are not merely financial, reputational, and potentially legal, especially for public services; it can risk exacerbating discriminatory and unfair outcomes for communities, which go directly against the ethical principles of the public sector and government itself. "
To safely implement AI into the public sector, Wong advocated for more rigorous requirements and the creation of an independent audit system, arguing that right now tech firms are benefitting from a lack of scrutiny and vetting.
"Normally, the public would expect technology in the public sector are well-vetted and rigorously assessed before being deployed. But that is far from being true about AI.
"AI regulation is so weak and broken, which means evaluation of AI risks often rely on partial and often unsubstantiated claims provided by businesses and tech companies who are trying to sell their AI products.
"Research has shown that many of these claims, such as cost saving, workplace efficiency, or green credential, are largely untested and speculative, which means many so-called AI benefits may have never proven to work in real-world contexts or the context it is being applied to."
Wong previously told The National that the UK is unlikely to see even half of the productivity gains being predicted by the Government from AI.
"There is, therefore, a blind leap of faith taken when largely untested, unproven AI technologies are believed as the saviour of all the problems of the NHS and the public sector," he added.
"There is no regulator requirement that ensure AI risks are rigorously evaluated or audited by an independent organisation. There are also hardly any standards that regulate how AI audits should be conducted and how risks should be mitigated.
"AI governance and audits remain the 'Achilles' heel' that companies exploit and reap massive profits from.
The Department for Science, Innovation and Technology has been approached for comment.
