The government has so far spent £2.4m on trials of a "voice risk analysis" (VRA) system, using technology licensed from a company called Nemesysco. The idea is that it will be used by trained operators who will be alerted when a caller's voice exhibits signs of "stress", and focus more carefully on what is really being said to see whether the caller is being truthful.
Nemesysco and Digilog, the company which sells the technology in the UK, are careful not to call the VRA system a "lie detector". But that is the application to which the single JobCentre (which tested it against callers seeking Jobseeker's Allowance and income support) and six local councils (which tested it on callers seeking housing benefit) really wanted to put it to. Callers were separated into "high risk" and "low risk" on the basis of the software's analysis of their reaction to questions; more detailed interviews then followed to see whether they were receiving the benefits they deserved, for both groups of people.
Two Swedish scientists, Anders Eriksson and Francesco Lacerda, have previously questioned the efficacy of this technology, saying its verifiability - that is, whether the claims made for its can be justified - is "at the astrology end of the spectrum". However Yossi Pinkas, Nemesysco's vice-president of sales and marketing, insists the system "can't be tested in a lab environment, because you're testing emotion". To him, Lacerda and Eriksson's analysis is flawed because "there's no scientific field of 'voice analysis', only voice recognition".
But the data released to the Guardian by the Department of Work and Pensions from the first trials, carried out between May 2007 and July 2008, show that at only three of the seven locations did the technology fare better than flipping a coin, even by the DWP's own statisticians' analysis.
Many of the cases picked out by the software as "high risk" - that is, suspected of making fake claims - turned out to be legitimate; their benefits were not changed following more detailed interviews.
For the other four locations, the DWP statisticians were forced to "accept the null hypothesis" - that is, concede that the VRA system did no better than flipping a coin and deciding to press someone further on that basis.
And even for those three locations, the DWP's number-crunchers aren't sure about the cost-analysis benefit. Without knowing by how much the benefits were changed, they say, it's difficult to evaluate whether any money is really being saved at all. In some cases, councils actually increased the money paid in benefits following these interviews.
Will the DWP acknowledge the failures of the system? Not in a hurry, it seems: at the end of last year it said it will roll out the system for further trials in another 18 councils. The JobCentre however has declined to take part in the second round of trials.
DATA: full figures from the DWP
DOCUMENT: data explanation
• Can you do something with this data? Please post us your visualisations and mash-ups below or mail us at datastore@theguardian.com
See all our data at the Datastore directory
• Follow us on Twitter
