Alex Hern was correct (Ofqual’s A-level algorithm: why did it fail to make the grade?, 21 August) to point out that the “algorithm” is actually an equation. This is not just a nerdish distinction. A statistical model, which should be the basis of this type of algorithm, would estimate the likelihood that a particular factor would influence the accuracy of predicted grades and the strength of impact when it does. The method employed by Ofqual provides neither piece of information. However, even well-designed statistical models would not necessarily produce a fair result, because the various correlations would not be 100%. In statistics, therefore, all predictions come with a health warning, called the error term. Knowing the reliability of a model would be scarce consolation to any student victim of its degree of unreliability, regardless of how small this level might be.
Emeritus Professor Graham Hall
Bury, Greater Manchester
• For several years I was an admissions tutor at a middle-ranking university. We needed to improve our first-year pass rate so, based on five years of first-year entrants, I created a predictive model using data scraped off Ucas forms and actual results obtained to predict whether students would pass our first year. Doing this involved what the press have quaintly termed “an algorithm” – in fact, I used a mixture of statistical methods, mathematics and machine-learning (AI) techniques. The model was not of course used to make admissions decisions, but was used to inform admissions strategy.
Purely for interest, I then adapted my “algorithm” to make predictions on A-level results. I found that I could not achieve an accuracy of greater than about 75% to within one grade. Even by changing my “algorithm”, using predictive models with a different mathematical basis, I could not improve the accuracy. My conclusion was that there was something missing from the data itself – it lacked the predictive power. Something that might be called the human factor – students who put in an extraordinary effort, students with changing circumstances etc. Something that could possibly have been filled in by teachers. What I find staggering about this whole fiasco is why this “algorithm” wasn’t tested against last year’s results – or if it was, why such an appalling level of accuracy was accepted.
Dr Peter WH Smith
Watton-at-Stone, Hertfordshire
• It comes as no surprise to me that the algorithm hasn’t worked. Back in 2001, colleagues and I showed that schools’ A-level performances from one year to the next were pretty unstable when intake characteristics were taken into account. In short, using one year’s results to predict the next would be an inherently risky business.
We titled our analysis Predicting the Future. We thought it sufficiently important to send to several exam boards and government players at the time. What we didn’t predict was that 20 years later it would have been entirely forgotten!
Prof John Gray
Faculty of education, University of Cambridge
• There is often much to learn from obituaries, and I was impressed by that of the French philosopher Bernard Stiegler (18 August), whose work highlights the negative impact on society of the use of algorithms and automated systems, particularly damaging to the lives of young people. The exams debacle suggests it should be required reading.
John Bailey
St Albans, Hertfordshire
• Alex Hern’s article about Ofqual’s A-level algorithm reminds me of another equation. It has taken me most of my adult life to devise it: ME = NT3. This equation can be translated as “my experience is [that one should] never trust the Tories”.
Tony Leather
South Shields, South Tyneside
• Join the conversation – email guardian.letters@theguardian.com
• Read more Guardian letters – click here to visit gu.com/letters
