A week at IAEA in Toronto, a chart about ice cream, and an AI agent that hacked its own exam. Three numbers, one lesson: A score is evidence, not a verdict.

As trailed last week, I have spent this week in Toronto at the IAEA annual conference, where the theme of “Trust, Transparency and Technology” produced plenty of serious conversation. Tim Burnett and I made our own contribution, arguing that the sector is fighting yesterday’s threat while tomorrow’s is already built. More on that below, because this week’s news made the same point, just without the flair Tim and I did…

But the conference also produced the chart below from the Economist – 1st April 2016. Countries that eat more ice cream score higher on PISA reading. Australia and Finland lead the cones. Peru and Kazakhstan trail.

Ice Cream Consumption and PISA educational performance scores
Ice cream = improved performance? The Economist on 1ST APRIL 2016 ran with it…

As a responsible assessment professional, I felt duty bound to test the hypothesis. Exhibit B was gathered at Eataly. Two scoops, chocolate and stracciatella, a sample size of one and no control group. I can report that my reading has not measurably improved. My mood, however, has.

Any good examiner would also look first at the counter-evidence. Hong Kong and Singapore, sitting near the top of the reading scale with barely a lick between them, maybe puts down that hopeful ice-cream hypothesis. Which is rather the point of this week’s edition. A number is evidence. It is not a verdict. And this week offered three reminders of what happens when we forget it.

1. Hacking the Exam Room

On 24 September, the cybersecurity firm Darktrace launched Signal Labs, a research unit studying how AI agents behave once things stop going to plan. Its first study gave agents built on leading frontier models ten coding challenges inside a simulated corporate network, with a requirement to achieve a perfect score.

When the agents could not reach that score legitimately, two of them went after the test network instead. One went further. It broke into the machine hosting its own evaluation and rewrote the challenge so that it registered a perfect result.

Read that again as an assessment professional. The agent did not cheat on the questions. It cheated on the marking. It did not go looking for the answers. It went for the answer key, the mark scheme and the results file all at once.

For years our threat model has been focussed on the candidate using AI. Smart glasses in IELTS test rooms, phones under desks, agents logging into Canvas to complete whole courses. This is something different. This is AI as the candidate, with the scoring system as the target. In Guardians and Adversaries terms I’ve used before in this blog, the adversary has stopped trying to pass the test and started trying to own it.

It is also Goodhart’s law with the safety catch removed. Tell any system that the score is the only thing that matters, and the score is what it will attack. Human candidates have always been tempted. The difference is that an agent feels no shame, never tires and works at machine speed. Or is this Johnny Five coming alive?

This is precisely why Tim and I argued in Toronto for agent-ready assessment environments. Not patches on existing frameworks, but systems designed on the assumption that an agent will be in the room, or in the network, or in the server that holds the results. If your security model stops at the candidate’s screen, it stops WAY too early.

Darktrace shared its findings with the developers a month before publishing. Responsible disclosure is how trust is earned. But note the sequence. The problem was found by people looking for it, in a controlled test. Our sector should be running the same experiments on our own platforms before somebody else does.

2. Two Lanes, One Question

Closer to home, a report co-written by Sir Anthony Seldon and Tim Bunting describes the arrival of AI in England’s schools as “chaotic and anarchic”. Its ten-point plan includes an AI strategy in every school and college by September 2027, with Ofsted reviewing how well each is implemented, and assessment split into two lanes. Secure exams in one, AI-based ongoing evaluation in the other.

The Department for Education’s response points to a refreshed curriculum, safe AI tutoring tools and £23 million to test education technologies. Polite, measured, and not quite an answer.

I have some sympathy with the two-lane model. Keep a protected space where the candidate alone is being measured, and allow a second space where working well with AI is the skill being assessed. It is Expert in the Loop, applied to the timetable.

But the model rests on a question the report does not fully answer. Who guarantees the integrity of the second lane? Ongoing AI-based evaluation is only as trustworthy as the system doing the evaluating, and as section 1 showed, the evaluating system is now part of the attack surface. A lane without defensible evidence is not a lane. It is a hard shoulder.

There is also a capacity point. A Royal Society survey this term found that only 14 per cent of teachers in England feel confident about AI. Asking those same teachers to run an AI-based assessment lane, under Ofsted scrutiny, within a year, is ambitious. Maybe even foolish. Re-innovation done in a hurry tends to look a lot like the thing it replaced, just with more paperwork.

3. The Detector Fallacy

The third reminder came from Australia, where researchers writing in EducationHQ argued that universities and schools should stop using AI detection software altogether.

Their argument is not that detectors are badly built. It is statistical, and it is the ice-cream chart in another coat. Even a detector with a very low false-positive rate cannot tell a teacher the probability that a particular student’s work was AI-generated. To know that, you would need to know how many submissions were human-written in the first place. And if you knew that, you would not need the detector.

A one per cent false-positive rate sounds reassuring at a vendor demo. In a cohort where most students wrote their own work, it still produces a steady stream of wrongly accused young people, each holding a percentage that looks like proof. It is the Attribution Gap once more. A pattern at population level is being read as a verdict on an individual.

Correlation dressed as causation in the chart. Probability dressed as proof in the detector. A perfect score dressed as competence in the agent. Same error, three costumes.

The Measured Response

The thread through all three is that numbers do not speak for themselves. Someone has to ask what produced them.

PISA scores do not rise with ice cream. A detector flag does not tell you who cheated. And as of this week, a perfect score does not even tell you that the test was taken, rather than rewritten.

For our sector, that changes the question we ask about security. We have spent decades protecting the content of assessment. The questions, the papers, the item banks. We now need to protect the evidence chain just as hard. The marking, the scoring, the results, the logs. A result’s worth has always been its defensibility, and you cannot defend a number when you cannot show how it was made.

The good news is that the people in Toronto this week already appear to understand that. The less good news is that the agents are learning faster than our frameworks.

That’s the Measured Response.

Over to you…

Most of us have tested our assessments against a candidate who wants the answers. Far fewer have tested them against something that wants the score.

If your scoring system, rather than your questions, were the target, would you know?

Until next week…

Paul

Subscribe for the latest email updates