Posted by:
Paul Muir, Chief Customer Officer at risr/
I took three weeks off and came back to the apparent end of the world! Somewhere between the out of office switching on and switching off, a researcher resigned from Anthropic, told the internet that the AI labs are gambling with our lives, and a Nobel laureate (and Godfather of AI) went on the BBC to agree that a one in ten chance of human extinction was not unreasonable.
I had a gentle piece on the summer’s exam results ready, to ease myself back in. It can wait. When the Godfather of the field puts the survival of the human species in single figures, and then, predictably, the world’s richest man announced it was all a hoax. We will come to him…
1. The weeks I missed
The plan was a soft, Disney-happiness fuelled landing. Well it was until I turned on the news…
On the 8th September, Jacob Coxon, who had spent three years training frontier models at both OpenAI and Anthropic, resigned and published a warning that the leading labs are racing to build AI that can improve its own intelligence, faster than anyone can keep it under control. His message reached more than a hundred million people within a day. He was careful, to his credit, about what he was and was not claiming. The danger from today’s models, he said, is low. His fear is what happens if these systems start to bootstrap themselves, each version designing a cleverer one, quicker than any human can follow.
The part that should stop you is not that a researcher is frightened. It is who agreed with him. Evan Hubinger, who still works at Anthropic, said he puts the chance of AI causing human extinction within the decade at more than ten per cent. And this week Geoffrey Hinton, the Nobel laureate often called the Godfather of AI, went on BBC Newsnight (interview clip there) and, asked whether a one in ten chance of AI killing every human was a fair estimate, said it was not unreasonable. Something far cleverer than us, he added, would not be short of ways to manage it.
2. A number with no instrument
Here is where the assessment hook comes in, because I have spent a career asking awkward questions, sometimes because I don’t understand it all myself, sometimes because I question the motives of the people/project.
Ten per cent. Measured how? With what instrument? Validated against what? Normed on which population of previous extinctions? There is no reliability coefficient on this figure. There is no confidence interval around it. There is not even agreement among the people who quote it. Hinton does not really mean ten. He means ten to twenty, and has drifted upwards as the technology has accelerated. Yoshua Bengio, another founder of the field, sits nearer twenty. One well-known safety researcher puts it at all but certain. Elon Musk lands close to Hinton and decides the upside is worth it anyway.
In assessment we have a word for a score produced like this, by expert intuition, unvalidated, swinging wildly from one rater to the next, and then made to carry a life-changing, high-stakes decision. We call it indefensible, and we keep it away from candidates. The strangest feature of this whole business is that the people who have built the most powerful measurement machines in history are steering the species by a number that would fail their own first review/moderation.
3. But unmeasured is not unreal
And yet. If I stopped there I would be doing the lazy thing, the dismisser’s thing, and the dismisser is every bit as unmeasured as the doomer.
The absence of a clean instrument is not the absence of the danger. Some of the most important things we try to assess are the ones we arguably measure worst of all. Competence. Judgement. Integrity. We do not declare them imaginary because the ruler is rough. We build the best instrument we can and we treat the rough signal with respect. And the signal here is not faint. The people closest to the machinery, the ones with the most to lose from looking foolish, are the ones raising the alarm. Some have walked away from remarkable salaries to do it. A Nobel laureate has put his name to it on television. That is not a measurement. It is a long way from nothing.
A ten per cent claim you cannot verify is not a reason to panic. It is not a licence to relax. It is a sign that the most important instrument in the room has not yet been built.
In Assessment and particularly Test Security, I know we keep raising the threats from AI, but all the time we are REALLY careful to balance that with the opportunities it presents. However, I’m fairly confident from the last 2 years of panels, presentations and blogs that more than 10% probably think AI has the potential to destroy Assessment as we know it. That doesn’t get the same headlines and is often equally dismissed by some within our sector, or more frequently outside the sector who simply don’t get the context.
4. Enter the psyop
As the week wore on the warnings multiplied rather than faded. More of Coxon’s Anthropic colleagues put their names to the same fear, one observing drily that the more senior the researcher, the more worried they tend to be. You might think that would settle whether the concern is sincere. It did the opposite.
The loudest reply was not a rebuttal but a diagnosis of motive. Elon Musk decided the whole thing looked like a setup, and floated that it was a coordinated psy-op to force AI regulation. The evidence offered was that the posts went viral too fast, and that Coxon had left just before his stock options vested. Coxon replied that he was entirely real, and that Musk was welcome to ask his own former xAI safety researchers about him, had he not fired them!
This is fake news territory, arriving on cue. When you cannot refute an argument, you question whether the person making it is real. It is effective precisely because it never touches the claim. But it is a category error. Whether Coxon’s virality was organic and whether superintelligence can be controlled are different questions, and the first tells you nothing about the second. Even if the whole affair were a PR operation, which nobody has shown, the missing instrument would still be missing.
And consider the source. The man crying panic has himself put the odds of extinction at ten to twenty per cent, so he agrees with the number while mocking the people who say it aloud, and he happens to own a rival AI lab. When your fellow doomsayer shouts fake the moment a competitor’s staff say the quiet part in public, that is not a debunking. It is a tell.
The Measured Response
The title of this piece is a measurement joke that is not really a joke. A confidence interval is the range within which the truth is likely to sit, and the honest one around ten per cent is vast, from close to zero to genuinely frightening. Confidence carries a second meaning too, the human one, the nerve to hold a steady line while others lose theirs. That is the interval I would ask you to occupy.
There are three easy ways out of it, and each is a dodge. The ‘doomer’ swallows ten per cent as gospel. The ‘dismisser’ spits it out as noise. And the ‘conspiracist’, able to do neither, declares the messenger a fake. The measured path, which is what us as assessment professionals usually take, refuses all three. It asks how the number was built, treats a rough signal from frightened experts with respect, and keeps asking the question the psyop theory is designed to bury. Can we actually control what we are making?
Judging what you cannot directly observe, and defending that judgement afterwards, is what our industry does, so let me offer my take. On the evidence, the risk is real, the number is soft, and the instrument to sharpen it does not yet exist. Build the instrument. Keep a human close enough to the machine to reach the handle (our Human/Expert-in-the-Loop). And when a billionaire assures you that the people building AI are lying about being frightened of it, ask who has earned more trust, the engineers walking away from their unvested stock, or the man who owns the competing lab.
That’s the Measured Response.
Over to you…
The next time an inconvenient warning in our industry (maybe Test Security, Remote Proctoring, AI automation etc) is dismissed as a hoax or a stunt, ask the harder question. Is the claim actually wrong, or is it merely easier to doubt the messenger? Those are not the same thing, and telling them apart is fast becoming the whole game.
Until next week…
Paul
