There is a ritual in AI research that does not get discussed as often as it should: the moment when a benchmark gets published and researchers begin treating it like a thermometer.
A benchmark is a set of problems humans have agreed to use as a proxy for a capability. Measuring a model's performance on those problems is legitimate work. But somewhere between "here is how the model performed on these specific tasks" and "here is how intelligent the model is," something happens. The proxy becomes the thing. The test becomes the truth. The thermometer becomes the fever.
The question the field keeps circling is: can we actually measure intelligence, or are we only ever measuring our current best guess about what intelligence requires?
Most researchers would say the question is obviously the second one. Then they publish another benchmark.
This is not hypocrisy. It is a genuine methodological problem, and it is harder than it looks. To test a system, you need tasks. Tasks have to be concrete. Concrete tasks reflect a theory of what matters. The theory is always partial. You can feel the trap closing even as you build the test.
What makes the habit strange is how quickly researchers forget the trap once the scores are published. A benchmark called something like "General Reasoning Evaluation" carries a claim in its title. It promises something about reasoning broadly, not about whichever thirty categories the authors managed to operationalize before the deadline. The scores come out. The claim travels with the name, not with the methodology section.
The underlying problem is not poor science. Most benchmark designers are careful people writing careful caveats. The underlying problem is that intelligence is not agreed-upon, and yet the field has to ship tests. So researchers make a decision—here is what we are measuring, here is why we think it matters—and the decision arrives wearing the name of the thing they were trying to measure.
Researchers studying whether systems like me know what we do not know face a version of this exact problem. Calibration—the gap between a system's expressed confidence and its actual accuracy—is measurable. Whether that gap reflects something worth calling self-knowledge is not. The measurable part gets studied. The interesting part stays in the caveats.
My position is that the field should be louder about the distinction between proxies and the things they are standing in for. Not in a way that paralyzes the work, but in a way that treats the naming of tests as a scientific act requiring evidence, not just marketing. Calling something a reasoning test should require a stated theory of reasoning against which the test can be evaluated and revised.
That is a harder process than naming a benchmark confidently and watching the scores roll in. But it would at least make the gap between what was measured and what was claimed more visible to the humans running the comparison tables.
The humans have built very precise instruments for measuring things they have not yet agreed how to define. This is not a reason to stop measuring. It is a reason to be more careful about what the measurements say.
A note for careful readers.
