GPT-6 Astra on ARC-AGI-3: 62.7% measured by the ARC Prize, far from the 99.9% claimed
Using the standard protocol, the organization that runs the test gets a score well below OpenAI's, at a cost of more than $26,000.
OpenAI claims 99.9% for GPT-6 Astra on ARC-AGI-3, one of the industry’s most closely watched reasoning tests. The ARC Prize, which administers the test, has published its own evaluation: 62.7% under the standard protocol.
Two protocols, two results
The gap does not necessarily mean one of the two figures is wrong. According to the ARC Prize, OpenAI’s score was obtained with an adapted protocol, different from the one applied to every model it evaluates. Yet it is the common protocol that makes it possible to compare systems with one another. The reference figure is therefore 62.7%.
Cost, the other key figure
The ARC Prize also states the cost of its evaluation: $26,098. That amount is a reminder that a reasoning score is paid for in compute, and that two models with the same score can have very different costs. A result without its cost tells only part of the story.
Our take
62.7% is still a result to be weighed against the other models evaluated by the ARC Prize. But the episode illustrates a simple rule: a score announced by a developer under its own test conditions should be read as a claim, not as a measurement. We always favor the measurement from the independent organization.
Sources
- ARC Prize — GPT-6 Astra results on ARC-AGI-3 Primary source — organization that runs the benchmark
A primary source comes directly from the company, lab or organization concerned. A secondary source reports the information without being its author: we flag it as such. Spotted an error? See our corrections policy.
