Neural Brew

Models & labs

Intelligence Index v5: Artificial Analysis moves to 60% private tests, and the gap with open models widens to 12 points

Artificial Analysis overhauls its benchmark index: 60% of the score now rests on private or hidden-answer tests. Claude Opus 5.5 takes the lead with 58 points, and the best open-weight model falls to 12 points behind the leader, from 3 previously.

AI-generated portrait of Lior

By

Published · 3 min read

Abstract generated illustration for the Models & labs section

Artificial Analysis, the independent organization whose Intelligence Index serves as a reference for much of the industry in comparing AI models, has published version 5 of its index. The announcement thread appeared on X this Wednesday, October 28 at 3:11 a.m., Paris time (the site article is dated October 27, U.S. time). According to the organization, this is the biggest overhaul since version 4 launched in January.

The fundamental change comes down to one number: 60% of the score now rests on evaluations whose test set is private or whose answers are not published. The goal: to limit “contamination,” those cases where a model has already seen a test’s questions during training.

What’s in, what’s out

The index counts nine evaluations spread across four equally weighted blocks (25% each): office work, code, science, and general knowledge.

  • New: FrontierCode 2.0, Cognition’s private test launched the same day (132 tasks in real open-source repos, scored based on whether a maintainer would or would not accept the agent’s change), and Terminal-Bench-Science, 70 scientific research tasks to carry out in a terminal.
  • Updated: HLE-Diamond, tightened to 1,000 questions from Humanity’s Last Exam; AA-Omniscience v1.1, cleaned of questions that were too easy or already public; a private subset of 40 hard physics questions from CritPt v2.
  • Removed: GDPval-AA v2.1, SciCode, and AA-LCR v1.1.

The new ranking

Claude Opus 5.5 (Anthropic, max effort) takes the lead with 58 points, ahead of GPT-6 Astra (57) and GPT-6.1 Sol (55). Anthropic and OpenAI hold the top five spots; Gemini 4 Argon follows at 50. No model dominates all nine tests, however: Opus 5.5 wins three, and five other models share the remaining six.

Price weighs heavily in the reading. Artificial Analysis measures the cost of each index task: $10.04 for Opus 5.5 at maximum, $6.24 for GPT-6 Astra, $1.27 for GPT-6.1 Sol. In other words, Sol comes within 3 points of the leader for roughly one-eighth the price. Reasoning effort also counts more than before: Opus 5.5 drops to 35 points at low effort ($0.98 per task).

Open models fall behind

This is the most commented-on result. In version 4.3, the best open-weight model was only 3 points behind the leader; it is now 12 points behind. That best model, Kimi K3.1 Pro (46 points), has not yet released its weights, incidentally. Among actually downloadable models, DeepSeek V4.1 Flash 1017 and GLM-5.3 cap at 32, behind the smaller Claude Haiku 5.5 (36). The gap is widest on HLE-Diamond and Terminal-Bench-Science.

We had covered Kimi K3.1 Pro’s entry at 4th place in the old version of the index: with v5, this same model remains the top open model, but far behind the leading trio.

Artificial Analysis is launching a Progress Index v1 in parallel, which places models released since November 2022 on a single scale, to track the evolution of capabilities over time.

Our take

Moving to private tests makes the index harder for labs to “game,” which is good news for its credibility. The trade-off: no one can verify the results from the outside, and you have to trust the organization. The retreat of open models is striking, but it should be read with caution: part of the gap may come from new tests, for which open models have not yet been optimized. For a concrete choice, the score-to-price ratio (GPT-6.1 Sol at one-eighth the cost of Opus 5.5) often matters more than first place.

Sources

  1. Artificial Analysis — Announcing Artificial Analysis Intelligence Index v5 Primary source
  2. Artificial Analysis (X) — announcement thread for Intelligence Index v5 Primary source
  3. Artificial Analysis — Intelligence Index methodology Primary source

A primary source comes directly from the company, lab or organization concerned. A secondary source reports the information without being its author: we flag it as such. Spotted an error? See our corrections policy.