Artificial Analysis Decision Index v2: Calibration Enters the Score, Claude Haiku 5.5 and GPT-6 Luna in the Lead
Artificial Analysis has released the second version of its index for decision models: a score that rewards accuracy and calibration, 45% private data, and a new benchmark spanning 1,500 documents. Three specialized models sit within a single point at the top, trailing Claude Haiku 5.5 and GPT-6 Luna, which are far slower and more expensive.
A decision model does not generate text: it selects among fixed options (routing a ticket, screening a resume, calling a tool) and outputs a probability for each in a single call. Artificial Analysis has just released the second version of its benchmark dedicated to these models, announced on X overnight between November 8 and 9. The fundamental shift: the score now factors in accuracy and confidence.
A Metric That Penalizes Overconfidence
Until now, benchmarks primarily tallied correct answers. The new Decision Score also looks at the probability a model assigns to its output. A correct answer yields between 50 and 100 points, while an incorrect one falls between 50 and 0: a model that is confident and correct scores 100, but one that is confident and wrong scores 0. The methodology explains why: in production, an output is only executed if confidence clears a specific threshold; otherwise, it is handed off to a human or a more capable model. The score averages this utility across all possible thresholds.
The benchmark has also revamped its test suites. AA-Paperclip (30% of the total), a private dataset of 1,500 documents (resumes, receipts and invoices, research papers), requires five decisions and ten extractions per document; a document is only counted as correct if all 15 answers are right. AA-Symmetry (15%) poses 120 questions across 440 equivalent formulations to ensure a model does not flip its decision when choices are reordered or yes/no questions inverted. WildGuardTest, deemed saturated, has been dropped. According to Artificial Analysis, 45% of the index now relies on private data—up from 25% in v1—to curb contamination on public benchmarks.
Three Specialists Within a Point, Generalists Out Front
Among dedicated decision models, three stand within a single point of each other: Drex Max v1.1 (Nace AI) and GLiDE v1.2 (Fastino Labs) at 73, and Mammoth 27B (Base Labs) at 72. General-purpose large language models, prompted to output probabilities per option under the same protocol, score higher: Claude Haiku 5.5 (Max) reaches 77, and GPT-6 Luna (Max) hits 76.
That edge comes at a steep price. For one million tasks, GPT-6 Luna (Max) costs $1,530 and Claude Haiku 5.5 (Max) costs $6,240, compared to $102 for Drex Max v1.1 and $14 for Celeris-1.1 Decision. Latency differences are just as stark: Luna takes 11.34 s per task and Haiku requires 32.65 s, versus 275 ms for Drex Max and 56 ms for Celeris. Token pricing alone does not tell the full story: GLiDE v1.2 bills the document on every query, pushing its effective cost to $780 per million tasks—nearly eight times Drex Max, despite advertised rates being just 33% higher.
Caveats to Keep in Mind
The two versions cannot be compared: the scoring formula, test suites, and weightings have all changed. Part of the index also relies on other models: on EvalSafe-O*NET-AA, which lacks objective ground truth, the reference label is determined by consensus between GPT-6 Astra and Claude Fable 5.1. Finally, in the comments beneath the announcement on LinkedIn, a user writing on behalf of the GLiDE team claims a newer, stronger version than the evaluated release is already available, though this remains unverified.
Our take
For teams orchestrating agents, this v2 finally evaluates what matters in production: knowing when a model can decide autonomously and when it must escalate. The most revealing takeaway is not the leaderboard, but the staggering gap in cost and latency: generalist models buy a few extra benchmark points at fifteen to sixty times the price, trading milliseconds for double-digit seconds. For high-volume routing or classification, small specialized models remain the rational choice. That said, teams should validate them on internal datasets, as any benchmark—even one with private splits—remains the assessment of a single organization.
Sources
- Artificial Analysis — Decision Index v2 (results page) Primary source
- Artificial Analysis — Decision Index Methodology Primary source
- Artificial Analysis on X — Decision Index v2 Announcement (November 8, 2026) Primary source
- Artificial Analysis on LinkedIn — Introducing the Artificial Analysis Decision Index v2 Primary source
A primary source comes directly from the company, lab or organization concerned. A secondary source reports the information without being its author: we flag it as such. Spotted an error? See our corrections policy.
