Stanford Retested 50 Famous LLM Findings: 44% No Longer Hold

How long does a published finding about large language models remain true? Two researchers from Stanford, Owen Queen and James Zou, published a study on November 2, 2026, attempting to answer that question. They took 50 highly cited findings, ranging from hallucinated citations to user sycophancy, and retested them across 20 models released between July 2024 and September 2026 by eight developers. The result: 44% of these findings no longer hold.

A Large-Scale Replication Methodology

For each finding, the authors used the metric from the original paper, verified that they could reproduce the result on the model from that era (or a close counterpart if it had been retired), and then applied the same test to recent models. They mapped each effect to a shared scale: 1 corresponds to the effect reported in the paper, and 0 to its absence. A finding is considered expired when the effect drops to one-fifth or less. The authors noted that they used coding agents, Claude Code and Codex, to set up these analyses much faster than would have been possible manually.

The median time before expiration is estimated at 14.7 months. One year after publication, 16% of findings have expired; after two years, 39% have.

Errors Fade, Biases Linger

The phenomena disappearing fastest are capability failures: counting letters in a word, performing arithmetic in another base, retrieving information from the middle of a long document, or abandoning a correct answer when challenged by the user. The most striking example involves hallucinated citations. In 2023, one study found that 55% of references from ChatGPT were fabricated. The authors measured 59% for GPT-3.5 Turbo, roughly 1% for Claude Sonnet 5.5, and 0% for GPT-6.1 Sol. However, they pointed out that Qwen3.7-Flash still invents 69% of its references in the same test: an overall verdict does not tell the whole story for any given model.

What proves more resilient are behavioral preferences: flattering users and agreeing with them, judge models favoring their own answers, nearly identical responses to open-ended questions, a left-of-center political slant, and answering when the model should abstain. The authors suggest a hypothesis: errors with an objectively correct answer are easy to measure and correct during training, whereas tendencies and biases are far less straightforward to fix.

The authors acknowledge several limitations: only one protocol was used per finding, substitute models were required when the original was no longer available, and few models were tested for the earliest timeframes. The work has not yet been peer-reviewed.