Evaluating AI Benchmarks: Why MMLU and HumanEval Don't Tell the Whole Story

Model cards used to lead with MMLU and HumanEval scores. In 2026, most frontier labs have quietly stopped reporting them at all — not because the models got worse at those tasks, but because the tasks stopped measuring anything useful. Understanding why matters for anyone choosing a model based on a benchmark table rather than actual task performance.

The saturation problem, in real numbers

MMLU, HumanEval, and GSM8K have been dropped from many frontier model comparison tables because all frontier models have essentially saturated them — MMLU scores above 90% for top models, which structurally limits the benchmark's ability to differentiate between frontier-tier models anymore. (lxt.ai)

The saturation timeline is stark. MMLU shipped in 2020 with frontier model accuracy near 32% — barely above random guessing on a 4-choice format. By Q1 2026, every frontier system reports above 92%. As of May 2026, the entire canonical 2022–2023 benchmark suite — MMLU, GSM8K, HumanEval, MT-Bench, HellaSwag, ARC — is saturated, contaminated, or both, with GPT-5.x, Claude Opus 4.7, Gemini 3.x, and the Llama 4 family all sitting above 90% on every one of them. (hidekazu-konishi.com)

A benchmark that every frontier model clears at 90%+ has stopped doing its job. It can still tell you a model is competent; it can no longer tell you which competent model is actually better, which is the comparison anyone choosing between models actually needs.

Why MMLU specifically has a real structural flaw

Its multiple-choice format is far easier than open-ended generation — models can score well by eliminating obviously wrong answers through pattern-matching, rather than by genuinely understanding the underlying subject matter being tested. (lxt.ai)

Why HumanEval specifically doesn't reflect real coding work

Its problems are self-contained algorithm puzzles, not representative of actual production engineering tasks like debugging, refactoring, or navigating a large existing codebase. A model scoring 95% on HumanEval may genuinely struggle with real-world coding tasks that require understanding broader project context — the benchmark and the real job are measuring different things. (lxt.ai)

A separate, real contamination problem

HumanEval specifically suffers from training-data contamination — models effectively "have seen" benchmark questions during training, inflating scores without reflecting genuine improved capability. Test set contamination generally is a known issue undermining benchmark scores across the field. (lxt.ai)

The contamination problem isn't confined to the older, retired benchmarks either — it's showing up in benchmarks still actively used for frontier model comparisons. OpenAI's own internal audit found that every major frontier model tested — GPT-5.2, Claude Opus 4.5, Gemini 3 Flash — could reproduce verbatim gold patches for some SWE-bench Verified tasks, a strong signal the benchmark's answers leaked into training data somewhere along the way. OpenAI subsequently stopped reporting SWE-bench Verified scores in early 2026 and now recommends SWE-bench Pro instead. (CodeAnt)

What replaced the saturated benchmarks

The field's response to saturation was to build harder successors, not to abandon benchmarking. MMLU-Pro (10-choice format with chain-of-thought pressure to defeat pattern-matching), GPQA Diamond ("Google-proof" PhD-level questions), ARC-AGI 2 (private holdout grid puzzles designed to resist memorization), FrontierMath (research-level math validated by expert mathematicians), and HLE — Humanity's Last Exam, 3,000 questions from over 1,000 domain experts across 100+ subjects — are the current generation. (hidekazu-konishi.com)

The benchmarks frontier labs actually report on model cards in 2026: HLE, FrontierMath, ARC-AGI 2, GPQA Diamond, SWE-Bench Verified (with the contamination caveat above), Aider Polyglot, AIME 2025, τ-bench, BFCL, MMMU-Pro, RULER, and LiveBench. (hidekazu-konishi.com)

ARC-AGI 2 is a useful illustration of how a replacement benchmark tries to resist the same fate as its predecessor: it introduced adversarial task construction and contamination resistance specifically to measure generalization under abstraction pressure, rather than pattern-matching against familiar puzzle shapes. Even so, frontier models "still don't generalize" well on it according to 2026 analysis — a genuinely useful signal that the older, saturated ARC benchmark could no longer provide. (Adaline Labs)

SWE-bench Verified scores actually vary a lot depending on who's reporting

Even among sources still using SWE-bench Verified, reported scores diverge meaningfully depending on measurement date and methodology. One September 2026 source shows a top score of 95.0% across 116 evaluated models. A separate April 2026 source shows Claude Opus 4.6 leading at 80.8%, Gemini 3.1 Pro at 80.6%, and Claude Opus 4.5 at 80.9%. Another April 2026 source shows a different leader entirely at 93.9%, with GPT-5.3 Codex at 85% and Claude Opus 4.5 at 80.9%. Frontier models generally cluster between 54% and 81% on this benchmark depending on which report you're reading. (CodeAnt)

That spread — some sources 15+ points apart on the same nominal benchmark within the same few months — is the practical version of the "evaluation methodology matters" point below. It's not that the sources are wrong; they're measuring under different harness versions, prompt setups, or model snapshot dates, and none of that is obvious from a single headline number.

Another reason scores aren't directly comparable

Different evaluation methods (0-shot vs. 5-shot prompting, different prompt formats) produce meaningfully different results on the same benchmark against the same model — a reported benchmark score is only comparable to another score run under the identical evaluation methodology, not a universal fixed number. (lxt.ai)

Human-preference leaderboards aren't a clean substitute either

The other common alternative to static academic benchmarks is a crowdsourced human-preference leaderboard, the best-known being Chatbot Arena (rebranded "Arena" on January 28, 2026), which ranks models using Elo ratings derived from over 6 million real-user blind pairwise votes. Unlike MMLU or GSM8K, it evaluates open-ended, conversational performance — genuinely more relevant to how most people actually use these models day to day. (Messenger Bot)

But it has its own distinct failure modes, not the saturation/contamination pair above. Model providers can and do optimize specifically for Arena-style prompts, producing inflated scores that don't generalize to domains like coding, safety, or multilingual reasoning — a model can be tuned to "feel" good in a blind head-to-head without being better at the tasks a business actually needs done. There's also a coverage skew: high-visibility models like Claude Opus or Gemini 3.1 Pro appear in far more battles than newer or niche models, so the statistical reliability of the ranking is uneven across the leaderboard — top-tier entries are well-measured, mid-tier ones are comparatively under-evaluated. (Messenger Bot)

The 2026 best-practice response is the same pattern as the benchmark table above: don't rely on any single leaderboard. A serious model comparison combines Arena-style human preference (for open-ended usability) alongside MMLU-Pro (knowledge), GPQA Diamond (reasoning), SWE-bench Verified (engineering), and at least one agent-specific benchmark relevant to the actual deployment context — because human preference, filtered through whatever prompts users happen to submit, is meaningful but is one signal among several, not a final verdict. (Messenger Bot)

Goodhart's Law, and why every new benchmark faces the same fate

The underlying dynamic behind saturation and contamination isn't a one-time accident that happened to hit MMLU and HumanEval — it's a structural pattern known as Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. Every new benchmark gets compromised within months of release, because training data inevitably comes to include test samples, paraphrased versions of test samples, or conceptually similar problems, whether or not anyone deliberately trained on the answer key. (FourWeekMBA)

There's also a more deliberate form of gaming beyond passive contamination: labs can submit multiple variant runs to a leaderboard, test privately, and publish only the most favorable result — a strategy estimated to be worth roughly 100 points of leaderboard gain per strategic multi-submission cycle on some agent benchmarks. This isn't hypothetical or fringe; Meta itself acknowledged it "cheated a little bit" when submitting Llama 4 results to a public leaderboard. (UC Strategies)

By January 2026, frontier models routinely exceed 90% on math, coding, and QA benchmarks that were considered genuinely difficult only two or three years earlier, and the differences between top models on these older benchmarks now fall within measurement noise rather than reflecting a real capability gap. The gap between test performance and real-world utility has, if anything, widened rather than closed as this saturation has occurred — a high score increasingly tells you a model is good at benchmarks, not necessarily that it's better at your actual task than a competitor scoring a few points lower. (UC Strategies)

The practical takeaway

Don't pick a model based on a headline benchmark score alone. For coding specifically, check performance on something closer to real work (SWE-bench Verified or SWE-bench Pro, which test actual GitHub issue resolution) rather than HumanEval's isolated puzzle format — but even then, cross-check against at least two independently reported scores given how much SWE-bench Verified numbers vary by source and date, and stay aware of the documented contamination risk in its answer set. For general reasoning claims, prefer model-card benchmarks from the current generation (HLE, GPQA Diamond, ARC-AGI 2, FrontierMath) over any 2022–2023-era number, since those are already saturated across the board and no longer differentiate frontier models from each other.

Sources: LXT — LLM Benchmarks Compared: MMLU, HumanEval, GSM8K and More (2026), CodeAnt — SWE-bench Leaderboard 2026: All Model Scores, Rankings & What They Actually Mean, hidekazu-konishi.com — LLM Benchmark History and Timeline, Adaline Labs — ARC-AGI In 2026: Why Frontier Models Still Don't Generalize, Messenger Bot — Chatbot Arena 2026: How To Use It & Read The Rankings, FourWeekMBA — The Goodhart's Law Trap: When AI Metrics Become Useless, UC Strategies — AI Benchmarks Are a Game Now — And the Industry Is Cheating to Win