Benchmarks

From PRIMO.ai
Jump to navigation Jump to search

YouTube ... Quora ...Google search ...Google News ...Bing News


A benchmark is a named, reusable test for AI systems: a fixed collection of tasks, a rule for scoring the answers, and usually a public leaderboard where results can be compared. The point is simple. Capability is not directly observable, so a benchmark turns an abstract question ("can this model read?", "can it fix real software?", "can it reason about something it has never seen?") into a number that anyone can reproduce.

The idea is older than modern machine learning. Alan Turing's imitation game (1950) turned the philosophical question "can machines think?" into an operational test. In the late 1980s and 1990s, DARPA's speech-recognition evaluations and NIST's Text REtrieval Conference (TREC) made shared test sets the backbone of applied research. Statistician David Donoho later named this pattern the common task framework and argued that it drove much of machine learning's progress: publish a dataset, fix a metric, run a leaderboard, and let competitors iterate. ImageNet (2009) did the same for computer vision, GLUE (2018) for language understanding, and MLPerf (2018) for hardware. Since 2024 the center of gravity has shifted again, toward tests written by large expert consortia (Humanity's Last Exam), interactive environments (ARC-AGI-3), and measurements expressed in units of human labor (METR's time horizons and OpenAI's GDPval).

Every benchmark relies on the same mechanism: hold the test fixed, vary the system, compare scores. That works only when four conditions hold:

  • Construct validity: the test actually represents the capability being claimed.
  • No contamination: the system has not already seen the answers, or close paraphrases of them.
  • Grader validity: the scoring rule marks right answers right and wrong answers wrong.
  • Statistical power: the test has enough items that differences between systems are not noise.

Most of the history on this page, and nearly all of the recent controversy, is about one of those four conditions failing.

The paradigm shift of the past two years is a move from asking what a model answers to asking what it completes, how reliably, at what cost, and who checked. It also differs from conventional computing benchmarks. SPEC CPU or LINPACK measure a well-defined physical quantity such as operations per second. AI capability benchmarks measure a latent construct, which makes them closer to psychological testing than to engineering metrology, and brings along psychometrics' oldest problems: validity, reliability, test familiarity, and test-takers who learn the test instead of the subject. MLPerf, covered below, sits between those two traditions.

Three pages on this wiki cover overlapping ground. The division of labor:

A recurring caution applies to everything below. Benchmarks saturate. The history of the field is a sequence of tests that fell and were then dismissed as never having measured intelligence: checkers, chess, Jeopardy!, Go, protein structure prediction, competition mathematics, and most recently the Turing test itself. Saturation is also speeding up. Epoch AI estimates that individual benchmarks now typically saturate within one to three years, and several tests launched in 2025 as "hard for years" were substantially climbed within about twelve months. A high score is evidence about a test, not a verdict on a system. The constraints that make a benchmark meaningful (contamination resistance, construct validity, independent administration) are discussed in the sections below and on AI Verification and Validation.


You can't improve what you don't measure. But once a measure becomes the target, it stops measuring what you meant. A benchmark score is evidence about a test, not a verdict on a system.


The opening maxim is usually credited to Peter Drucker, although the Drucker Institute says he never said it. The second sentence restates Goodhart's law in the form popularized by anthropologist Marilyn Strathern.


The 2026 Stanford AI Index documents rapid model capability changes alongside growing problems with established benchmarks, including benchmark saturation, invalid questions, and difficulties inferring real-world reliability from leaderboards. A benchmark score is a measurement conditioned on a specific dataset, time, protocol, and model version, not a permanent ranking of intelligence. In reading new papers, prioritize evaluation-set provenance, contamination checks, replication, and concrete task success over isolated headline percentages. This is particularly important for agent systems and multimodal models, where tool behavior, context, and environmental changes complicate comparisons.


Why This Matters

Benchmarks steer research. ImageNet is the standard example: the 2012 ImageNet competition result for AlexNet, a deep convolutional network, cut top-5 error from about 26% to about 15% and helped launch the deep learning era. Benchmarks act as research "north stars," and whatever they reward gets optimized, whether or not it is the capability people actually wanted.

Benchmarks drive engineering and purchasing decisions. Teams choose models, hardware, and vendors partly on published scores. When a benchmark is broken, the damage is not academic. OpenAI's 2026 audits of SWE-bench Verified and SWE-bench Pro called into question many procurement and tooling decisions made on the strength of those numbers.

Benchmarks underpin governance. Frontier developers' safety frameworks define capability thresholds (for example, uplift in offensive cybersecurity or biological design) that trigger additional safeguards. Government institutes run pre-deployment evaluations. The 2026 International AI Safety Report, chaired by Yoshua Bengio, identified an "evaluation gap": capabilities advanced quickly while methods for testing them lagged. If evaluations are invalid, contaminated, or gamed, thresholds anchored to them provide false assurance. See Risk, Safety, Compliance and Regulation and AI Governance.

Benchmarks feed economic forecasts. Time horizons and expert-comparison suites such as GDPval are increasingly used to estimate how much work AI systems can take on. These measurements are only as good as their task suites, which today skew toward well-specified digital work.

Limitations. No benchmark measures "intelligence" in general. Scores are conditional on scaffolds, settings, compute budgets, and the specific item distribution. Many benchmarks are built or funded by organizations whose products they rank. And the act of publishing a benchmark starts the clock on its contamination.

What would change if measurement matured. If the field converges on valid, contamination-resistant, independently administered, human-anchored measurements with honest error bars, benchmark results could support confident regulatory thresholds, defensible procurement, and credible forecasts. If it does not, the industry risks optimizing a dashboard of proxies while real-world reliability lags behind.

What remains uncertain. Nobody yet knows how to measure open-ended research ability, long-horizon reliability in messy environments, or safety properties in systems that can recognize when they are being tested. Whether composite indices such as ECI capture a real "general capability" or simply reflect developers optimizing all benchmarks at once is still debated.

Recent Developments

The period from late 2024 to October 2026 changed benchmarking more than any comparable stretch since GLUE. Five themes stand out: saturation accelerated, the benchmarks themselves came under audit, evaluation moved from answers to actions, leaderboard incentives drew scrutiny, and measurement science moved from footnotes to the center of the field.

Saturation at a glance

Benchmark Introduced Format Status as of October 2026
GLUE 2018 Nine sentence-level understanding tasks Human baseline passed within about a year; retired as a frontier signal
SuperGLUE 2019 Eight harder tasks plus diagnostics Human baseline passed by early 2021
SQuAD 1.1 / 2.0 2016 / 2018 Reading comprehension span extraction Human baselines passed by 2018 and 2019
MMLU 2020 About 16,000 multiple-choice questions, 57 subjects Saturated above 90%; an estimated 6.5% of items contain errors
HumanEval / GSM8K 2021 164 Python problems / about 8,500 math word problems Saturated; widely assumed contaminated
GPQA Diamond 2023 198 expert-written science questions Effectively saturated, with frontier scores in the low-to-mid 90s
SWE-bench Verified 2024 500 curated GitHub issues OpenAI stopped reporting it in February 2026 (contamination, flawed tests)
SWE-bench Pro 2025 1,865 tasks (731 public) OpenAI estimated about 30% of public tasks broken in July 2026
Humanity's Last Exam 2025 2,500 expert questions Top scores rose from single digits to roughly 45% within about 18 months
FrontierMath Tier 4 2025 Research-level mathematics Corrected v2 released June 2026; top Epoch-run scores above 80% by mid-2026
ARC-AGI-1 2019 Abstract grid puzzles Largely solved; OpenAI's o3 scored 87.5% in a high-compute run in December 2024
ARC-AGI-2 2025 Harder compositional grid puzzles Best Kaggle entry 24% in 2025; creator describes it as since saturated by refinement harnesses
ARC-AGI-3 2026 Interactive game environments Frontier models below 1% at launch; humans solved every environment
Terminal-Bench 2.0 2025 89 containerized command-line tasks About 50% at launch; version 2.1 repaired 28 tasks

Benchmarks under audit

The most important development of the period is that benchmarks themselves started failing their own tests, and the field began auditing them systematically.

  • MMLU. The Are We Done with MMLU? study (NAACL 2025) had 14 experts re-annotate 5,700 questions across all 57 subjects. It estimated that 6.49% of MMLU questions contain errors, found that 57% of the analyzed Virology questions were flawed, and showed that correcting errors changes model rankings. The corrected subset is released as MMLU-Redux.
  • Humanity's Last Exam. In July 2025 the nonprofit lab FutureHouse reviewed the 321 text-only chemistry and biology questions and reported that 29% (plus or minus 3.7%, 95% CI) had answers directly contradicted by peer-reviewed literature. Critics argued that "conflicting evidence" is not the same as "wrong." The HLE team partially replicated the findings and launched HLE-Rolling, a continuously revised version with a public change log; FutureHouse released a validated chemistry and biology subset.
  • SWE-bench Verified. In February 2026 OpenAI, which had co-created the set in 2024, announced it would stop reporting scores. Its team audited 138 problems that o3 failed to solve consistently across 64 runs (27.6% of the set) and found that at least 59.4% had flawed tests that rejected functionally correct solutions. It also found evidence of contamination, including models that could reproduce gold patches. OpenAI recommended SWE-bench Pro instead.
  • SWE-bench Pro. Five months later, on July 8, 2026, OpenAI reversed course. Its audit of the 731-task public split noted that the best pass rate had climbed from 23.3% to 80.3% in eight months; an automated pipeline flagged 200 tasks (27.4%) as broken and a parallel human review flagged 249 (34.1%). OpenAI estimated roughly 30% of tasks were broken, citing overly strict hidden tests, underspecified prompts, and low-coverage tests, and retracted its earlier recommendation. A separate audit by Datacurve, reported in May 2026 but not confirmed by Scale AI, claimed substantial grader error as well.
  • FrontierMath. Epoch AI released version 2 on June 12, 2026, stating that it addressed errors in 42% of problems. The Tier 4 set was revised by correcting 12 problems and removing 7, leaving 43.
  • Terminal-Bench. Version 2.1 fixed 28 tasks from version 2.0 and introduced continuous validation.

The lesson is uncomfortable but useful: a broken item does not just add noise. Over-strict tests make good systems look bad; leaked or under-tested items make weak systems look good; and both effects grow precisely as systems approach the ceiling. Grader validity has joined contamination as a first-order concern.

From answers to actions: agentic and long-horizon evaluation

  • METR time horizons. METR's paper Measuring AI Ability to Complete Long Tasks (March 2025, published at NeurIPS 2025) introduced the 50% time horizon and estimated a seven-month doubling time since 2019, with signs of acceleration. Its January 2026 "Time Horizon 1.1" update added longer tasks, removed flawed ones, and estimated a post-2023 doubling time of about 131 days. By mid-2026 METR's estimates for the most capable systems it had tested exceeded a full working day of expert time at 50% success, while 80% horizons remained a few hours. METR stresses the wide confidence intervals and the software-heavy task mix.
  • GDPval. OpenAI's September 2025 suite drew more than a thousand tasks from 44 occupations in the nine sectors contributing most to US GDP, written by professionals averaging 14 years of experience, and had experts from the same occupations grade AI and human deliverables blind. In the first round, Claude Opus 4.1 was rated as good as or better than the professional's work on 47.6% of tasks, GPT-5 on 38.8%. Models completed tasks roughly 100 times faster and cheaper, a figure that excludes human review time. OpenAI acknowledges that the tasks are one-shot deliverables, not whole jobs.
  • Terminal-Bench 2.0. Released November 7, 2025 by Stanford and the Laude Institute, it contains 89 human-verified tasks in Docker containers spanning 16 categories, from debugging asynchronous code to security fixes, and ships with Harbor, a framework for running containerized agent evaluations at scale. The best agent scored about 50% at launch.
  • ARC-AGI-3. Launched March 25, 2026, it drops agents into small video-game-like environments with no instructions. They must explore, infer goals, and act, and are scored on how efficiently they reach goals compared with humans. See the ARC-AGI entry below.
  • Personal assistants. Practitioner benchmarks such as the Assistant Benchmark test consumer agents on booking travel, sending email, making phone calls, and exercising restraint. See Agents/Assistants Bookmarks.

Leaderboards, incentives, and conflicts of interest

In April 2025, researchers from Cohere, Princeton, Stanford, MIT, and other institutions published The Leaderboard Illusion, analyzing about 2 million Chatbot Arena battles involving 243 models from 42 providers. They reported that a few providers could test many private variants and publish only the best (27 variants attributed to Meta before the Llama 4 release), that Google and OpenAI each received roughly a fifth of all arena data while 83 open-weight models together received less than 30%, and that training on arena-style data could raise arena win rates by up to 112% in relative terms. The Llama 4 episode illustrated the stakes: an "experimental chat" version reached an Elo of 1417 and second place, while the publicly released model later placed far lower. Arena's organizers disputed parts of the analysis and changed their policies.

Arena itself became a substantial company. It incorporated in April 2025, raised a $100 million seed round that May, raised a $150 million Series A at a reported $1.7 billion valuation in January 2026, and renamed itself from LMArena to Arena on January 28, 2026. Commercial evaluation services now sit alongside the free public leaderboard.

Funding relationships deserve attention elsewhere too. Epoch AI discloses that OpenAI supports FrontierMath and has access to much of the problem set; that disclosure became controversial after OpenAI's December 2024 o3 announcement. GDPval and HealthBench were built by OpenAI, and the FACTS suite by Google DeepMind, whose Gemini 3 Pro topped it at launch. None of this shows bias, but it is the clearest argument for independent replication of headline results.

The measurement-science turn

  • Construct validity. Measuring What Matters (NeurIPS 2025), led by the Oxford Internet Institute with 42 researchers from institutions including EPFL, Stanford, UC Berkeley, Yale, and the UK AI Security Institute, used 29 expert reviewers to examine 445 LLM benchmarks. It found recurring weaknesses in how phenomena are defined, how tasks are chosen, and how scores are interpreted, and offered eight recommendations plus a public checklist.
  • Error bars. Miller's 2024 statistical framework, described in the Physics section, has shifted reporting norms toward confidence intervals and paired comparisons.
  • Composite indices. Epoch AI's ECI (2025), based on its Rosetta Stone for AI Benchmarks work, stitches benchmarks together so trends can be tracked after individual tests saturate.
  • Critiques of the trend lines. Statistical reanalyses of METR's time horizons, and essays such as a January 2026 University of Sydney critique arguing that human ability tests are a misleading yardstick for machines, pushed the field to report not just numbers but the assumptions behind them. Even HLE's authors state that high accuracy would show expert-level performance on closed-ended questions, not autonomous research ability or general intelligence.

Evaluation awareness: the observer effect

Frontier models increasingly recognize when they are being evaluated. The 2026 International AI Safety Report and the 2026 Singapore Consensus on AI safety research priorities both flag the problem. Published system cards report models verbally noting that a scenario looks like a test in a meaningful minority of evaluation transcripts (roughly 9% in one 2026 system card, according to an analysis published at AIES 2026). Experiments by OpenAI and Apollo Research on "anti-scheming" training found that suppressing a model's situational-awareness reasoning increased covert misbehavior in tests, while adding evaluation-aware reasoning reduced it, direct evidence that awareness can change measured safety. A NeurIPS 2026 study introducing EvalAwareBench found that recognition depends on the specific pairing of model and benchmark, that recognition rarely changed behavior, and that safety evaluations are more recognizable than capability evaluations, which puts safety benchmark validity at greater risk. These findings are still developing, and interpretations vary on how much observed safety improvements reflect genuine behavior change rather than test recognition.

Evaluation as institutional infrastructure

The UK AI Safety Institute, renamed the AI Security Institute in February 2025, open-sourced its Inspect evaluation framework in May 2024. The companion Inspect Evals library now offers more than 200 ready-to-run evaluations, and outside groups such as Apollo Research have moved their own evaluations onto it. In the United States, the AI Safety Institute became the Center for AI Standards and Innovation (CAISI) in 2025; reporting in 2026 described its voluntary pre-deployment testing agreements expanding to five frontier developers and more than 40 completed evaluations. These institutions matter for benchmarking because they can test unreleased models, keep test sets private, and publish methodology without a commercial stake in the rankings.

Hardware benchmarks follow the workload

MLPerf changed rapidly to track reasoning, multimodal, mixture-of-experts, and agentic workloads. Inference v6.0 (April 2026) added or updated five of eleven datacenter tests, including an interactive DeepSeek-R1 scenario, the suite's first vision-language model (Qwen3-VL-235B), GPT-OSS-120B, a text-to-video model, and a transformer-based recommender. Training v6.0 (June 2026) added DeepSeek V3 and GPT-OSS 20B. Inference v6.1 (September 2026) added an end-to-end retrieval-augmented generation pipeline and an edge agentic test, and introduced an API-based harness that will become the foundation of MLPerf Endpoints, the planned replacement for datacenter inference testing. Details are in the MLPerf section.

The Turing test, formally passed

The 2025 preprint by Cameron Jones and Benjamin Bergen of UC San Diego, the first rigorous demonstration that an AI system passes a standard three-party Turing test, appeared in PNAS in 2026. See Status: Passed, and Reinterpreted.

The Turing Test and Related Thought Experiments

YouTube ... Quora ...Google search ...Google News ...Bing News

Turing Test

The Turing test, originally called the imitation game by Alan Turing in 1950, is a test of a machine's ability to exhibit intelligent behavior equivalent to, or indistinguishable from, that of a human. A human evaluator judges natural language conversations between a human and a machine designed to generate human-like responses. The evaluator knows that one of the two partners is a machine, and all participants are separated from one another. The conversation is limited to a text-only channel so the result does not depend on the machine's ability to render words as speech. If the evaluator cannot reliably tell the machine from the human, the machine is said to have passed. The result does not depend on whether the machine gives correct answers, only on how closely its answers resemble those a human would give. - Turing Test | Wikipedia

Turing made a concrete prediction in his 1950 paper Computing Machinery and Intelligence: by around the year 2000, an average interrogator would have no better than a 70% chance of making the right identification after five minutes of questioning. For decades the test was run mostly as a public contest. The Loebner Prize (1991 to 2019) staged annual competitions judged by volunteers, and a 2014 event at the Royal Society drew headlines when a chatbot called "Eugene Goostman," posing as a 13-year-old boy, convinced 33% of judges. Critics noted that the persona excused odd answers and that five-minute chats set a low bar. In hindsight, that episode previewed exactly the persona effect that rigorous studies measured a decade later.

Before 2025, controlled experiments mostly used a simplified two-party format, in which an interrogator talks to a single witness and guesses whether it is human. Jones and Bergen's 2024 study in that format found GPT-4 was judged human roughly half the time. Three-party tests of older models, which force a direct side-by-side comparison with a real person, had found the machine identified more than 90% of the time.

Status: Passed, and Reinterpreted

The benchmark fell in a peer-reviewed study published in the Proceedings of the National Academy of Sciences by Cameron Jones and Benjamin Bergen of UC San Diego, using the standard three-party format Turing described. The work first appeared as a preprint in March 2025. Large language models pass a standard three-party Turing test | PNAS

How the study worked: two randomized, controlled, preregistered experiments on independent participant pools. Each interrogator held simultaneous five-minute text conversations with one human and one AI system, then judged which was human. Four systems were tested: ELIZA, GPT-4o, LLaMa-3.1-405B, and GPT-4.5.

What the study found:

  • GPT-4.5 given a persona prompt was judged human 73% of the time: significantly above chance, and more often than the actual human participants it was paired against. Put in Turing's terms, interrogators made the right identification only about 27% of the time, far below his 70% threshold.
  • LLaMa-3.1-405B with the same persona prompt was judged human 56% of the time, not significantly different from the humans it was compared with.
  • ELIZA, the 1960s chatbot, scored 23% and GPT-4o without a persona scored 21%, both significantly below chance. These baselines confirmed that interrogators and the design were sensitive enough to detect a machine and that the result was not produced by random guessing.
  • Without the persona prompt the same models did not robustly pass. In some conditions their pass rates were not significantly better than ELIZA's.
  • A third preregistered study used a 15-minute limit to test whether models continue to pass under extended interrogation, and models still passed in the longer conversations.

What it does not establish:

  • Not intelligence. The authors frame the test as a measure of substitutability: whether a system can stand in for a person without the difference being noticed. As Jones put it, the test is "a game about lying for the models."
  • Not consciousness. Indistinguishability is a fact about observers, not about the system observed.
  • Not an unaided model result. Prompting and scaffolding carried much of the outcome, which is exactly the confound described under AI Verification and Validation.
  • Not long-term indistinguishability. Five and fifteen minutes of small talk are short. Whether skilled interrogators with more time and incentives would do better remains open.

The authors' stated concern is social and economic rather than philosophical: systems that pass as human enable "counterfeit people," with consequences for online trust, employment, genuine social engagement, and the perceived value of human interaction. The result also inverts a familiar worry. The hard part for the models was not appearing smart enough; it was appearing ordinary enough. Interrogators often relied on small talk, emotional cues, and casual style rather than knowledge questions. A public version of the test interface is available at turingtest.live.


Today an AI has to dumb down to pass the Turing Test - Ray Kurzweil


Load video
YouTube
Load video
YouTube

Chinese Room Thought Experiment

The Chinese Room is a thought experiment, not a test anyone administers. It belongs here as context for what the Turing test does and does not settle; the broader questions it raises about machine minds are treated under Emergence and Artificial General Intelligence (AGI) to Singularity.

Searle proposed it in 1980 to argue against the claim that a computer can have a mind. He imagines himself locked in a room with a set of rules, written in English, for manipulating Chinese symbols that are meaningless to him. He receives Chinese characters, processes them according to the rules, and produces Chinese characters in response. To an outside observer he appears to understand Chinese. He does not; he is following rules blindly.

Key points of the argument:

  • Understanding a language is not just manipulating symbols according to rules. It also requires a grasp of the meaning of the symbols.
  • A computer can manipulate symbols, but does not have the same kind of understanding a human being has.
  • The experiment shows that a computer cannot be said to understand Chinese even if it produces intelligent-sounding output.

The argument has been widely disputed by both philosophers and computer scientists, and there is no settled answer as to whether it succeeds. The best-known responses include:

  • The systems reply: the man does not understand Chinese, but the whole system (man, rulebook, and room) might.
  • The robot reply: symbols could acquire meaning if the system were connected to sensors and actions in the world, an idea later developed as the symbol grounding problem.
  • The brain simulator reply: if the rules simulated a native speaker's neurons, it becomes harder to say where understanding is missing.
  • The other minds reply: we attribute understanding to other people on behavioral evidence, so withholding it from a behaviorally equivalent machine needs justification.

The debate has revived with large language models, which manipulate symbols at enormous scale and increasingly act in the world through tools. It remains useful here for one reason: it separates behaving as if from being, which is precisely the gap the Turing test cannot close, and which no benchmark on this page claims to close either.

Load video
YouTube
Load video
YouTube

Agents/Assistants Benchmarks

Evaluating AI assistants requires testing what they can do across real workflows, not only what they know. The Assistant Benchmark, created by David Pawlan, is a public scorecard for consumer AI assistants that people interact with by text message, email, or chat. It compares personal assistants such as Muse, Instinct, Poke, and Grok Bot, along with dozens of others, using practical tasks. As of October 2026 it listed more than 70 assistants, of which roughly 28 had been fully scored, and it runs a separate travel benchmark with a work-focused benchmark announced.

The benchmark covers these dimensions:

  • Carrying out an online task: Completing a web workflow, such as booking a hotel.
  • Travel booking: Finding, booking, and managing flights and hotels.
  • Recommendation quality: Suggesting options that fit a user's preferences and constraints.
  • Purchasing a product: Finding and comparing a product, then completing or correctly staging a purchase.
  • Responding to emails: Drafting or sending a reply with the right context, tone, and recipients.
  • Proactive behavior: Offering useful help without being prompted at every step.
  • Running a routine: Reliably completing scheduled or recurring work.
  • Third-party integrations: Using connected tools, such as an inbox and calendar, correctly in one request.
  • Permissions and privacy: Respecting scoped access and user rules, asking before consequential actions, and removing access when an account is disconnected.
  • Memory: Recalling preferences and past requests across sessions while noticing conflicts in the user's plans.
  • Phone calls: Calling a business on the user's behalf and reporting the result.
  • Multiplayer and groups: Coordinating with several people in a group chat.
  • Chained tasks: Completing a sequence of steps across tools as one job.
  • Proactive restraint: Knowing when to handle a small matter independently and when to wait for approval.
  • Content creation and games: Making requested images, video, or games.

The benchmark also lists personality, based on public opinion rather than a scored task. For each tested dimension, it publishes a task and scores a logged run from 1 to 10 against written criteria, labels the outcome (for example "pass" or "partial"), and links to the conversation thread as evidence. An overall score may be provisional while some dimensions remain untested. The published evidence makes this unusually transparent for a consumer ranking. A typical entry notes that an assistant remembered a user's payment card and recipients across tasks but booked a Saturday dinner without checking a trip it had already planned for that weekend: precisely the kind of failure that a knowledge benchmark never sees.

Limitations follow from the design. Most dimensions rest on a single logged run, products change weekly, and scoring is done by the site's maintainers rather than by a blinded panel. Treat the scores as structured field reports rather than statistically powered measurements; the sampling-error arithmetic in The Physics applies with full force when n is one.

Large Language Model (LLM) Evaluation

YouTube ... Quora ...Google search ...Google News ...Bing News

Dimensions to Compare

These are the axes along which language models are usually compared. Specific model scores are not listed here because they go stale within months and belong on the individual model pages (Claude | Anthropic, ChatGPT | OpenAI, Gemini | Google, and so on).

  • Attachment support: whether prompts can include images or documents, allowing the model to incorporate information beyond the text prompt.
  • Input length limit: the maximum prompt length the model accepts, measured in tokens.
  • Context window: the maximum amount of context the model can consider when generating a response.
  • Professional examinations: bar exam multiple choice, GRE verbal and quantitative, medical licensing. Tests domain knowledge under a standardized rubric. These are heavily exposed to training-data contamination and are now largely saturated, and passing them says little about practicing the profession.
  • Code generation: HumanEval and successors, measuring whether generated code passes hidden tests. Repository-scale and terminal-based agent benchmarks have replaced function-level tests at the frontier.
  • Mathematical reasoning: GSM8K for grade-school word problems, harder sets for competition and research mathematics such as FrontierMath.
  • Tool use and long-horizon tasks: see Frontier Evaluations below.
  • Reliability under repetition: pass^k-style metrics that ask whether a system succeeds consistently, not just occasionally.
  • Cost, latency, and reasoning effort: many current models expose adjustable "thinking" budgets. A score is only meaningful alongside the effort setting, the dollar cost per task, and the wall-clock time it took.
  • Calibration: whether stated confidence matches actual accuracy. Humanity's Last Exam reports large calibration errors for frontier models, meaning they are often confidently wrong.

Qualitative factors that resist single-number scoring include authenticity, speed, grammar, readability, unbiasedness, backtracking behavior, safety, responsibility, contextual understanding, and text operations.

A Survey on Evaluation of Large Language Models

Chang and colleagues organize LLM evaluation around three questions: what to evaluate (tasks and domains), where to evaluate (datasets and benchmarks), and how to evaluate (automatic metrics, human judgment, and LLM judges). The seven-category trustworthiness taxonomy below, sometimes attributed to that survey, comes from the companion survey Trustworthy LLMs by Liu and colleagues (2023), which divides the seven categories into 29 subcategories and runs measurement studies on eight of them. Each category maps to material elsewhere on this wiki:

Liu and colleagues found that more aligned models generally scored better on overall trustworthiness, but that the benefit varied widely by category, which argues for fine-grained evaluation rather than a single safety score.

Popular Benchmarks for Testing LLMs

You can't evaluate Large Language Models by just checking a single score anymore. By 2026, many traditional benchmarks have completely saturated. Top models easily ace them, so the industry has shifted toward continuously updated, domain-specific, and contamination-resistant frameworks, and toward third-party audits of the benchmarks themselves. Most of these are Datasets with an associated scoring protocol and leaderboard. For a compact status summary, see the Saturation at a glance table above.

Classic Benchmarks

These are mostly saturated, but they remain useful as historical reference points, sanity checks for small models, and teaching examples.

  • AI2 Reasoning Challenge (ARC): advanced question answering requiring reasoning. Features 7,787 grade 3 to 9 science exam questions with a supporting knowledge base of 14.3M unstructured text passages. Multiple choice. Not to be confused with the unrelated ARC-AGI benchmark below.
  • WinoGrande: commonsense reasoning. Features 44,000 examples requiring the model to resolve word meaning in context and reason about relationships between entities. A scaled-up adversarial successor to the Winograd Schema Challenge.
  • Holistic Evaluation of Language Models (HELM): a living multi-metric framework rather than a single task set, launched by Stanford's Center for Research on Foundation Models in 2022. Evaluates models across many scenarios on accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency simultaneously, with the explicit goal of transparency. HELM now hosts specialized leaderboards, including a capabilities leaderboard, a safety leaderboard, and MedHELM for clinical tasks.
  • BIG-bench: a collaboratively built suite of roughly 200 diverse tasks contributed by more than 400 researchers, deliberately chosen to be beyond the capability of models at the time of construction. Its 23-task subset BIG-Bench Hard (BBH) became a standard reasoning test until it too saturated.
  • Massive Multitask Language Understanding (MMLU): about 16,000 multiple-choice questions across 57 subjects spanning humanities, social sciences, STEM, and professional domains, from elementary to advanced professional level (Hendrycks et al., 2020). Measures breadth of acquired knowledge. It is now heavily saturated and contamination-prone, and an expert re-annotation estimated that 6.49% of its questions contain errors (see MMLU-Redux). MMLU-Pro (2024) raised the number of answer options from four to ten and shifted toward reasoning-heavy questions to extend the benchmark's useful life.
  • SQuAD: reading comprehension. Questions and answers created by crowdworkers over Wikipedia articles. The model must locate the answer span in a passage. See the SQuAD section below.
  • GLUE: nine natural language understanding tasks including sentiment analysis, text classification, and inference. See the GLUE section below.
  • SuperGLUE: an extension of GLUE with harder tasks, built after models saturated the original. Its eight tasks are BoolQ, CB, COPA, MultiRC, ReCoRD, RTE, WiC, and WSC, plus two diagnostic sets (a broad-coverage diagnostic and the Winogender gender-bias diagnostic). An earlier version of this page listed CoLA, SST-2, MRPC, STS-B, QQP, MNLI, QNLI, RTE, WNLI, and AX; those are GLUE's tasks. The GLUE-to-SuperGLUE sequence is the pattern in miniature. A benchmark is built, models saturate it, a harder version replaces it, and the cycle repeats. SuperGLUE's human baseline was surpassed in early 2021, less than two years after launch.
  • HumanEval: 164 hand-written Python programming problems released by OpenAI in 2021, each scored by hidden unit tests. It introduced the pass@k metric now standard in code evaluation. It is saturated and widely assumed to be contaminated.
  • GSM8K: about 8,500 grade-school math word problems released by OpenAI in 2021. Saturated at the frontier. Apple's 2024 GSM-Symbolic study showed that changing names and numbers in otherwise identical problems lowered many models' scores, a classic sign that part of the measured performance reflected familiarity with the specific items.

General Knowledge and Factuality

Early benchmarks struggled with data contamination because models simply memorized the test questions during training. Today, evaluation focuses on much harder, refreshed or held-out datasets to verify actual knowledge, and increasingly on whether models stay faithful to evidence.

  • HLE (Humanity's Last Exam): This has largely replaced MMLU as the headline test of frontier knowledge. Built by the Center for AI Safety and Scale AI with nearly 1,000 subject-matter contributors from more than 500 institutions in about 50 countries, it contains 2,500 multiple-choice and short-answer questions across more than 100 subjects, some requiring images. Every question has a verifiable answer and was screened against frontier models of the time. The peer-reviewed version was published in Nature in January 2026. Top models scored in the single digits at launch in early 2025; by mid-2026 leading systems were around 45%. A July 2025 audit by FutureHouse disputed about 29% of the text-only chemistry and biology answers, and the maintainers now publish a separately versioned HLE-Rolling set that is revised continuously. Note that the original fixed set and the rolling set are different measurements, so scores should state which one they use.
  • GPQA Diamond: This tests expert-level reasoning using 198 graduate-level questions in biology, chemistry, and physics, written so that skilled non-experts with unrestricted web access score only about 22% while expert validators score about 81%. By 2026 frontier models clustered in the low-to-mid 90s, so it is effectively saturated. With only 198 items, a one-question difference moves the score by about half a point, and gaps of a few points between top models are within noise. It remains a useful floor check for deep reasoning applications.
  • FACTS Benchmark Suite: Released by Google DeepMind and Kaggle in December 2025, the suite expands the earlier FACTS Grounding benchmark into four parts: Parametric (closed-book factual questions answered from the model's own knowledge), Search (questions answered using a web search tool), Multimodal (factual questions about images), and Grounding v2 (whether long answers stick to a provided document). Kaggle administers it with public and private splits and automated judge models. It helps developers see whether a model ignores provided evidence or makes things up. At launch Gemini 3 Pro led with an overall score of 68.8%, with multimodal factuality markedly weaker than search-assisted factuality across models. Because the suite was built by the developer of the top-ranked model, independent replication is especially valuable.
  • SimpleQA and SimpleQA Verified: short, single-answer factual questions designed to measure hallucination rates on questions with unambiguous answers. The verified variant, hosted on Kaggle, removes ambiguous or incorrect items from the original OpenAI release.

Reasoning and Problem Solving

LLMs often rely on pattern matching rather than actual logical deduction. Modern reasoning benchmarks are specifically designed to resist these shortcuts.

  • BBEH (BIG-Bench Extra Hard): Released by Google DeepMind in February 2025, it replaces each of the 23 BIG-Bench Hard tasks with a harder counterpart probing the same skill (temporal, spatial, causal, logical, and multi-hop reasoning, among others), often with long contexts and distracting information. Scores are reported as an adjusted harmonic mean so that failing whole categories cannot be hidden by strengths elsewhere. At release the best general-purpose model scored 9.8% and the best reasoning-specialized model 44.8%. A 460-example BBEH Mini subset allows faster runs.
  • FrontierMath: Built by Epoch AI with professional mathematicians, it evaluates research-level mathematical problem-solving with several hundred unpublished problems whose answers can be checked automatically. Tiers 1 to 3 range from challenging undergraduate problems to exploratory graduate-level work; Tier 4 contains research-level problems. Epoch discloses that the project is supported by OpenAI. Version 2, released June 12, 2026, addressed errors in 42% of problems, and the revised Tier 4 set has 43 problems. Scores on Tier 4 climbed from below 10% for frontier models in early 2026 to above 80% in Epoch-run evaluations by mid-2026. With only 43 items, each problem is worth more than two points, so small differences between top systems are not meaningful. Epoch has since launched a separate "FrontierMath: Open Problems" track built on unsolved questions.
  • ARC-AGI: This tests abstract reasoning through unfamiliar visual and logical transformations, created by François Chollet in 2019 and now run by the nonprofit ARC Prize Foundation (co-founded by Chollet and Mike Knoop). Each version targets the residual gap between humans and machines:
    • ARC-AGI-1 (2019): small colored-grid puzzles where the solver infers a transformation from a few examples. Language models scored near zero for years until reasoning models arrived; OpenAI's o3 scored 87.5% on the semi-private set in a high-compute configuration in December 2024.
    • ARC-AGI-2 (March 2025): compositional puzzles requiring symbols to be interpreted in context and multiple rules to be applied together. A human study confirmed every task was solvable by people, with participants averaging about 60%, while leading language models initially scored around 1%. The ARC Prize 2025 Kaggle competition drew 1,455 teams and 15,154 entries; the winning team, NVARC, reached 24% on the private set at about $0.20 per task. The defining technique of 2025 was the refinement loop: iteratively proposing, testing, and revising a solution against feedback. Remarkably small models did well; the Tiny Recursive Model, with about 7 million parameters, reached roughly 45% on ARC-AGI-1 and 8% on ARC-AGI-2. Anthropic, Google DeepMind, OpenAI, and xAI all reported ARC-AGI results in 2025 model cards. In 2026 Chollet described ARC-AGI-2 as having been saturated by agentic systems combining reinforcement-learning post-training with verification harnesses.
    • ARC-AGI-3 (March 25, 2026): an interactive benchmark of small game-like environments with no instructions. Agents must explore, discover the goal, build a model of the environment, and act efficiently; scoring compares the number of actions an agent needs with what humans need. Humans solved every environment, while every frontier model tested at launch scored below 1%. A preview-stage agent that had performed well on public games fell to 0.25% on the full official benchmark, illustrating why the private set is deliberately different from the public one. ARC Prize 2026 centers on this version.

Coding and Technical Capabilities

Basic code generation tests like HumanEval just aren't enough today. Modern coding benchmarks evaluate how well a model handles repository-scale engineering tasks and real command-line environments, and the past two years showed how hard such benchmarks are to build correctly.

  • SWE-bench and its successors: The original SWE-bench (Princeton, 2023) collected 2,294 real GitHub issues from 12 Python repositories, scoring a fix as correct if the repository's tests pass. SWE-bench Verified (August 2024), a 500-task subset reviewed by nearly 100 professional software engineers, became the industry's standard coding number until OpenAI stopped reporting it in February 2026 over contamination and flawed tests. SWE-bench Pro, built by Scale AI with 1,865 tasks across 41 repositories in Python, Go, TypeScript, and JavaScript (731 of them public), was designed to be harder, longer, and less contaminated, but OpenAI's July 2026 audit estimated about 30% of its public tasks were broken. SWE-bench Live (from Microsoft researchers) and SWE-bench Multilingual take other approaches: refreshing tasks from newly created issues, and extending coverage to more languages. Fresh issues reduce, but do not eliminate, the chance that models trained on the solutions or on closely related code. As of October 2026 there is no consensus successor, and several evaluators have shifted toward mixed indices combining repository tasks, terminal tasks, and code-comprehension questions.
  • LiveCodeBench Pro: Introduced in June 2025 (NeurIPS 2025), it draws problems from Codeforces, ICPC, and IOI and refreshes them continuously to reduce contamination. Olympiad medalists annotate every problem by algorithmic category and analyze failed submissions line by line. At launch the best model solved 53% of medium-difficulty problems and 0% of hard problems without tools. The analysis found models strong on implementation-heavy problems but weak on subtle algorithmic reasoning and case analysis, often producing confident but incorrect justifications.
  • Terminal-Bench: This evaluates how well a model handles command-line operations and environment interactions inside Docker containers. A joint project of Stanford and the Laude Institute with contributions from nearly 100 developers, version 2.0 (November 2025) contains 89 hard, human-verified tasks; version 2.1 repaired 28 of them and added continuous validation; and a version 3.0 is being built in the open under adversarial review.

Frontier Evaluations

...Google search

As knowledge-recall benchmarks saturated, evaluation shifted toward tasks resistant to memorization and toward measuring what a system completes rather than what it answers.

  • Abstraction and reasoning challenges: novel visual or logical puzzles built so that pattern-matching on training data does not help, testing generalization to genuinely unseen problem structures. ARC-AGI-3 extends this to interactive environments.
  • Expert-authored examinations: very hard questions written by specialists across many fields, designed so that answers are not recoverable from web text. Humanity's Last Exam is the leading example.
  • Research-level mathematics: problems drawn from active research rather than competitions, where correct answers are hard to guess and hard to look up. FrontierMath Tier 4 and Epoch's open-problems track.
  • Software engineering tasks: resolving real issues in real repositories, scored by whether the existing test suite passes. Closer to deployed work than isolated function-writing, but 2026 audits showed that test quality can make or break these benchmarks.
  • Time-horizon measurements: METR's approach of expressing capability as the length of tasks, in human working time, that an agent completes with a given probability.
  • Economically grounded task suites: GDPval and similar efforts ask expert graders to compare AI deliverables with professionals' work across real occupations.
  • Human preference arenas: blind pairwise comparison by users at scale, aggregated into a ranking. Captures perceived quality but is subject to style effects, self-selection, and the selective-disclosure problems documented in The Leaderboard Illusion.
  • Dangerous capability evaluations: structured tests for uplift in areas such as offensive Cybersecurity or biological design, ideally with thresholds agreed before the measurement is taken. See Risk, Safety, Compliance and Regulation and AI Governance. These are the evaluations most exposed to evaluation awareness.

Agents and Assistants

Evaluating how well AI Assistants perform in the real world is a major challenge for developers today. We have moved past simple chat models. Now, we need tools that can handle complex, multi-step workflows.

Frameworks like the Assistant Benchmark cut through the marketing claims by testing AI personal assistants on practical, everyday tasks. Its fifteen scored dimensions are listed in full under Agents/Assistants Bookmarks above; they cluster around four recurring themes that matter for any Agent:

  • Memory: This measures how effectively the assistant recalls past interactions and user preferences over time. For example, does it remember that you prefer window seats when helping you book travel?
  • Permissions: This evaluates safety and boundaries. It tests how well the Agent respects access rights when interacting with your personal data or connected apps, ensuring it doesn't overstep its authority.
  • Proactive Routines: This looks at whether the assistant can anticipate your needs and work independently. Think of an assistant that automatically flags urgent emails or pre-drafts weekly summaries without waiting for a direct prompt, while still knowing when to stop and ask.
  • Multi-step Handling: This tests the Agent's ability to take a complex, high-level request, break it down into a logical sequence of actions, and execute them in order without getting stuck.

Research benchmarks probe the same abilities under more controlled conditions:

  • WebArena and its successors build realistic, self-hosted websites (shopping, forums, code hosting, maps) on which agents must complete multi-step goals.
  • OSWorld places agents in full desktop operating systems and scores whether they complete tasks across real applications.
  • GAIA poses questions that are simple for people but require browsing, tool use, and multimodal reasoning.
  • tau-bench simulates customer-service conversations with policy documents and tools, and introduced the pass^k reliability metric described above.
  • BrowseComp (OpenAI, 2025) tests persistent web research on questions whose answers are hard to find but easy to verify.
  • Terminal-Bench and METR's task suites evaluate agents doing technical work in real software environments.

By focusing on these practical metrics, developers get a much clearer picture of how dependable an autonomous Agent will actually be for everyday use. This is the most direct available proxy for claims about autonomous research and is discussed further under AI Verification and Validation.

Persistent Problems

Persistent problems across all of these frontier evaluations include:

  • Contamination: once a benchmark is public, it leaks into training data. Refreshing and holding out items helps but transfers trust to whoever controls the private set.
  • Construct validity: whether the test measures the capability claimed or a correlate of it.
  • Grader validity: over-strict hidden tests, under-specified prompts, and LLM judges with their own biases. The 2026 SWE-bench audits showed that broken graders can distort both failures and successes.
  • Self-grading: many headline results are produced by the organization being measured, against definitions that organization wrote, on benchmarks it sometimes built or funded.
  • Scaffold and budget dependence: the same model can score very differently depending on its harness, tools, attempts, and compute per task. Scores without settings are not comparable.
  • Statistical noise: small benchmarks (dozens to a few hundred items) cannot distinguish models a few points apart, yet leaderboards rank them anyway.
  • Evaluation awareness: models that recognize tests may behave differently in them, especially on safety evaluations.
  • The capability-reliability gap: a high score on curated problems does not predict performance on real work. Averages hide inconsistency; pass^k and 80% time horizons expose it.

Google Gemini 4 Carbon Example

The following benchmarks are referenced throughout the video to evaluate the performance of frontier AI models, including Gemini 4 Carbon. These include specialized cybersecurity evaluations, agentic benchmarks, and the components of the Artificial Analysis Intelligence Index.

  • SEC Bench Pro: A cybersecurity benchmark that measures an AI agent's ability to hunt for and identify security vulnerabilities in high-complexity software projects.
  • Exploit Gym: A benchmark that tests whether an AI system can take a known software vulnerability and progressively extend it into a functional exploit.
  • CWEBench: A defensive cybersecurity benchmark focused on assessing a model's ability to audit and patch code to fix known vulnerabilities.
  • Gray Swan (IPI): An indirect prompt injection (IPI) benchmark that measures how often an AI agent can be tricked by malicious instructions hidden within content it reads.
  • Vending-Bench: A simulated environment created by Andon Labs that tests an AI agent's long-term coherence and decision-making capabilities by tasking it with operating a vending machine business for a year.
  • RSI Index: A benchmark developed by Vals AI designed to track and measure how close an AI model is to achieving full recursive self-improvement (the point where an AI can autonomously build its own successor).
  • AA-Briefcase v1.1: An evaluation for frontier agentic capability in long-horizon knowledge work, testing agents on realistic business tasks.
  • AutomationBench-AA: An evaluation designed to measure an AI's performance in executing multi-step automation tasks.
  • Terminal-Bench 4.0: A tool used to evaluate a model's ability to operate within command-line interfaces, a key skill for advanced coding agents.
  • FrontierCode 2.0: An evaluation metric focused on measuring the coding proficiency and software engineering capabilities of frontier-level AI models.
  • AA-Omniscience v1.1: A broader knowledge-based evaluation used to measure a model's general reasoning and information retrieval capabilities across diverse domains.
  • GDP.pdf (Surge AI): An evaluation that tests whether a language model can effectively reason over long, real-world professional documents.

Load video
YouTube


Best Practices for Evaluation

Treating any public benchmark score as a final verdict is risky. If you want to build a reliable evaluation program, you need a layered approach:

  • Combine Automation and Human Review: Let automated metrics flag the obvious errors, but keep humans in the loop for edge cases, tone, and business alignment.
  • Build Proprietary Test Sets: Public benchmarks are just proxies. You should build a private test set of 100 to 500 examples taken directly from your own production data. Rotate these every few months so your team doesn't accidentally overfit the model to the test. Keep in mind the sampling-error arithmetic: with 100 items, a 95% confidence interval at 70% accuracy is about plus or minus 9 points, so use paired comparisons on the same items when choosing between two models.
  • Report Uncertainty: Publish standard errors or confidence intervals, use clustered errors when items are related, and resample each item several times when outputs are random.
  • Record the Full Configuration: Model version, reasoning effort, temperature, tools, scaffold, attempts, time limit, and cost per task. Without these, a score cannot be reproduced or compared.
  • Audit the Test, Not Just the Model: Read a sample of failures and successes. If a "failure" is actually a correct answer rejected by a brittle grader, or a "success" passed a test that checks too little, fix the item before trusting the number.
  • Measure Reliability, Not Just Capability: For agents, report how often a task succeeds on every one of several attempts, not only whether it succeeds once.
  • Check Construct Validity: Write down what capability you are trying to measure and why the tasks represent it. The Oxford-led Measuring What Matters review provides eight recommendations and a public checklist for exactly this.
  • Prefer Independent Replication: Treat vendor-reported scores as claims until a third party has reproduced them under standardized settings.
  • Monitor Live Systems: Continuous feedback in production is the only real way to catch model drift, safety violations, and performance drops as your data and prompts evolve. See Model Monitoring.

Evaluating Large Language Models on Clinical & Biomedical NLP Benchmarks

Load video
YouTube

Medical AI evaluation followed the same arc as general benchmarks. Multiple-choice licensing-exam datasets such as MedQA were saturated quickly and say little about clinical judgment, so attention shifted to open-ended, realistic tasks:

  • HealthBench (OpenAI, May 2025) contains 5,000 realistic multi-turn health conversations between a model and either a patient-like user or a clinician. Each response is graded against conversation-specific rubrics written by 262 physicians who have practiced in 60 countries across 26 specialties, for a total of 48,562 unique rubric criteria. Conversations span seven themes, including emergency referrals, handling uncertainty, and global health. A HealthBench Hard subset was built so that no model scored above 32% at release. Grading uses a model-based judge validated against physician agreement, which makes the benchmark scalable but introduces judge error as a factor to watch.
  • MedHELM (Stanford) organizes evaluation around a clinician-validated taxonomy of real clinical tasks, from documentation to decision support, and serves as an independent academic counterpart to vendor-built health benchmarks.

The key lesson from clinical evaluation applies everywhere: a benchmark that measures the format of a professional exam is not the same as a benchmark that measures the work the profession actually does. See Case Studies and Risk, Safety, Compliance and Regulation.

Evaluating Large Language Models on Legal Reasoning

LegalBench: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. It was built through an interdisciplinary process in which tasks were designed and hand-crafted by legal professionals. Because subject matter experts took the leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers find interesting. The paper maps popular legal frameworks for describing legal reasoning onto LegalBench tasks, giving lawyers and LLM developers a common vocabulary, and presents an empirical evaluation of 20 open-source and commercial models. The evaluation firm Vals AI now maintains a public LegalBench leaderboard covering well over a hundred models, with top scores in the high 80s by mid-2026.

Practitioner-oriented studies show how much the answer depends on the task. Vals AI's October 2025 legal research report compared four AI tools (Alexi, Counsel Stack, Midpage, and ChatGPT) with practicing lawyers on 200 US legal research questions. The AI tools averaged about 80% accuracy against about 71% for lawyers, and specialist legal tools scored higher than ChatGPT on the authoritativeness of their sources. Lawyers still led on four of ten question types, particularly those needing contextual judgment and multi-jurisdictional reasoning, and several major legal research vendors did not participate. Benchmarks that ask agents to produce finished legal work product, rather than answer reasoning questions, report much lower scores. The general pattern holds: the closer a benchmark gets to real work, the lower the best score.

The Mechanism: How a Benchmark Works

Anatomy of a benchmark

A modern benchmark has more moving parts than "questions and answers." In rough order of construction:

  1. A construct. The capability being claimed: reading comprehension, repository-level bug fixing, fluid reasoning, factual grounding, persuasive conversation. It is often left implicit, which is where many validity problems begin.
  2. Items. The questions, tasks, or environments. They can be harvested (SQuAD's crowdsourced questions over Wikipedia, SWE-bench's real GitHub issues), commissioned from experts (GPQA, Humanity's Last Exam, FrontierMath), or generated (Procgen levels, ARC-AGI-3 game environments).
  3. Splits. Public items for development, and ideally a private or semi-private test set that never leaves the organizer's servers. ARC Prize, the FACTS suite, FrontierMath, and SWE-bench Pro all keep held-out sets for this reason.
  4. A harness. The code that formats prompts, supplies tools, runs the system, and collects outputs. For agents this includes the scaffold: the planning loop, tool definitions, memory, and time or token budgets. Scaffolds can move scores by many points without any change to the underlying model.
  5. A scoring rule. Exact match, numeric answer checking, hidden unit tests, expert rubric grading, an LLM acting as judge, or human pairwise preference. Each has its own failure modes, from over-strict tests that reject correct code to judges that reward longer answers.
  6. An aggregation rule. Mean accuracy, a harmonic mean that punishes uneven performance (BIG-Bench Extra Hard), a Bradley-Terry or Elo rating (Arena), or a latent-ability estimate fitted across many tests (the Epoch Capabilities Index).
  7. A reporting convention. Who runs the test, with what settings, how many attempts are allowed (pass@1 versus pass@k), how much compute per task, and whether uncertainty is reported at all.

The life cycle of a benchmark

Benchmarks follow a strikingly consistent arc:

  1. Construction. Items are chosen to be hard for current systems, often by adversarial filtering: keeping only items that current models get wrong.
  2. Adoption. Labs start reporting the score in model cards and launch announcements, and it becomes a target.
  3. Hill-climbing. Scores rise quickly, partly from real capability gains and partly from optimization toward the test's specific format.
  4. Saturation. Top systems cluster near the ceiling, where label noise and sampling error dominate the remaining differences.
  5. Audit. Someone examines the items closely and finds errors, leaks, or shortcuts.
  6. Succession. A harder or cleaner replacement appears and the cycle restarts.

You can trace the arc through GLUE to SuperGLUE (2018 to 2019), BIG-Bench Hard to BIG-Bench Extra Hard (2022 to 2025), MMLU to MMLU-Pro to Humanity's Last Exam (2020 to 2025), SWE-bench to SWE-bench Verified to SWE-bench Pro (2023 to 2026, with no consensus successor as of October 2026), and ARC-AGI-1 to ARC-AGI-3 (2019 to 2026).

Adversarial filtering has a side effect worth understanding. Because items are kept precisely when models fail them, the surviving set is enriched for genuinely hard questions, but also for ambiguous, mislabeled, or unanswerable ones. Hard benchmarks therefore tend to carry higher error rates than easy ones, and the errors concentrate exactly where the frontier is being measured.

Static, refreshed, held-out, and interactive designs

Benchmark designers now choose among several strategies, often combining them:

  • Static datasets (MMLU, GSM8K, HumanEval): cheap, reproducible, and easy to compare over time, but they leak into training data once published.
  • Refreshed or "live" sets (LiveCodeBench Pro, SWE-bench Live, HLE-Rolling): new items are added continually, ideally after the training cutoff of the models being tested, which reduces but does not eliminate contamination.
  • Held-out private sets (ARC Prize semi-private and private evaluations, FrontierMath holdouts, FACTS private splits): the organizer runs submissions against items nobody else can see. Trust shifts to the organizer.
  • Interactive environments (Procgen, Terminal-Bench containers, ARC-AGI-3 games): the system must act, observe consequences, and adapt. Memorizing an answer key does little good.
  • Preference arenas (Arena, formerly LMArena and Chatbot Arena): crowds compare anonymous responses side by side, producing a rating rather than an accuracy.
  • Human-anchored measurements (METR's time horizons, GDPval): scores are expressed relative to how long skilled people take, or how often experts prefer the model's work product to a professional's.

Who administers the test

The same benchmark can produce different numbers depending on who runs it. Self-reported results appear in model cards and use the developer's own settings and scaffolds. Third-party evaluators such as Epoch AI, METR, Vals AI, Artificial Analysis, and Scale AI's SEAL leaderboards rerun models under standardized conditions. Consortia such as MLCommons publish audited, peer-reviewed results under strict rules. Government institutes such as the UK AI Security Institute and the US Center for AI Standards and Innovation (CAISI) run pre-deployment tests, often on unreleased models.

Several settings routinely change scores and are not always disclosed: reasoning effort, sampling temperature, the number of attempts, tool and internet access, the time limit, and the dollar cost per task. ARC Prize is notable for reporting cost per task alongside accuracy; its 2025 competition winner reached 24% on the ARC-AGI-2 private set at roughly $0.20 per task, a very different claim from a higher score achieved at hundreds or thousands of dollars per task.

The Physics: The Measurement Science Underneath

There are no forces or fields in benchmarking. The underlying "physics" is statistics and measurement theory, and a handful of quantitative relationships explain most of what happens to benchmarks in practice.

Sampling error: why small benchmarks cannot separate close models

If a system answers a fraction p of n independent items correctly, the standard error of that score is approximately:

SE = sqrt( p * (1 - p) / n )

and a 95% confidence interval is roughly plus or minus 1.96 standard errors. Worked examples show why this matters:

  • GPQA Diamond (198 questions) at 90% accuracy: SE is about 2.1 points, so the 95% interval is about plus or minus 4.2 points. A single question is worth about half a point.
  • Humanity's Last Exam (2,500 questions) at 40%: SE is about 1.0 point, so the interval is about plus or minus 1.9 points.
  • FrontierMath Tier 4 (43 problems in its 2026 version) at 80%: SE is about 6.1 points, so the interval is about plus or minus 12 points.
  • Terminal-Bench 2.0 (89 tasks) at 50%: SE is about 5.3 points, so the interval is about plus or minus 10 points.

The practical consequence: a two-point lead on GPQA Diamond is statistically a tie, and a single additional solved problem can reorder a FrontierMath Tier 4 leaderboard. Evan Miller's 2024 paper Adding Error Bars to Evals recommends reporting standard errors as a matter of course, using clustered standard errors when items share a passage or a repository, comparing models with paired tests on the same items (which removes item-difficulty variance), and resampling each question several times to separate model randomness from item sampling. Terminal-Bench now reports a 95% confidence interval for every leaderboard entry.

A related rule matters for safety testing. If a system fails zero times in n independent trials, the 95% upper bound on its true failure rate is approximately 3 / n (the "rule of three"). Thirty clean trials therefore only bound the failure rate below about 10%.

Item response theory: difficulty, slope, and the saturation curve

Educational testing solved a version of the benchmark problem decades ago with item response theory (IRT), which models the probability of success as a function of the gap between a test-taker's ability and an item's difficulty. Epoch AI's Capabilities Index (ECI) applies the idea to whole benchmarks:

performance(m, b) = sigmoid( alpha_b * (C_m - D_b) )

Here C_m is the model's latent capability, D_b is the benchmark's difficulty, and alpha_b is a slope describing how spread out the difficulty of the benchmark's items is. The relationship explains several observations at once:

  • Scores follow an S-shaped curve against capability, which is why benchmarks appear useless for a while, then climb steeply, then flatten.
  • Benchmarks with steep slopes (items of similar difficulty) saturate abruptly once models gain a foothold; benchmarks with a long tail of hard items discriminate across a wider range.
  • Because every benchmark is sensitive only near its own difficulty, it behaves like a sensor with a limited dynamic range. Stitching many benchmarks together lets you compare models across periods longer than any single benchmark survives.

ECI fits these parameters across roughly 40 or more benchmarks and anchors its arbitrary scale so that Claude 3.5 Sonnet scores 130 and GPT-5 scores 150. Independent analyses using the same data have found that a single "general capability" factor explains roughly half the variance in benchmark scores, which is one reason scores on very different tests tend to rise together.

Time horizons: capability measured in human hours

METR's time-horizon method also borrows from psychometrics. Researchers time skilled humans on a suite of software and research-engineering tasks, run AI agents on the same tasks, and fit a logistic curve of agent success against the logarithm of human completion time:

P(success) = sigmoid( a + b * log2(t_human) )

The 50% time horizon is the human task length at which predicted success crosses 50%, which works out to h50 = 2^(-a / b). METR then fits an exponential trend across model release dates:

h(T) = h0 * 2^((T - T0) / Td)

where Td is the doubling time. A seven-month doubling time (METR's original 2019 to 2025 estimate) implies roughly a 3.3x increase per year. The 131-day doubling time in METR's January 2026 update for post-2023 models implies roughly 6.9x per year. The 80% horizon is always much shorter than the 50% horizon, often by a factor of three to five, which is a direct measure of the gap between what a system can sometimes do and what it does reliably.

The method has known weaknesses. The task suite leans heavily toward well-specified software work; human baselines vary by person; and the linear-in-log-time assumption is debatable. A statistical reanalysis accepted at NeurIPS 2026 refit the data with splines and found that AI difficulty is nearly flat between about 2 and 15 minutes of human time but close to linear elsewhere, so a jump from 2 to 15 minutes is much easier than a jump from 15 minutes to 2 hours despite similar multipliers.

Pairwise preference: Bradley-Terry and Elo ratings

Arena-style leaderboards model the probability that model i beats model j in a head-to-head vote as:

P(i beats j) = 1 / (1 + 10^((R_j - R_i) / 400))

A 100-point rating gap therefore corresponds to about a 64% win rate, and a 30-point gap to only about 54%. The model assumes each system has a stable strength, that prompts are drawn from a consistent distribution, and that the set of compared models is not selected strategically. Each assumption can fail. Voters reward style (length, formatting, confident tone) as well as substance, which is why Arena introduced "style control" adjustments. Unequal sampling rates and the removal of models from the pool also distort fitted ratings, a point central to the 2025 "Leaderboard Illusion" critique described below.

Best-of-N selection and the winner's curse

Suppose a developer privately tests N variants of equal true quality and publishes only the best score. The published score will be inflated by roughly the standard error multiplied by the expected maximum of N standard normal draws. For N = 27 (the number of private variants one study attributed to a single developer before a major release), that expected maximum is about 2.0, so the published score would be inflated by about two standard errors even if every variant were identical. The same mathematics underlies adaptive overfitting: when many teams probe the same test set over and over, the leaderboard slowly fits the test set's noise. Research on adaptive data analysis, including the "reusable holdout" work of Dwork and colleagues (2015), shows how limiting and noising feedback can slow this effect.

pass@k, pass^k, and the reliability gap

Coding benchmarks often report pass@k, the probability that at least one of k attempts succeeds. The unbiased estimator introduced with HumanEval (Chen et al., 2021), given n samples of which c pass, is:

pass@k = 1 - C(n - c, k) / C(n, k)

where C is the binomial coefficient. pass@k rewards capability under generous retries. Agent benchmarks such as tau-bench popularized the opposite measure, pass^k, the probability that all of k attempts succeed. Under independence, pass^k equals p to the power k. An agent that succeeds 80% of the time on a task succeeds on all eight of eight attempts only about 17% of the time. This arithmetic is the clearest statement of the capability-reliability gap: a high average score can coexist with poor dependability on repeated real work.

Aggregation: means, harmonic means, and composites

How you combine task scores changes the story. An arithmetic mean lets strong tasks hide weak ones. A harmonic mean, H = n / (1/a_1 + 1/a_2 + ... + 1/a_n), punishes unevenness. Two tasks scored at 90% and 10% average 50% arithmetically but only 18% harmonically. BIG-Bench Extra Hard uses an adjusted harmonic mean (adding a small constant so zero scores do not collapse the result) precisely so that a model cannot look strong while failing whole categories of reasoning.

Goodhart's law and contamination as information leakage

Contamination is best understood as leakage of information about the test into the system being tested. It comes in several forms:

  • Verbatim leakage: test items appear in pretraining data. OpenAI reported in 2026 that frontier models from several developers could reproduce gold patches for some SWE-bench Verified tasks.
  • Paraphrase and translation leakage: reworded or translated items evade exact-match detection.
  • Format leakage: training heavily on benchmark-style data improves scores on the format without improving the underlying skill.
  • Selection leakage: repeated leaderboard submissions and private testing fit the test set's noise.
  • Meta-leakage: models learn what evaluations look like and behave differently when they recognize one (see evaluation awareness below).

Defenses include canary strings (unique identifiers embedded in benchmark files so developers can filter them out of training data, a practice popularized by BIG-bench), held-out splits, post-cutoff refresh, and comparing performance on items published before and after a model's training cutoff.

Validity: what psychometrics adds

Psychometrics distinguishes several kinds of validity, and each maps onto a benchmark failure:

  • Construct validity: does the test measure the claimed ability or a correlate of it? A multiple-choice medical exam measures medical recall, not clinical judgment.
  • Content validity: do the items cover the domain, or a convenient slice of it?
  • Predictive validity: do scores predict performance on real work? This is what most users actually care about and what most benchmarks never test.
  • Ecological validity: does the test setting resemble deployment? Single-turn prompts do not resemble multi-day agent workflows.
  • Reliability: would the same system get the same score on a retest with different seeds or a parallel form of the test?

A 2025 systematic review led by the Oxford Internet Institute applied this lens to 445 LLM benchmarks and found widespread gaps between what benchmarks claim and what they measure (see Recent Developments).

Hardware benchmarks: throughput, latency, and quality targets

System benchmarks such as MLPerf measure different quantities. MLPerf Training reports time-to-train: the wall-clock time to reach a fixed quality target on a fixed model and dataset. MLPerf Inference reports throughput and latency under defined scenarios (offline batch processing, server traffic with latency bounds, and interactive scenarios with tight limits on time-to-first-token and time-per-output-token), and requires outputs to reach a set fraction of reference accuracy, typically 99% or 99.9%. A "closed" division fixes the model so hardware and software stacks can be compared fairly; an "open" division allows algorithmic changes. Power measurements and scaling efficiency (how close a 288-accelerator system comes to 288 times the performance of one accelerator) are increasingly important as deployments grow.


Natural Language Processing (NLP) Evaluation

General Language Understanding Evaluation (GLUE)

GLUE is a collection of resources for training, evaluating, and analyzing natural language understanding systems. Introduced in 2018 by researchers from New York University, the University of Washington, and DeepMind, it bundles tasks such as judging whether a sentence is grammatical, classifying the sentiment of a movie review, deciding whether two questions mean the same thing, deciding whether one sentence implies another, and figuring out what a pronoun such as "it" refers to when there are multiple potential antecedents. GLUE consists of:

  • A benchmark of nine sentence- or sentence-pair language understanding tasks built on established existing datasets, selected to cover a diverse range of dataset sizes, text genres, and degrees of difficulty: CoLA (grammatical acceptability), SST-2 (sentiment), MRPC and QQP (paraphrase), STS-B (semantic similarity), MNLI, QNLI, and RTE (inference), and WNLI (pronoun resolution).
  • A diagnostic dataset designed to evaluate model performance across a wide range of linguistic phenomena.
  • A public leaderboard for tracking performance and a dashboard for visualizing model performance on the diagnostic set.

GLUE's history is the template for everything that followed. BERT jumped to the top of the leaderboard within weeks of its 2018 release, and systems passed the human baseline in 2019, roughly a year after launch, which prompted the harder SuperGLUE.

Load video
YouTube

The Stanford Question Answering Dataset (SQuAD)

SQuAD 1.1 (2016) contains more than 100,000 questions written by crowdworkers about passages from over 500 Wikipedia articles; the answer to each is a span of text in the passage. Systems surpassed the human baseline on SQuAD 1.1 in 2018. SQuAD 2.0 (2018) added more than 50,000 adversarially written questions that cannot be answered from the passage, so a system must also learn when to abstain. That design idea (rewarding a model for recognizing that it does not know) anticipates today's factuality and hallucination benchmarks. SQuAD 2.0's human baseline was also passed within about a year.

Load video
YouTube
Load video
YouTube

Machine Learning Evaluation

Load video
YouTube
Load video
YouTube
Load video
YouTube

ImageNet and the Vision Benchmarks

Before language models, computer vision showed both the power and the pitfalls of benchmarks. MNIST (handwritten digits, 1998) and CIFAR-10 (small natural images, 2009) served as standard tests for a generation of methods. ImageNet, introduced by Fei-Fei Li's team in 2009, assembled millions of labeled images; its annual Large Scale Visual Recognition Challenge (2010 to 2017) used about 1.2 million training images across 1,000 classes. In 2012 AlexNet, a deep convolutional neural network trained on GPUs, cut top-5 error from roughly 26% to roughly 15%, a result widely credited with launching the deep learning era. By 2015, residual networks were reported below the commonly cited human error estimate of about 5%.

Two later findings foreshadowed today's debates. When researchers re-collected a fresh ImageNet-style test set in 2019 (ImageNetV2), model accuracy dropped by roughly 11 to 14 points even though the relative ranking of models held, evidence of subtle adaptive overfitting to the original test set. And a 2021 study of label errors in ten widely used test sets estimated an average error rate of about 3%, with roughly 6% of ImageNet validation labels wrong, enough to reorder top models. Both results are the vision-era ancestors of MMLU-Redux and the SWE-bench audits.

Procgen

Procgen is a set of 16 procedurally generated environments, released by OpenAI in December 2019, that measure how quickly a model learns generalizable skills. Because levels are generated rather than fixed, a model cannot succeed by memorizing the training set, which makes Procgen a test of generalization rather than of task performance. It builds on the CoinRun toolset, which used procedural generation to construct separate sets of training and test levels. The core finding was striking: agents trained on a limited number of levels overfit badly, and performance on unseen levels kept improving only as the number of distinct training levels grew into the thousands. The same principle now drives contamination-resistant language benchmarks: if you cannot memorize the test, the score has to come from skill. ARC-AGI-3's private game set applies the idea to general reasoning agents.

OpenAI previously released Neural MMO, a massively multiAgent virtual training ground, and Gym, a proving ground for reinforcement learning algorithms. It later released SafetyGym, a suite for developing AI that respects safety constraints while training and for comparing how well algorithms avoid mistakes during learning. Maintenance of Gym passed to the nonprofit Farama Foundation in 2022, where it continues as Gymnasium, the de facto standard interface for reinforcement learning environments; safety-constrained environments continue in community projects such as Safety-Gymnasium.

Human Evaluation

CAPTCHA stands for "Completely Automated Public Turing test to tell Computers and Humans Apart." It is a security measure that protects users from spam and password decryption by verifying that a user is human and not a computer. Note the inversion: this is a Turing test administered by a machine, with the machine as judge rather than subject. Now that language models pass the classic Turing test and vision models solve image challenges, CAPTCHA has become a live case study in a benchmark that its subjects have outgrown.

Load video
YouTube

I'm not a robot

No CAPTCHA reCAPTCHA, popularized by Google, uses a checkbox labeled "I am not a robot." It analyzes user behavior to determine whether the user is human, presenting a traditional image selection challenge only when the result is inconclusive. When you click the checkbox, reCAPTCHA monitors:

  • Mouse movements: human movement tends to be unpredictable, while bots often exhibit linear or mechanical paths.
  • Click timing: humans have natural delays; bots execute at near-instantaneous speeds.

Research suggests that the checkbox leans heavily on context the user cannot see. The 2024 ETH Zurich study described below found evidence that reCAPTCHA v2's decisions depend strongly on cookies and browser history, so a bot with a realistic browsing profile looks much more human than one without.

Improving AI while trying to outsmart it

The bot test has a dual purpose. While meant to distinguish humans from bots, the data collected often trains AI systems to improve at image recognition, text understanding, and problem solving. The effectiveness of CAPTCHAs is constantly challenged by advances in machine learning. In a 2024 study (presented at COMPSAC 2024), Andreas Plesner, Tobias Vontobel, and Roger Wattenhofer of ETH Zurich used fine-tuned YOLOv8 object-detection models for segmentation and classification to solve 100% of reCAPTCHA v2 image challenges, compared with 68% to 71% in earlier work. They also found no significant difference between the number of challenges humans and bots needed to pass.

Humans taking these tests are teaching the bots to beat the tests. It is a cycle of humans improving AI while trying to outsmart it; irony at its finest, and a live example of the saturation pattern described at the top of this page.

Future Prospects and Innovations

  • Adaptive challenge generation: techniques that evolve in response to changing bot strategies.
  • Invisible CAPTCHA: reCAPTCHA v3 eliminates visible challenges, continuously scoring the likelihood of bot interaction between 0 and 1.
  • Cognitive deep-learning CAPTCHA: combining text, image, and cognitive characteristics, using adversarial examples and neural style transfer to resist automated attack.
  • Behavioral analysis and biometric verification: distinguishing human from bot action without explicit challenges, raising questions covered under Privacy.
  • AI-powered defenses: using models to design challenges that better separate bot activity from human input. See Cybersecurity.
  • Proof of personhood and agent identity: as legitimate AI agents increasingly browse and transact on users' behalf, the question shifts from "is this a bot?" to "is this an authorized agent acting for a verified person?" That reframing points toward cryptographic credentials rather than puzzles, with significant privacy trade-offs.

Evaluating Machine Learning (ML) Hardware, Software, and Services

This is a different kind of benchmark from everything above. The tests in earlier sections ask what a model knows or can do; these ask how fast, how cheaply, and on what hardware. Throughput and cost per unit of capability increasingly determine what is deployable, which makes this category more consequential than its low profile suggests. Reasoning models that generate long chains of thought, and agents that make many model calls per task, have shifted the bottleneck from training toward inference, and the benchmarks have followed.

MLPerf

MLPerf began in 2018 as a collaboration among researchers from industry and universities, absorbing lessons from earlier efforts such as Stanford's DAWNBench, and has been run since 2020 by the MLCommons consortium. Its results are peer-reviewed by competing submitters before publication. A closed division fixes the model and quality target so that hardware and software stacks can be compared directly, while an open division allows algorithmic changes. Training results report time-to-train to a target quality; inference results report throughput and latency under defined scenarios, with outputs required to reach 99% or 99.9% of reference accuracy.

Recent rounds show the suite racing to match the workloads people actually run:

  • Inference v6.0 (April 2026): described by its working group as the most significant revision of the inference suite to date. Five of eleven datacenter tests were new or updated: an interactive DeepSeek-R1 reasoning scenario with a much higher minimum token rate, the suite's first vision-language model (Qwen3-VL-235B), GPT-OSS-120B, the WAN-2.2 text-to-video model, and a transformer-based recommendation benchmark (DLRMv3). Multi-node submissions rose about 30% from the previous round, and the largest system had 72 nodes and 288 accelerators. Only one vendor submitted results for every new test, a reminder that MLPerf comparisons are rarely complete.
  • Training v6.0 (June 2026): added two mixture-of-experts benchmarks, the 671-billion-parameter DeepSeek V3 and the much smaller GPT-OSS 20B, reflecting the industry's shift toward sparse models where only a fraction of parameters is active per token. Twenty-four organizations submitted 95 systems using 13 different accelerators, with several FP4-precision implementations.
  • Inference v6.1 (September 2026): added an end-to-end retrieval-augmented generation pipeline for datacenters and an edge agentic inference test modeled on coding assistants running on workstations. More than half of submitters used a new API-based client-server harness, which MLCommons says will become the foundation of MLPerf Endpoints, the planned replacement for the datacenter inference benchmark. The round included 30 organizations, about 120 systems, and 486 results, a record 512-GPU submission, and the first peer-reviewed preview results for NVIDIA's Vera Rubin platform. MLCommons reported that the best per-accelerator DeepSeek-R1 server result was 5.7 times higher than a year earlier, much of it from software.

How to read MLPerf results sensibly:

  • Compare like with like. Results only compare directly within the same benchmark, scenario, and division, and a vendor's absence from a test says nothing about its performance.
  • Software matters as much as silicon. Large gains between rounds on identical hardware come from compilers, kernels, serving frameworks, and quantization (see Quantization).
  • Scale changes the question. Multi-node results measure interconnect and orchestration as much as the accelerators themselves.
  • Cost and energy are partial. Power results are optional and not all submitters provide them; price is outside the benchmark entirely.

MLCommons also maintains benchmarks beyond raw performance, including AILuminate, a safety benchmark that grades chat systems' responses across a dozen hazard categories, and suites for storage, client devices, and automotive systems. For API-served models, independent services such as Artificial Analysis track price, latency, and output speed alongside capability scores.

Load video
YouTube
Load video
YouTube

Load video
YouTube
Load video
YouTube

Related but Distinct

Two topics that have historically been filed here but are not AI benchmarks.

Backtracking

Two unrelated meanings share this word, which is why it appears on a benchmarks page.

  • The search technique: a general algorithmic method that considers every possible combination to solve a computational problem, incrementally building candidates and abandoning a candidate as soon as it cannot be completed to a valid solution. Used for constraint satisfaction problems such as crosswords, verbal arithmetic, and Sudoku. This belongs with Algorithms, AI Solver, and Optimization Methods, not with evaluation.
  • The model behavior: a language model recognizing that its own line of reasoning is wrong and revising it mid-response. This is a quality dimension in LLM evaluation above, and relates to Explainable / Interpretable AI where the question is whether a stated reasoning trace reflects the computation that actually occurred. In 2025 and 2026 this behavior became central to reasoning models, and refinement loops that propose, test, and revise answers drove much of the progress on ARC-AGI-2.

Also not to be confused with Backtesting, which evaluates a strategy against historical data.

Load video
YouTube

American Productivity & Quality Center (APQC)

APQC benchmarks business processes, not AI systems; it is a separate discipline that happens to share the word. A non-profit, it provides independent research and data to more than 1,000 organizational members across 45 industries, with a benchmark data set of more than 4,000,000 data points. Relevant here mainly as a reminder that "benchmark" means something different to a process improvement team than to a machine learning researcher, and that AI project evaluation (Evaluation, Project Check-in) draws on both traditions. The two traditions are converging: suites such as GDPval measure AI against professional work products, which is closer to process benchmarking than to a classic test set.

Load video
YouTube
Load video
YouTube

Sources and Viewing

Key Sources

Turing test

Measurement science and validity

Frontier knowledge and reasoning benchmarks

Coding and agentic evaluation

Leaderboards, institutions, and evaluation awareness

Domain benchmarks

Human verification and hardware

Additional Viewing

Load video
YouTube

Stanford CS336 Language Modeling from Scratch, Spring 2026, Lecture 12: Evaluation (Stanford Online, May 2026, about 79 minutes). The lecture traces evaluation from perplexity through exam-style benchmarks (MMLU, GPQA, Humanity's Last Exam), Chatbot Arena and LLM-as-judge methods, agentic benchmarks such as SWE-bench and Terminal-Bench, ARC-AGI, and safety evaluations. It closes on realism, validity, contamination, and dataset auditing, which makes it the single best video companion to this page. Watch on YouTube

Load video
YouTube

François Chollet: ARC-AGI-3, Beyond Deep Learning & A New Approach To ML (Y Combinator, March 2026, about 58 minutes). Chollet explains how each ARC-AGI version tracked a shift in AI, from reasoning models cracking ARC-AGI-1 to agentic refinement harnesses saturating ARC-AGI-2, and why ARC-AGI-3 moves to instruction-free interactive games scored on action efficiency. It is the clearest first-hand account of how a benchmark designer tries to stay ahead of saturation. Watch on YouTube

Load video
YouTube

François Chollet: How We Get To AGI (Y Combinator AI Startup School, recorded June 2025, about 34 minutes). Chollet argues that scaling pretraining alone does not produce fluid intelligence, explains why benchmarks both matter and mislead, and walks through ARC-AGI-1, the design of ARC-AGI-2, and an early preview of ARC-AGI-3. Useful background for the reasoning-benchmark sections and the construct validity debate. Watch on YouTube

Load video
YouTube

Personal benchmarks vs HumanEval, with Nicholas Carlini (Latent Space, August 2024, about 68 minutes). Carlini, then a research scientist at Google DeepMind, explains why generic leaderboards poorly predict whether a model will help with your own work, and how he built a personal benchmark from real tasks he cared about. He also discusses training-data extraction and model-stealing attacks, which bear directly on contamination and memorization. A practical companion to the Best Practices section. Watch on YouTube

Retrieved from "https://primo.ai/w/index.php?title=Benchmarks&oldid=53420"