Difference between revisions of "Benchmarks"

From
Jump to: navigation, search
m
m
Line 1: Line 1:
 +
__NOTOC__
 
{{#seo:
 
{{#seo:
 
|title=PRIMO.ai
 
|title=PRIMO.ai

Revision as of 10:21, 10 September 2026

YouTube ... Quora ...Google search ...Google News ...Bing News

Three pages on this wiki cover overlapping ground. The division of labor:

A recurring caution applies to everything below. Benchmarks saturate. The history of the field is a sequence of tests that fell and were then dismissed as never having measured intelligence — checkers, chess, Jeopardy!, Go, protein structure prediction, competition mathematics, and most recently the Turing test itself. A high score is evidence about a test, not a verdict on a system. The constraints that make a benchmark meaningful — contamination resistance, construct validity, independent administration — are discussed on AI Verification and Validation.

You can’t improve what you don’t measure. — Peter Drucker


The Turing Test and Related Thought Experiments

YouTube ... Quora ...Google search ...Google News ...Bing News

Turing Test

The Turing test, originally called the imitation game by Alan Turing in 1950, is a test of a machine's ability to exhibit intelligent behavior equivalent to, or indistinguishable from, that of a human. A human evaluator judges natural language conversations between a human and a machine designed to generate human-like responses. The evaluator knows that one of the two partners is a machine, and all participants are separated from one another. The conversation is limited to a text-only channel so the result does not depend on the machine's ability to render words as speech. If the evaluator cannot reliably tell the machine from the human, the machine is said to have passed. The result does not depend on whether the machine gives correct answers, only on how closely its answers resemble those a human would give. - Turing Test | Wikipedia

Status: Passed, and Reinterpreted

The benchmark fell in a peer-reviewed study published in the Proceedings of the National Academy of Sciences by Cameron Jones and Benjamin Bergen of UC San Diego, using the standard three-party format Turing described. Large language models pass a standard three-party Turing test | PNAS

What the study found:

  • GPT-4.5 given a persona prompt was judged human 73% of the time — significantly above chance, and more often than the actual human participants it was paired against.
  • ELIZA, the 1960s chatbot, scored 23% as a manipulation check, confirming that interrogators and the design were sensitive enough to detect a machine and that the result was not produced by random guessing.
  • Without the persona prompt the same models did not robustly pass. In some conditions their pass rates were not significantly better than ELIZA's.
  • A third preregistered study used a 15-minute limit to test whether models continue to pass under extended interrogation.

What it does not establish:

  • Not intelligence. The authors frame the test as a measure of substitutability — whether a system can stand in for a person without the difference being noticed.
  • Not consciousness. Indistinguishability is a fact about observers, not about the system observed.
  • Not an unaided model result. Prompting and scaffolding carried much of the outcome, which is exactly the confound described under AI Verification and Validation.

The authors' stated concern is social and economic rather than philosophical: systems that pass as human enable "counterfeit people," with consequences for online trust, employment, genuine social engagement, and the perceived value of human interaction.

Today an AI has to dumb down to pass the Turing Test - Ray Kurzweil

Chinese Room Thought Experiment

The Chinese Room is a thought experiment, not a test anyone administers. It belongs here as context for what the Turing test does and does not settle; the broader questions it raises about machine minds are treated under Emergence and Artificial General Intelligence (AGI) to Singularity.

Searle proposed it in 1980 to argue against the claim that a computer can have a mind. He imagines himself locked in a room with a set of rules, written in English, for manipulating Chinese symbols that are meaningless to him. He receives Chinese characters, processes them according to the rules, and produces Chinese characters in response. To an outside observer he appears to understand Chinese. He does not; he is following rules blindly.

Key points of the argument:

  • Understanding a language is not just manipulating symbols according to rules. It also requires a grasp of the meaning of the symbols.
  • A computer can manipulate symbols, but does not have the same kind of understanding a human being has.
  • The experiment shows that a computer cannot be said to understand Chinese even if it produces intelligent-sounding output.

The argument has been widely disputed by both philosophers and computer scientists, and there is no settled answer as to whether it succeeds. It remains useful here for one reason: it separates behaving as if from being, which is precisely the gap the Turing test cannot close.

Large Language Model (LLM) Evaluation

YouTube ... Quora ...Google search ...Google News ...Bing News

Dimensions to Compare

These are the axes along which language models are usually compared. Specific model scores are not listed here — they go stale within months and belong on the individual model pages (Claude | Anthropic, ChatGPT | OpenAI, Gemini | Google, and so on).

  • Attachment support — whether prompts can include images or documents, allowing the model to incorporate information beyond the text prompt.
  • Input length limit — the maximum prompt length the model accepts, measured in tokens.
  • Context window — the maximum amount of context the model can consider when generating a response.
  • Professional examinations — bar exam multiple choice, GRE verbal and quantitative, medical licensing. Tests domain knowledge under a standardized rubric; heavily exposed to training-data contamination.
  • Code generation — HumanEval and successors, measuring whether generated code passes hidden tests.
  • Mathematical reasoning — GSM8K for grade-school word problems, harder sets for competition and research mathematics.
  • Tool use and long-horizon tasks — see Frontier Evaluations below.

Qualitative factors that resist single-number scoring: authenticity, speed, grammar, readability, unbiasedness, backtracking behavior, safety, responsibility, contextual understanding, and text operations.

A Survey on Evaluation of Large Language Models

The survey covers seven categories of LLM trustworthiness. Each maps to material elsewhere on this wiki:

Popular Benchmarks for Testing LLMs

Most of these are Datasets with an associated scoring protocol and leaderboard.

  • AI2 Reasoning Challenge (ARC) — advanced question answering requiring reasoning. 7,787 grade 3–9 science exam questions with a supporting knowledge base of 14.3M unstructured text passages. Multiple choice.
  • WinoGrande — commonsense reasoning. 44,000 examples requiring the model to resolve word meaning in context and reason about relationships between entities. A scaled-up adversarial successor to the Winograd Schema Challenge.
  • Holistic Evaluation of Language Models (HELM) — a living multi-metric framework rather than a single task set. Evaluates models across many scenarios on accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency simultaneously, with the explicit goal of transparency.
  • BIG-bench — a collaboratively built suite of roughly 200 diverse tasks contributed by researchers, deliberately chosen to be beyond the capability of models at the time of construction.
  • Massive Multitask Language Understanding (MMLU) — multiple-choice questions across 57 subjects spanning humanities, social sciences, STEM, and professional domains, from elementary to advanced professional level. Measures breadth of acquired knowledge. Now heavily saturated and contamination-prone.
  • SQuAD — reading comprehension. Questions and answers created by crowdworkers over Wikipedia articles; the model must locate the answer span in a passage.
  • GLUE — nine natural language understanding tasks including sentiment analysis, text classification, and inference.
  • SuperGLUE — an extension of GLUE with harder tasks, built after models saturated the original. Tasks include:
 ** CoLA: Corpus of Linguistic Acceptability
 ** SST-2: Stanford Sentiment Treebank
 ** MRPC: Microsoft Research Paraphrase Corpus
 ** STS-B: Semantic Textual Similarity Benchmark
 ** QQP: Quora Question Pairs
 ** MNLI: MultiNLI
 ** QNLI: Question Natural Language Inference
 ** RTE: Recognizing Textual Entailment
 ** WNLI: Winograd Schema Challenge
 ** AX: Adversarial Textual Entailment

The GLUE-to-SuperGLUE sequence is the pattern in miniature: a benchmark is built, models saturate it, a harder version replaces it, and the cycle repeats.

Frontier Evaluations

...Google search

As knowledge-recall benchmarks saturated, evaluation shifted toward tasks resistant to memorization and toward measuring what a system completes rather than what it answers.

  • Abstraction and reasoning challenges — novel visual or logical puzzles built so that pattern-matching on training data does not help, testing generalization to genuinely unseen problem structures.
  • Expert-authored examinations — very hard questions written by specialists across many fields, designed so that answers are not recoverable from web text.
  • Research-level mathematics — problems drawn from active research rather than competitions, where correct answers are hard to guess and hard to look up.
  • Software engineering tasks — resolving real issues in real repositories, scored by whether the existing test suite passes. Closer to deployed work than isolated function-writing.
  • Agentic and long-horizon evaluation — measuring how long a task a system completes without human correction, and how fast that duration is growing. This is the most direct available proxy for claims about autonomous research and is discussed further under AI Verification and Validation.
  • Human preference arenas — blind pairwise comparison by users at scale, aggregated into a ranking. Captures perceived quality but is subject to style effects and self-selection.
  • Dangerous capability evaluations — structured tests for uplift in areas such as offensive Cybersecurity or biological design, ideally with thresholds agreed before the measurement is taken. See Risk, Compliance and Regulation and AI Governance.

Persistent problems across all of these:

  • Contamination — once a benchmark is public, it leaks into training data.
  • Construct validity — whether the test measures the capability claimed or a correlate of it.
  • Self-grading — many headline results are produced by the organization being measured, against definitions that organization wrote.
  • The capability–reliability gap — a high score on curated problems does not predict performance on real work.

Evaluating Large Language Models on Clinical & Biomedical NLP Benchmarks

Evaluating Large Language Models on Legal Reasoning

LegalBench: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. It was built through an interdisciplinary process in which tasks were designed and hand-crafted by legal professionals. Because subject matter experts took the leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers find interesting. The paper maps popular legal frameworks for describing legal reasoning onto LegalBench tasks, giving lawyers and LLM developers a common vocabulary, and presents an empirical evaluation of 20 open-source and commercial models.

Natural Language Processing (NLP) Evaluation

General Language Understanding Evaluation (GLUE)

GLUE is a collection of resources for training, evaluating, and analyzing natural language understanding systems — tasks like picking out the names of people and organizations in a sentence, or figuring out what a pronoun such as "it" refers to when there are multiple potential antecedents. GLUE consists of:

  • A benchmark of nine sentence- or sentence-pair language understanding tasks built on established existing datasets, selected to cover a diverse range of dataset sizes, text genres, and degrees of difficulty.
  • A diagnostic dataset designed to evaluate model performance across a wide range of linguistic phenomena.
  • A public leaderboard for tracking performance and a dashboard for visualizing model performance on the diagnostic set.

The Stanford Question Answering Dataset (SQuAD)

Machine Learning Evaluation

Procgen

Procgen is a set of 16 procedurally generated environments that measure how quickly a model learns generalizable skills. Because levels are generated rather than fixed, a model cannot succeed by memorizing the training set — which makes Procgen a test of generalization rather than of task performance. It builds on the CoinRun toolset, which used procedural generation to construct separate sets of training and test levels.

OpenAI previously released Neural MMO, a massively multiagent virtual training ground, and Gym, a proving ground for reinforcement learning algorithms. It later released SafetyGym, a suite for developing AI that respects safety constraints while training and for comparing how well algorithms avoid mistakes during learning.

Human Evaluation

CAPTCHA stands for "Completely Automated Public Turing test to tell Computers and Humans Apart." It is a security measure that protects users from spam and password decryption by verifying that a user is human and not a computer. Note the inversion: this is a Turing test administered by a machine, with the machine as judge rather than subject.

I'm not a robot

No CAPTCHA reCAPTCHA, popularized by Google, uses a checkbox labeled "I am not a robot." It analyzes user behavior to determine whether the user is human, presenting a traditional image selection challenge only when the result is inconclusive. When you click the checkbox, reCAPTCHA monitors:

  • Mouse movements — human movement tends to be unpredictable, while bots often exhibit linear or mechanical paths.
  • Click timing — humans have natural delays; bots execute at near-instantaneous speeds.

Improving AI while trying to outsmart it

The bot test has a dual purpose. While meant to distinguish humans from bots, the data collected often trains AI systems to improve at image recognition, text understanding, and problem solving. The effectiveness of CAPTCHAs is constantly challenged by advances in machine learning: research has demonstrated that advanced systems can solve image-based CAPTCHAs, including reCAPTCHA v2, with a 100% success rate using object-detection models for segmentation and classification.

Humans taking these tests are teaching the bots to beat the tests. It is a cycle of humans improving AI while trying to outsmart it — irony at its finest, and a live example of the saturation pattern described at the top of this page.

Future Prospects and Innovations

  • Adaptive challenge generation — techniques that evolve in response to changing bot strategies.
  • Invisible CAPTCHA — reCAPTCHA v3 eliminates visible challenges, continuously scoring the likelihood of bot interaction between 0 and 1.
  • Cognitive deep-learning CAPTCHA — combining text, image, and cognitive characteristics, using adversarial examples and neural style transfer to resist automated attack.
  • Behavioral analysis and biometric verification — distinguishing human from bot action without explicit challenges, raising questions covered under Privacy.
  • AI-powered defenses — using models to design challenges that better separate bot activity from human input. See Cybersecurity.

Evaluating Machine Learning (ML) Hardware, Software, and Services

This is a different kind of benchmark from everything above. The tests in earlier sections ask what a model knows or can do; these ask how fast, how cheaply, and on what hardware. Throughput and cost per unit of capability increasingly determine what is deployable, which makes this category more consequential than its low profile suggests.

MLPerf

Related but Distinct

Two topics that have historically been filed here but are not AI benchmarks.

Backtracking

Two unrelated meanings share this word, which is why it appears on a benchmarks page.

  • The search technique — a general algorithmic method that considers every possible combination to solve a computational problem, incrementally building candidates and abandoning a candidate as soon as it cannot be completed to a valid solution. Used for constraint satisfaction problems such as crosswords, verbal arithmetic, and Sudoku. This belongs with Algorithms, AI Solver, and Optimization Methods, not with evaluation.
  • The model behavior — a language model recognizing that its own line of reasoning is wrong and revising it mid-response. This is a quality dimension in LLM evaluation above, and relates to Explainable / Interpretable AI where the question is whether a stated reasoning trace reflects the computation that actually occurred.

Also not to be confused with Backtesting, which evaluates a strategy against historical data.

American Productivity & Quality Center (APQC)

APQC benchmarks business processes, not AI systems — a separate discipline that happens to share the word. A non-profit, it provides independent research and data to more than 1,000 organizational members across 45 industries, with a benchmark data set of more than 4,000,000 data points. Relevant here mainly as a reminder that "benchmark" means something different to a process improvement team than to a machine learning researcher, and that AI project evaluation (Evaluation, Project Check-in) draws on both traditions.

Retrieved from "https://primo.ai/index.php?title=Benchmarks&oldid=43686"