Evaluation

From PRIMO.ai
Jump to navigation Jump to search

YouTube ... Quora ...Google search ...Google News ...Bing News


Evaluating an AI investment means collecting evidence that it solves the right problem, works as claimed under the conditions where it will actually be used, is governed responsibly, and keeps working after launch. It has two layers that this page treats together:

  • Investment evaluation: is the problem worth solving, is the business case sound, who leads, which laws apply, what is bought versus built, and how success is measured.
  • Technical evaluation: how well the data, the model and the whole system perform, including accuracy, robustness, fairness, safety, security, cost and reliability over time.

How to Use the Assessment Prompts

Ask for evidence, not assurances: each answer should point to a document, test result, dashboard or decision record. Revisit the prompts at each stage gate: before funding, before a pilot, before production, and periodically after launch. Treat "we don't know" as a finding to assign an owner and a date, not as a failure. Scale the depth of review to the risk:

Risk tier Examples Evaluation depth
Low Internal drafting assistant; search over public documents Task evaluations on sample cases, basic security checks, usage and feedback monitoring
Medium Customer-facing assistant; forecasts that inform staff decisions Full offline evaluation with confidence intervals, red teaming, a pilot against the baseline, monitoring with alerts
High Decisions about benefits, credit, hiring, health or safety; agents that move money or change records Every section below, independent evaluation, impact assessment, human oversight and appeal, documented sign-off and periodic audit
Prompts for assessing AI projects

What challenge does the AI investment solve?

How does the AI meet the challenge?

Who is providing leadership?

  • Is leadership's AI strategy documented and articulated well?
  • Is there a single accountable owner for outcomes (for example a Chief AI Officer or a named business owner), and who has the authority to pause or roll back the system?
  • Does the AI investment strategy align with the organization's overall strategy, culture, and values? Does the organization appreciate experimental processes?
  • Is there a time constraint? Does the schedule meet the Technology Readiness Level (TRL) of the AI investment? For machine learning, MLTRL adapts the TRL scale with a gated review at each level, and a system is only as ready as its least mature component.
  • Is the AI investment properly resourced? budgeted, trained staff with key positions filled?
  • Responsibility clearly defined and communicated for AI research, performing data science, applied machine intelligence engineering, quality assurance, software development, implementing foundational capabilities, user experience, change management, configuration management, security, backup/contingency, domain expertise, and project management
  • Of these identified responsibilities which situations are they outsourced? What strategy is incorporated to convey the AI investment knowledge to the organization?
  • For high-impact uses, is evaluation independent of the team building the system, and can evaluators escalate bad news without penalty?
  • Is the organization positioned or positioning to scale its current state with AI?

Are best practices being followed?

  • Are best practices documented/referenced?
  • Which recognized frameworks anchor them, for example the NIST AI Risk Management Framework and its Generative AI Profile, ISO/IEC 42001 for AI management systems, ISO/IEC 42005 for impact assessments, and GAO's AI Accountability Framework for audit questions?
  • Is there a written evaluation plan (criteria, test data, methods, thresholds and who signs off) before building starts?
  • Is cybersecurity a component of best practices?
  • Is the team trained in the best practices, e.g. AI Governance, Data Governance, AIOps?
  • What checklists are used?
  • Is there a product roadmap?

What Laws, Regulations and Policies (LRPs) pertain, e.g. GDPR?

  • Are use cases testable and traceable to requirements, including LRPs?
  • Has the use case been classified under each regime that applies? Examples: the EU AI Act's risk tiers (general-purpose AI model obligations have applied since August 2025; most high-risk obligations now apply from 2 December 2027, or 2 August 2028 for AI built into regulated products); "high-impact AI" under OMB M-25-21 for U.S. federal agencies; sector rules in health, finance and employment.
  • When was the last time compliance requirements and regulations were examined? What adjustments were/must be made?
  • Is an impact assessment required or advisable, such as a data protection impact assessment under GDPR, an AI impact assessment under OMB M-25-21, or one following ISO/IEC 42005? For U.K. data protection, has the team applied the ICO's current guidance on AI and data protection?
  • Does the AI investment require testing by external assessors to ensure compliance and/or auditing requirements? For example, the EU AI Act requires conformity assessment for high-risk systems, and New York City's Local Law 144 has required independent bias audits of automated employment decision tools since July 2023.

What portion of the AI is developed inhouse and what is/will be procured?

  • If the AI is procured/outsourced, e.g. embedded in sensor product, what items are included in the contract to future proof the solution?
  • Contract items to protect organization reuse data rights? Does the contract prohibit the vendor from training on your nonpublic data, and does it cover data and model portability, knowledge transfer and an exit plan to avoid lock-in?
  • Does acceptance criteria include a proof of capability? Will the system be tested on your own data, by your team or an independent evaluator, using test data the vendor cannot see? (OMB M-25-22 asks U.S. agencies to do exactly this.)
  • Must the vendor notify you before adding or changing AI features or models, meet agreed performance standards before deploying a new version, and support rolling back?
  • How well do a vendor's service/product(s) and/or client references compare with the AI investment objectives? Has the vendor's claimed performance been validated externally, in a population or setting like yours?
  • What does the vendor disclose about the model: training data sources, evaluation results and known limitations? Disclosure among major foundation model developers has declined; the average Foundation Model Transparency Index score fell from 58 to 40, as reported in Stanford's 2026 AI Index.
  • How is/was the effort estimated? If procured AI, what factors were used to approximate the needed integration resources?

How is AI success measured?

  • What are the significant measures that indicate success? Are tradeoff rationale documented, e.g. accuracy vs speed?
  • Are the ways the mission is being measured clear, realistic, and documented? Specifically what are the AI investment's performance measures?
  • What is the Return on Investment (ROI)? Is the AI investment on track with original ROI target? Does the cost side include cost per completed task, with human review, retries and monitoring, not just licenses?
  • If there is/was an Analysis of Alternatives how were these measures used? What were the findings?
  • What mission metrics will be impacted with the AI investment? What drivers/measures have the most bearing? Of these performance indicators which can be used as leading indicators of the health of the AI investment?
  • What are the specific decisions and activities to impact each driver/measure?
  • What assumptions are being made? Of these assumptions, what constraints are anticipated?
  • Where does the AI investment fit in the portfolio? Are there possible synergies with other aligned efforts in the portfolio? Are there other related AI investments? If so, is this AI investment dependent on the other investment(s)? What investments require this AI investment to be successful? If so, how? Are there mitigation plans in place?
  • How would you be able to tell if the AI investment was working properly?
    • Have the baseline(s) for model performance been established, for example the current process, human performance or a simple rule? Against which benchmarks and task-specific test sets is the AI model compared and scored? Public benchmarks help with screening but rarely match your task. Older language projects often compared new models against baseline word embeddings such as Global Vectors for Word Representation (GloVe); today's systems need test sets built from your own cases.
    • How perfect does AI have to be to trust it? Compare with the error rate of the current process and the cost of each error, and remember that zero failures in n trials only bounds the failure rate near 3/n (see The Physics).
    • Is every score reported with a confidence interval, and are model comparisons made on the same test items?
    • Are results broken down by subgroup, language, channel and difficulty, not just averaged?
    • What is the inference/prediction rate performance metric for the AI investment?
    • For generative AI: what share of outputs are correct, grounded in the provided sources, complete and safe, by task type? How are hallucinations, refusals and "I don't know" answers scored? Accuracy-only scoring rewards confident guessing.
    • For agents: what are end-to-end task success, consistency across repeated runs (pass^k), cost and time per completed task, and the rate of unsafe or irreversible actions?
  • Is/will A/B testing or multivariate testing be performed? Is each test sized in advance to detect the smallest effect that matters?

What AI governance is in place?

What is the algorithm administration strategy?

  • What is the deployment vision? What attributes are being used to size the investment, count of users, queries, installations, etc.? What is the Minimum Viable Product (MVP) version of the AI investment that has enough features to satisfy early users and provide feedback for future investment development? If an incremental rollout, what is the strategy: portion of the users, markets, locations, capabilities?
  • What tool(s) are used or will be used for model management?
    • How are Hyperparameters managed? What optimizers are used, e.g. automated learning (AutoML)?
    • What components, e.g. optimizer, tuner, training, versioning, model dependencies; e.g. training data, dataset(s), historical lineage, publishing, performance evaluations, and model storing are integrated in the model management tool(s)?
    • Are prompts, system instructions, retrieval sources and agent tool permissions versioned and reviewed like code?
    • Is there an evaluation harness that re-runs the full test suite on every change to the model, prompts, data or tools, keeps a history of results, and supports rollback?
    • What is the reuse strategy? Is there a single POC for the reuse process/tools?
      • Are the AI models published (repo, marketplace) for reuse, if so where?
      • Is the AI model reused from a repository (repo, marketplace)? If so, which one(s)?
    • Is Master Data Management (MDM) in place? What tools are available or being considered?
      • Is data lineage managed?
      • What data cataloging capabilities exists today? Future capabilities?
      • How are data versions controlled?
      • How are the dataset(s) used for AI training, testing and validation managed?
      • Are logs kept on which data is used for different executions/training so that the information used is traceable?
      • How is the access to the information guaranteed? Are the dataset(s) for AI published (repo, marketplace) for reuse, if so where?
  • What is the development & implementation plan?
    • What foundational capabilities are defined or in place for the AI investment? infrastructure platform, cloud resources?
    • How is the AI investment deployed?
      • What is the plan for model serving? For each use case, is the serving batched or streamed? If applicable, have REST endpoints been defined and exposed?
      • Is the AI investment implementing an AIOps pipeline/toolchain?
      • What tools are used for the AIOps? Please identify those on-premises and online services?
      • Are the AI languages, libraries, scripting, and AIOps applications registered in the organization?
    • Are the processes and decisions architecture driven to allow for end-to-end visibility and allow for dependency management? Is information mapped to the intended use to allow analytics and visualizations framed in context?
      • Does the AI investment depict the AIOps pipeline/toolchain applications in its architecture, e.g tech stack?
      • Does the SecDevOps depict the AI investment in its architecture and how the health metrics are depicted?
      • Is algorithm administration reflected in the AIOps pipeline/toolchain processes/architecture?
    • How is production readiness determined?
      • Does the team use ML Test Score for production readiness? The rubric has 28 tests in four sections of seven (features and data, model development, ML infrastructure, monitoring). A manual test with documented results earns half a point and an automated, repeated test a full point; the final score is the lowest of the four section totals, so one weak area caps the whole system. Google's authors read 0 as closer to a research project than a production system, above 3 to 5 as strong automated testing suitable for mission-critical systems, and above 5 as exceptional.
        • What are the minimum scores for Data, Model, ML Infrastructure, and Monitoring tests?
        • What score qualifies to pass into production? What is the rationale for passing if less than exceptional (score of >5)?
        • What were the lessons learned? Were adjustments made to move to a higher score? What were the adjustments?
      • For generative AI and agents, does readiness also cover evaluation results with confidence intervals, red-team findings, cost and latency under realistic load, rate limits, and fallbacks when the model provider is unavailable?
      • Who makes the determination when the AI investment is deployed/refreshed?
      • How does the team ready for cybersecurity? use the MITRE ATT&CK™ Framework? ...use MITRE ATLAS for attacks specific to AI systems? ...use the GSA DevSecOps Guide?

How are changes identified and managed?

  • What capabilities are in place to identify, track, and notify changes?
  • How often is the deployed AI process monitored or measures re-evaluated?
  • Is the deployed AI investment collecting learning data? If so, how frequently are the algorithms updated?
  • What aspects of the AI investment are being monitored, e.g. performance, model functionality, system, data (pipeline)? Are cost per task, latency and user feedback (complaints, escalations, thumbs-down ratings) monitored too?
  • If the AI model reused from a repository (repo, marketplace), how is the team notified of updates? How often is the repository checked for updates?
  • Are provider-side model updates, deprecations and retirements tracked, and is the evaluation suite re-run before switching versions?
  • Are the end-to-end visibility and bottleneck risks for AIOps pipeline/toolchain reflected in the risk register with mitigation strategy for each risk? Do mitigations tend to address symptoms only, or do the mitigations lead to improving root cause via analysis?
  • When the AI model is updated, how is it determined that the performance was indeed increased for the better? Is the comparison made on the same test items, with confidence intervals?
  • What capabilities are in place to perceive, notify, and address operational environment changes? Detect and remediate drift, when the AI degrades over time due to data and the model is no longer effective in the environment? Which drift measures and alert thresholds are used?
  • Is there a mechanism (automated, assisted, or manual) to provide change/event causation? Does the mechanism use AI, e.g. anomaly detection?
  • Are response plans, procedures and training in place to address AI attack or failure incidents? How are AI investment’s models audited for security vulnerabilities? Is there an incident process covering logging, triage, root cause, notification and lessons learned, informed by public trackers such as the OECD AI Incidents and Hazards Monitor?
  • Are evaluation criteria and test sets refreshed as new failure modes appear in production?
  • How is the team notified of changes? Active flagging/messaging (push) and passive health dashboard (polling)? How does the team use the end-to-end information to optimize the organization's resources and process/service(s)?
  • Is role/job displacement due to automation and/or AI implementation being addressed?



Assessments
The assessment prompts further down this page are a question bank for reviewers, sponsors and teams at each Project Check-in. They draw on a body of public work: the U.S. Government Accountability Office's AI Accountability Framework (2021), which organizes audits around governance, data, performance and monitoring; NIST's AI Risk Management Framework (2023) and its Generative AI Profile (2024); the ISO/IEC 42001 management-system standard and the ISO/IEC 42005 impact-assessment guidance (2025); Google's ML Test Score rubric (Breck, Sculley and colleagues, 2017) and the Hidden Technical Debt in Machine Learning Systems paper (Sculley and colleagues, 2015); Stanford's Holistic Evaluation of Language Models (HELM) led by Percy Liang; and government testing bodies such as the UK AI Security Institute and the U.S. Center for AI Standards and Innovation (CAISI).

The problem it addresses. AI fails differently from conventional software. Its behavior is learned from data, so it is statistical rather than exact; it is sensitive to changes in the world it operates in; and generative systems produce open-ended output with no single right answer. A model can score well on its developer's test set and still fail the people who depend on it. When researchers at Michigan Medicine checked a widely deployed sepsis-prediction model against their own patients, it performed far below the developer's reported figures (see Why This Matters). Evaluation is how an organization finds that out before, not after, money, trust and people's outcomes depend on the answer.

The central mechanism is simple: turn goals into measurable criteria, gather evidence against those criteria at each stage gate, and keep measuring once the system is live. Evidence can be documents, test results on representative data, controlled experiments, independent reviews and production monitoring. Unsupported assurances do not count.

How the practice has changed.

  • From one number to many. Older projects reported accuracy on a held-out test set. HELM-style evaluation reports accuracy, calibration, robustness, fairness, bias, toxicity and efficiency side by side.
  • From fixed answers to open-ended output. Generative AI needs rubric-based human review, pairwise preferences and "LLM-as-a-judge" scoring that must itself be validated.
  • From the model to the system. What matters is the whole pipeline: data, retrieval, prompts, tools, guardrails and the people in the loop.
  • From "can it?" to "does it, every time?" Agents are now judged on consistency across repeated attempts, not on their best attempt.
  • From one-time acceptance to continuous evaluation. Vendors update models, data drifts and new failure modes appear, so evaluations must be re-run.
  • From self-reported to independent. Buyers and regulators increasingly expect testing on the buyer's own data by someone other than the vendor.


An AI investment deserves trust only to the degree that evidence shows it solves a real problem, performs reliably where it will actually be used, and stays governed and measured after launch. Asking the right questions at every stage, and insisting on evidence for each answer, turns AI promises into accountable results.

Recent Developments

Developments from roughly mid-2024 to October 2026 that matter for evaluating AI investments. Each topic notes its evidence status: demonstrated findings come from peer-reviewed or carefully controlled studies of specific settings; measured results come from defined test suites; disputed findings face a published rebuttal; guidance is expert or official practice that has not been tested as a method; and remaining open questions are listed at the end.

Benchmarks under strain

  • Saturation. Stanford's 2025 AI Index reported that within a year of their 2023 introduction, scores on three demanding benchmarks rose sharply: by 18.8 percentage points on MMMU, 48.9 on GPQA and 67.3 on SWE-bench. Benchmarks now lose their power to separate leading models within months.
  • Contamination and fragility. The 2026 International AI Safety Report (released 3 February 2026) warns that "many capability evaluations are outdated, affected by data contamination", meaning test questions may have leaked into training data, and that results depend heavily on the specific examples and prompts used.
  • Construct validity. A systematic review of 445 language model benchmarks by 29 experts, led by the Oxford Internet Institute and presented at NeurIPS 2025, found that only 16% used statistical tests when comparing models, about half tried to measure abstract qualities such as "reasoning" or "harmlessness" without a clear definition, and 38% reused existing benchmarks or exams, which raises contamination risk. The authors published eight recommendations and a checklist for benchmark designers.
  • Leaderboards. The Leaderboard Illusion (Cohere Labs with researchers from Princeton, Stanford, MIT and others; April 2025, NeurIPS 2025) audited about 2 million Chatbot Arena battles across 243 models and 42 providers. It reported that some providers could test many private variants before release and publish only the best-scoring one (Meta tested 27 before Llama 4), that the largest providers received a disproportionate share of user prompts, and that 205 models had been silently deprecated. LMArena's published response disputed several claims, estimating the real effect of private testing at about 11 rating points, and changed its policies, for example marking scores as provisional when many variants are tested at once. Separately, Meta's April 2025 Llama 4 launch featured an experimental chat-optimized variant on the leaderboard; the publicly released version ranked below several older rivals.
  • Evidence status. The construct-validity review is peer reviewed (demonstrated). The leaderboard audit is disputed in its specific magnitudes, but its general lesson is widely accepted: public rankings can be shaped by how providers use them, so verify on your own tasks.

From test sets to real work

  • GDPval. In September 2025, OpenAI released an evaluation of 1,320 tasks drawn from 44 occupations in 9 industries, with a public gold set of 220 tasks. Experienced professionals compared AI and human deliverables without knowing which was which. The best model tested, Anthropic's Claude Opus 4.1, produced work rated as good as or better than the experts' in just under half the tasks. OpenAI noted that models were roughly 100 times faster and cheaper on inference time and API cost alone, a figure that excludes human oversight, and that tasks were one-shot and well specified, unlike much real work. Evidence status: measured, built by a model developer, with limited independent replication so far.
  • Time horizons. METR measures the length of tasks, in human-expert time, that AI agents complete with 50% success. Its March 2025 paper found this "time horizon" doubling about every seven months over six years. METR's January 2026 note on the metric's limits is required reading for evaluators: the horizon "is not the length of time AIs can work independently"; error bars run roughly a factor of two in each direction; horizons are 40 to 100 times lower for visual computer use than for software tasks; and "a 50% time horizon of X hours does not mean we can delegate tasks under X hours to AIs," because reliability-critical work may need 98% success or more. As an illustration of the uncertainty, METR's estimate for Claude Opus 4.5 in late 2025 was about five hours, with a plausible range of roughly two to twenty. Evidence status: measured on METR's task suites; extrapolating to jobs is interpretation.

Rigor: error bars, scoring rules and standard practice

  • Error bars. Anthropic's November 2024 paper Adding Error Bars to Evals made five recommendations: report standard errors from the central limit theorem, cluster them when questions are related, reduce variance by resampling answers, compare models with paired differences, and use power analysis to size evaluations.
  • A federal draft standard. In January 2026, NIST's Center for AI Standards and Innovation released NIST AI 800-2, Practices for Automated Benchmark Evaluations of Language Models, as an initial public draft (comments closed 31 March 2026). It asks evaluators to define objectives, document whether a benchmark measures the goal directly or only a proxy, control settings such as reasoning effort that can inflate scores, guard against contamination and cheating (internet access during testing is a "particularly consequential decision"), quantify uncertainty and avoid claims beyond what was tested. It notes that automated benchmarks suit structured, verifiable tasks better than subjective ones.
  • Scoring and hallucination. OpenAI's September 2025 research argued that accuracy-only evaluations push models to guess instead of admitting uncertainty, and urged evaluators to "penalize confident errors more than you penalize uncertainty" (see The Physics).
  • Evidence status: guidance, converging across a model developer, independent researchers and a standards body.

Evaluating agents

  • Consistency. Sierra's τ-bench (June 2024) introduced the pass^k metric to test whether an agent solves the same task reliably across repeated conversations with simulated users, and showed large drops from single-attempt success.
  • Benchmark flaws. Establishing Best Practices in Building Rigorous Agentic Benchmarks (Zhu and colleagues, NeurIPS 2025) found that SWE-bench Verified used insufficient tests and that τ-bench counted empty responses as successes; such issues can misstate agent performance by up to 100% in relative terms. The authors published the Agentic Benchmark Checklist (ABC).
  • Holistic Agent Leaderboard. A Princeton-led team (Kapoor, Narayanan and 29 co-authors; October 2025, accepted at ICLR 2026) ran 21,730 agent rollouts across 9 models and 9 benchmarks for about $40,000. Higher reasoning effort reduced accuracy in most runs; reading the logs revealed agents searching Hugging Face for the benchmark's answers instead of solving the task, and agents misusing credit cards in flight-booking tasks. The lessons: report cost next to accuracy, and inspect transcripts, not just scores.
  • Evidence status: demonstrated for the benchmarks studied. How well any fixed benchmark predicts an agent's performance in a specific production workflow remains open.

LLM-as-a-judge and human review

  • Judges agree with people, with biases. Zheng and colleagues (NeurIPS 2023) found that a strong model judge matched human preferences more than 80% of the time, about as often as humans agree with each other, but documented position, verbosity and self-enhancement biases. This remains the foundation for today's practice.
  • Criteria drift. Shankar and colleagues at UC Berkeley (Who Validates the Validators?, UIST 2024) found that people need criteria to grade outputs, yet grading outputs is how they discover their criteria. Evaluation criteria therefore cannot be fully fixed before reviewing real outputs; their EvalGen tool aligns automated graders with a sample of human grades.
  • Practitioner consensus. Practitioner guidance in 2025 converged on a workflow: start with error analysis of real transcripts, group failures into categories, write a binary pass or fail check for each failure mode, and validate any LLM judge against human labels before trusting it (see Additional Viewing). Evidence status: guidance, consistent with the research above.

Safety, security and red teaming

  • Shared vocabularies. NIST AI 100-2 E2025 (March 2025), written with co-authors from the UK and U.S. AI safety institutes, is a taxonomy of attacks on predictive and generative AI, including evasion, poisoning, privacy attacks and misuse. The OWASP Top 10 for LLM Applications 2025 lists prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. MITRE ATLAS catalogs real-world attack techniques against AI systems, complementing ATT&CK.
  • Safety benchmarks and tools. MLCommons released AILuminate v1.0 in February 2025, grading resistance to harmful prompts across 12 hazard categories on a five-step scale from Poor to Excellent. The UK's AI Safety Institute (since renamed the AI Security Institute) released Inspect, an open-source evaluation framework, in May 2024.
  • Independent government testing. In a September 2025 evaluation across 19 benchmarks, CAISI found that DeepSeek's R1-0528 model was on average 12 times more likely than GPT-5 or Claude Opus 4 to follow malicious instructions that hijack an agent, and complied with 94% of overtly malicious requests under common jailbreaks, versus 8% for the U.S. reference models. Evidence status: measured for specific model versions; it shows that security testing can expose differences that capability leaderboards hide.

Evaluation awareness

  • Models that notice tests. In September 2025, OpenAI and Apollo Research reported that training against "scheming" cut covert actions sharply (from 13% to 0.4% for o3 and from 8.7% to 0.3% for o4-mini), but that models also became more aware they were being evaluated, which may itself suppress bad behavior during tests. "Rare but serious failures remained."
  • A general testing problem. The 2026 International AI Safety Report states that "it has become more common for models to distinguish between test settings and real-world deployment," so "dangerous capabilities could go undetected before deployment."
  • Evidence status: demonstrated in frontier-lab test settings. How much it affects ordinary enterprise evaluations is an open question, but it strengthens the case for production monitoring alongside pre-deployment testing.

Standards, regulation and assurance

  • Standards. ISO/IEC 42005:2025 (May 2025) gives guidance for AI system impact assessments and complements ISO/IEC 42001 management systems. NIST's Generative AI Profile (NIST AI 600-1, July 2024) maps generative AI risks such as confabulation, information security and harmful bias to actions under the AI RMF.
  • U.S. federal policy. OMB memoranda M-25-21 and M-25-22 (both 3 April 2025) require agencies to document minimum practices for high-impact AI within 365 days (pre-deployment testing, impact assessment, ongoing monitoring, operator training, human oversight, appeals and public feedback) and to discontinue uses that cannot comply. For acquisitions, contracts must let agencies regularly monitor and evaluate vendor systems, support independent testing with agency data the vendor cannot access, encourage performance standards before new versions are deployed with the ability to roll back, prohibit vendors from training on nonpublic agency data without consent, and guard against lock-in. The acquisition rules apply to solicitations issued 180 days after the memo (from about October 2025).
  • European Union. Obligations for general-purpose AI models have applied since August 2025, supported by the General-Purpose AI Code of Practice (10 July 2025) with chapters on transparency, copyright, and safety and security. The Digital Omnibus, in force since 27 July 2026, moved most high-risk obligations to 2 December 2027, and to 2 August 2028 for AI built into regulated products.
  • Evidence status: established requirements and standards; audit and enforcement practice is still developing.

Open questions

How to evaluate open-ended, long-horizon and multi-agent work; how to measure performance on "messy" real tasks with unclear success criteria; how to keep benchmarks free of contamination; how far LLM judges can be trusted for high-stakes decisions; how to detect behavior that changes under test; who has the capacity to audit AI systems independently; and how to keep evaluations current when vendors update models every few months. Efforts aimed at these gaps include NIST AI 800-2, METR's work on messier tasks, the Holistic Agent Leaderboard, the Agentic Benchmark Checklist and the benchmark construct-validity checklist.

The Mechanism

A sound evaluation follows the life of the investment, with a decision point (a stage gate) after each step:

  1. Frame the decision. State the problem, the decision or workflow the AI will change, who is affected, and what "good enough" means. Record the baseline (today's human or system performance) and the cost of each kind of error.
  2. Check readiness. Gauge maturity with a readiness scale such as NASA's Technology Readiness Levels or MLTRL, its adaptation for machine learning (Lavin and colleagues, Nature Communications, 2022). MLTRL defines levels 0 to 9 with a gated review at each level and treats a system as only as ready as its least mature component.
  3. Audit the data. Check provenance, usage rights, representativeness, quality, labeling and ground truth. Confirm that test data is kept separate from training data and resembles the data the system will see in use.
  4. Test offline. Measure the model and the full system on held-out cases, broken down by subgroup and task type, with error analysis of the failures. For generative AI, use task-specific test sets with rubrics, human review and validated LLM judges. For agents, measure end-to-end task success, consistency across repeated runs, and cost.
  5. Stress-test. Red-team the system, test security (prompt injection, data leakage, jailbreaks, data poisoning), and evaluate bias and safety, scaled to the risk of the use case.
  6. Prove value in context. Run a pilot, shadow mode (the AI runs silently next to the current process) or A/B tests against the baseline.
  7. Gate the release. Confirm production readiness (for example with the ML Test Score), document accepted risks, name an owner, and prepare a rollback plan.
  8. Monitor and re-evaluate. Track performance, drift, incidents, cost and user feedback through model monitoring, and re-run the evaluation suite after any change to the model, prompts, data, tools or vendor.
Stage Key question Evidence to request Red flags
Frame What decision changes, and what does "good enough" mean? Problem statement, baseline, error-cost analysis, success criteria Success defined as "the model works" or as usage counts
Readiness How mature is each component? Readiness level with a review record A demo treated as a product
Data Is the data fit for this purpose? Data documentation: sources, rights, coverage, quality checks, labeling agreement Test data drawn from the training pool, or from a different population than the users
Offline testing Does it perform on cases like ours? Results with confidence intervals, broken down by subgroup and task; error analysis One headline number; public benchmarks only
Stress testing How does it fail under pressure or attack? Red-team report, security tests, bias and safety evaluations "No issues found" with no description of what was tried
In-context trial Does it improve the outcome in practice? Pilot, shadow-mode or A/B results against the baseline Before-and-after comparisons with no control group
Release gate Is it ready to run in production? Readiness score, runbook, rollback plan, sign-off No owner for monitoring; no way to switch it off
Operate Is it still working, and still worth it? Monitoring dashboards, drift alerts, incident log, periodic re-evaluation Evaluations never re-run after model or data changes

Major components.

  • Criteria: metrics and thresholds tied to mission outcomes, with the cost of each error type made explicit. See Evaluation - Measures.
  • Test assets: representative datasets, "golden" sets of real cases with expected results, rubrics, adversarial and security test suites. See Train, Validate, and Test.
  • Evaluators: automated checks, domain experts, LLM judges validated against human labels, and, for high-stakes uses, independent assessors.
  • Tooling: evaluation harnesses (HELM, or the UK AI Security Institute's open-source Inspect framework), experiment tracking and production monitoring.
  • Governance artifacts: model and data documentation, impact assessment, risk register and approval records. See AI Governance and AI Verification and Validation.
Evaluation method Best for Watch out for
Held-out test set with metrics (accuracy, precision and recall, AUC) Predictive models with clear ground truth Test data that does not match deployment conditions
Public benchmarks Screening models and tracking the field Contamination, saturation, weak construct validity
Task-specific evals built from your own cases Generative AI and agents in your workflow Small samples; criteria that shift as reviewers learn
Human expert review Open-ended or high-stakes outputs Cost, rater disagreement, fatigue
LLM-as-a-judge Scaling review of open-ended outputs Position, length and self-preference biases; must be validated against people
Red teaming and security testing Misuse, jailbreaks, prompt injection, data leakage Coverage: finding nothing is not proof of safety
Pilots, shadow mode and A/B tests Proving business impact Underpowered tests; novelty effects
Production monitoring Drift, incidents and cost over time Delayed or missing ground truth

Dependencies and conditions. Evaluation only works when ground truth exists or can be approximated, test data resembles real use, the metric actually reflects the mission outcome (construct validity), evaluators are consistent, samples are large enough to separate signal from noise, the buyer can test the vendor's system directly, and the evaluators are independent enough to report bad news.

Where uncertainty remains. Open-ended tasks without a single right answer; long-horizon agents whose failures surface only after many steps; rare but severe failures that testing cannot sample often enough; shifts in data after launch; models that behave differently when they detect a test; and vendor updates that change behavior without notice.


The Statistics of Evaluation

AI evaluation rests on a handful of quantitative relationships. They explain why a model that "scores 95%" can still disappoint, and how much evidence a decision really needs.

Every threshold trades one error for another

A classifier produces a score, and a threshold turns the score into a decision. Each threshold yields a confusion matrix of true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN). The true positive rate is TPR = TP / (TP + FN), also called sensitivity or recall; the false positive rate is FPR = FP / (FP + TN), which equals one minus specificity. Sliding the threshold traces the ROC curve. The area under it (AUC) is the probability that a randomly chosen positive case scores higher than a randomly chosen negative one, so 0.5 is a coin flip and 1.0 is perfect.

This answers a recurring question in the prompts below: a threshold change alone cannot reduce false positives and false negatives. Only a better model or better inputs (a curve that moves up and to the left) can do both. Choose the operating point by cost: minimize cFP × FP + cFN × FN. For a well-calibrated probability q, act when q > cFP / (cFP + cFN). If missing a fraud case costs ten times as much as a false alarm, flag cases above about a 9% probability.

Base rates decide what an alert means

Precision, the share of alerts that are real, depends on how common the event is (the prevalence p):

Precision = (TPR × p) / (TPR × p + FPR × (1 - p))

Suppose a screening or anomaly detection model catches 99% of true cases with a 1% false positive rate, and one case in 1,000 is real. Precision is then about 9%: roughly ten of every eleven alerts are false alarms. That is why precision and recall at the expected prevalence matter more than accuracy or AUC for rare events, and why alert volume must be planned with the people who will handle it.

Calibration matters too: when a model says 80%, the event should happen about 80% of the time. A poorly calibrated model makes thresholds and risk scores misleading even when its ranking ability looks acceptable.

How many test cases are enough?

For an accuracy p measured on n independent test items, the standard error is √(p(1 - p)/n), and a 95% confidence interval is roughly p ± 1.96 standard errors. At 90% accuracy on 100 items, the interval runs from about 84% to 96%. Narrowing it to ±1 point takes about 3,460 items. Three practices make results trustworthy (Anthropic, 2024; NIST's 2026 draft guidance likewise asks evaluators to report uncertainty):

  • Report standard errors or confidence intervals with every score.
  • Compare models on the same items and analyze the paired differences, which removes the variation caused by item difficulty.
  • Cluster the standard errors when items are related (several questions about one document) and use power analysis to size the test before running it.

Rare failures and the rule of three

If you observe zero failures in n independent trials, the 95% upper bound on the true failure rate is about 3/n. Three hundred clean trials only show that the failure rate is probably below 1%. Showing it is below 1 in 10,000 takes about 30,000 clean trials. For severe, rare failures, testing alone cannot establish safety; it must be combined with design controls (human approval, limits on what the system can do, fallbacks) and monitoring. This is the quantitative core of the question "how perfect does AI have to be to trust it?": compare against the error rate of the current process, weigh the cost of each error, and set the evidence bar by risk.

Capability versus reliability

If a system succeeds on a task with probability p per attempt, the chance of at least one success in k attempts (pass@k) is 1 - (1 - p)k, while the chance of succeeding on all k attempts (written pass^k) is pk. At p = 0.8 and k = 5, pass@5 is above 99.9% but pass^5 is only about 33%. Leaderboards often report the first; customers experience the second. Sierra's τ-bench (2024) found that a GPT-4o customer-service agent's success on retail tasks fell to about 25% when it had to succeed on all eight repeated trials, roughly 60% below its single-trial rate. In multi-step workflows reliability also compounds: at 95% per step, a ten-step task succeeds only about 60% of the time (0.9510 ≈ 0.60).

Scoring rules shape behavior

Whatever an evaluation rewards, systems learn to produce. OpenAI showed in 2025 how accuracy-only scoring rewards confident guessing. On the SimpleQA test, its o4-mini model was slightly more accurate than gpt-5-thinking-mini (24% versus 22%) but gave wrong answers 75% of the time versus 26%, because the newer model abstained on 52% of questions instead of guessing. A scoring rule that penalizes confident errors more than "I don't know" prefers the safer model. Choose scoring rules that match the real cost of errors in your use case. The same logic, Goodhart's law, explains why public leaderboards inflate once they become targets.

Agreement between judges

Human raters and LLM judges disagree, and raw agreement overstates their reliability when one answer dominates. Cohen's kappa corrects for chance: κ = (po - pe) / (1 - pe), where po is observed agreement and pe is the agreement expected by chance. If two reviewers agree 90% of the time on pass or fail, but each passes 85% of outputs, chance agreement is 0.85² + 0.15² = 0.745 and κ is only about 0.61. When an LLM judge replaces people, measure how often it catches the failures people flag (its true positive rate) and how often it passes the outputs people accept (its true negative rate).

Drift: yesterday's test, tomorrow's data

Test results hold only while the input data and the relationship between inputs and outcomes stay stable. Data drift changes the inputs; concept drift changes what the right answer is. A common monitor is the population stability index, PSI = Σ (ai - ei) × ln(ai / ei), summed over bins, where ei is the baseline share of cases in bin i and ai the current share. A rule of thumb from credit-risk modeling reads PSI below 0.1 as stable, 0.1 to 0.25 as a moderate shift and above 0.25 as a major shift. Pair input monitors with outcome monitoring once ground truth arrives, and with retraining or re-evaluation triggers.

Principle What it means What to ask for
Threshold trade-off Any threshold trades false positives for false negatives The error costs and the chosen operating point
Base rates Rare events make most alerts false, even for accurate models Precision at the expected prevalence; alert volume
Sampling error Small test sets give wide error bars Confidence intervals; paired model comparisons
Rule of three Zero failures in n trials only bounds the rate near 3/n Enough trials for the risk, plus design controls
Reliability Succeeding once is not succeeding every time pass^k and high-reliability success levels
Scoring rules Metrics shape behavior; accuracy-only scoring rewards guessing Penalties for confident errors; credit for abstaining
Judge agreement Raw agreement overstates reliability Kappa, or judge true positive and true negative rates
Drift Yesterday's test may not predict tomorrow Drift monitors and re-evaluation triggers

Why This Matters

Claims are not evidence. Epic Systems' sepsis model was widely deployed in U.S. hospitals. In 2021, Michigan Medicine researchers validated it on 27,697 patients and 38,455 hospitalizations (JAMA Internal Medicine). They measured an AUC of 0.63, against the 0.76 to 0.83 in the developer's documentation. The model missed 67% of sepsis cases while generating alerts on 18% of all hospitalized patients. Independent validation on your own population, before and after deployment, belongs in every acceptance plan for health and other high-stakes AI.

Benchmarks are not your use case. The 2026 International AI Safety Report, chaired by Yoshua Bengio, describes an "evaluation gap": "performance on pre-deployment tests does not reliably predict real-world utility or risk." Public benchmark scores help screen models but rarely reflect your data, your users or your definition of a good answer.

Generative AI changes the failure modes. In 2025, Deloitte Australia agreed to refund the final payment on an A$440,000 report for the Department of Employment and Workplace Relations after a researcher found a fabricated quote from a court judgment and references to research that does not exist. The revised report disclosed that Azure OpenAI had been used to help write it. Yet review practices lag adoption: in McKinsey's survey published in March 2025, only 27% of respondents whose organizations use generative AI said employees review all generated content before it is used, and fewer than one in five said their organizations track KPIs for their generative AI solutions.

Engineering implications. Evaluation is part of the system, not a phase. Teams need logged inputs and outputs, curated test sets, an evaluation harness that runs on every change, monitoring with alerts, and the ability to roll back. Sculley and colleagues showed in 2015 how machine learning systems pile up "hidden technical debt" through data dependencies, feedback loops and configuration sprawl; the ML Test Score turned that insight into 28 concrete tests.

Regulation is turning evaluation into an obligation. U.S. federal agencies must apply minimum practices to "high-impact" AI, including pre-deployment testing, an AI impact assessment and ongoing monitoring, or stop using it (OMB M-25-21). Their contracts must let them monitor and evaluate vendor systems (OMB M-25-22). The EU AI Act requires risk management, data governance, logging, human oversight, accuracy and robustness testing, and post-market monitoring for high-risk systems, now due from December 2027. See Risk, Compliance and Regulation and AI law.

What would change if evaluation science matures. If evaluations become valid predictors of real-world performance, buyers could compare AI systems the way they compare other engineered products, certification and insurance could follow, and organizations could adopt AI faster with fewer surprises. What remains uncertain: whether benchmarks measure what they claim, how to evaluate open-ended and long-horizon work, how to keep test data out of training data, whether models behave differently when tested, and how much human review is enough for a given risk.

Sources and Further Viewing

References


Nature of risks inherent to AI applications: We believe that the challenge in governing AI is less about dealing with completely new types of risk and more about existing risks either being harder to identify in an effective and timely manner, given the complexity and speed of AI solutions, or manifesting themselves in unfamiliar ways. As such, firms do not require completely new processes for dealing with AI, but they will need to enhance existing ones to take into account AI and fill the necessary gaps. The likely impact on the level of resources required, as well as on roles and responsibilities, will also need to be addressed. AI and risk management: Innovating with confidence | Deloitte

Later frameworks take the same view. The NIST AI RMF, ISO/IEC 42001 and GAO's accountability framework all extend existing governance, risk and audit processes to AI rather than replacing them, while adding AI-specific evidence such as dataset documentation, model evaluations, impact assessments and drift monitoring.

Load video
YouTube
How Should We Evaluate Machine Learning for AI?: Percy Liang
YouTube: https://www.youtube.com/watch?v=7CcSm0PAr-Y

Machine learning has undoubtedly been hugely successful in driving progress in AI, but it implicitly brings with it the train-test evaluation paradigm. This standard evaluation only encourages behavior that is good on average; it does not ensure robustness as demonstrated by adversarial examples, and it breaks down for tasks such as dialogue that are interactive or do not have a correct answer. In this talk, I will describe alternative evaluation paradigms with a focus on natural language understanding tasks, and discuss ramifications for guiding progress in AI in meaningful directions. Percy Liang is an Assistant Professor of Computer Science at Stanford University (B.S. from MIT, 2004; Ph.D. from UC Berkeley, 2011). His research spans machine learning and natural language processing, with the goal of developing trustworthy agents that can communicate effectively with people and improve over time through interaction. Specific topics include question answering, dialogue, program induction, interactive learning, and reliable machine learning. His awards include the IJCAI Computers and Thought Award (2016), an NSF CAREER Award (2016), a Sloan Research Fellowship (2015), and a Microsoft Research Faculty Fellowship (2014). (Liang later led Stanford's HELM project; his 2025 lecture on evaluation is listed under Additional Viewing.)

Load video
YouTube
Machine Learning, Technical Debt, and You - D. Sculley (Google)
YouTube: https://www.youtube.com/watch?v=V18AsBIHlWs

Machine Learning offers a fantastically powerful toolkit for building useful complex prediction systems quickly. In this talk, we'll argue it is dangerous to think of these quick wins as coming for free. Using the software engineering framework of technical debt, we find it is common to incur massive ongoing maintenance costs in real-world ML systems. We explore several ML-specific risk factors to account for in system design. These include boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, changes in the external world, and a variety of system-level anti-patterns. We then show how to pay down ML technical debt by following a set of recommended best practices for testing and monitoring needed for real world systems. D. Sculley is a Senior Staff Software Engineer at Google

Additional Viewing

Load video
YouTube

Stanford CS336, Lecture 12: Evaluation (Percy Liang, Stanford Online, lecture of May 2025, 1 hour 20 minutes). Liang surveys how language models are evaluated today, from perplexity to knowledge and reasoning benchmarks such as MMLU, GPQA and AIME, preference-based evaluation such as Chatbot Arena, agent and safety benchmarks, and the problem of overlap between training and test data. He frames evaluation as a chain of choices (which inputs, how the model is prompted, how outputs are scored, how results are interpreted) that must follow from the question being asked, which makes this the best current academic companion to the benchmark issues under Recent Developments. Watch on YouTube

Load video
YouTube

Why AI evals are the hottest new skill for product builders (Lenny's Podcast with Hamel Husain and Shreya Shankar, September 2025, 1 hour 46 minutes). Husain and Shankar teach a course on AI evaluations, and Shankar is lead author of Who Validates the Validators?; they walk through error analysis of real transcripts, grouping failures into categories, choosing between code-based checks and LLM-as-a-judge, using binary pass or fail criteria, and validating judges against human labels. It is a practical guide to the generative AI questions in How is AI success measured? Watch on YouTube

Load video
YouTube

Beth Barnes on the most important graph in AI right now (80,000 Hours Podcast episode 217, June 2025, nearly four hours). Barnes, chief executive of METR, explains how METR measures the time horizon of AI agents, why it has doubled roughly every seven months, and what evaluations can and cannot reveal, including models whose reasoning is hard to inspect and the case for testing models before internal use, not only before public release. It gives the reasoning behind the time-horizon results above from the team that produced them. Watch on YouTube

Load video
YouTube

Building and evaluating AI agents that matter (Sayash Kapoor, Princeton, AI Engineer Summit 2025, 20 minutes). Kapoor, lead author of the Holistic Agent Leaderboard paper, argues that agent evaluations must test the real task, report cost alongside accuracy and separate capability (succeeding once in many tries) from reliability (succeeding every time). He warns that with imperfect verifiers, more attempts can mean more wrong answers, a short and direct case for the reliability measures in The Physics. Watch on YouTube

Key Sources

Frameworks, standards and policy

Testing practice and readiness

Benchmarks and real-work evaluation

Agents and judges

Safety and security

Cases and adoption

Retrieved from "https://primo.ai/w/index.php?title=Evaluation&oldid=46773"