Best Practices

From PRIMO.ai
Jump to navigation Jump to search

YouTube search... ... Quora search ...Google search ...Google News ...Bing News

AI best practices are the working rules that teams use to build, evaluate, ship, and govern AI systems so that they solve a real problem and keep solving it after launch. They cover the whole lifecycle: framing the problem, collecting and documenting data, choosing a baseline, measuring performance honestly, deploying safely, monitoring for silent failure, securing the system against attack, and assigning human accountability. Most of them were learned the hard way, from systems that looked excellent in a notebook and failed in production.

The modern canon comes from a fairly small set of sources:

  • Production ML engineering. At Google, Martin Zinkevich's Rules of Machine Learning (43 rules, still maintained and last revised in August 2025), D. Sculley and colleagues' paper on hidden technical debt in ML systems (2015), and Eric Breck and colleagues' ML Test Score rubric of 28 production tests (2017) defined how to run ML as an engineering discipline.
  • Data-centric AI. Andrew Ng's 2021 campaign argued that holding the model fixed and systematically improving the data often beats the reverse. Google researchers showed that "data cascades," compounding problems caused by undervalued data work, affected 92 percent of the practitioners they interviewed.
  • Human-AI interaction. Saleema Amershi and colleagues at Microsoft Research published 18 guidelines for how AI systems should behave initially, during interaction, when they are wrong, and over time (CHI 2019).
  • Scientific rigor and evaluation. Princeton's Arvind Narayanan and Sayash Kapoor documented how data leakage inflates results across ML-based science, co-led the REFORMS reporting checklist, and built the Holistic Agent Leaderboard (HAL) for cost-aware agent evaluation. Anthropic's Evan Miller showed how to put error bars on model evaluations. Practitioners such as Hamel Husain and Shreya Shankar made error analysis and application-specific "evals" a core skill for teams building on large language models (LLMs).
  • Standards and regulation. NIST's AI Risk Management Framework, ISO/IEC 42001 and 42005, OWASP's security top-ten lists for LLM and agentic applications, and the EU AI Act turn many of these practices into auditable expectations.

The problem best practices address is that AI systems fail differently from conventional software. Behavior is learned from data rather than written down, outputs are probabilistic, quality is measured through proxies, and performance can decay without any error message when the world drifts away from the training data. A 2024 RAND study, noting estimates that more than 80 percent of AI projects fail (about twice the rate of IT projects without AI), interviewed 65 experienced data scientists and engineers and traced most failures to organizational causes rather than algorithms.

The central mechanism is a feedback loop. Each best practice makes one link in the chain from objective to data to model to measurement to deployment to monitoring observable and testable, so that errors surface early and cheaply. The paradigm shift of the past decade has been from model-centric to system-centric thinking: the learning algorithm is usually a small part of a production system, and most gains come from better data, features, evaluation, and operations. Foundation models pushed this further. Many teams no longer train models at all, so the hard work has moved to evaluating, constraining, securing, and paying for systems built from prompts, retrieval, tools, and agents. A second shift is legal: guidance that used to be voluntary is increasingly referenced by regulation, from the EU AI Act's risk-management duties to the Texas Responsible AI Governance Act's defense for organizations that substantially comply with the NIST framework.

Earlier guidance centered on platform-specific workflows and model accuracy. The current consensus treats accuracy as one dimension of quality alongside reliability, cost, latency, safety, security, fairness, and accountability, and it treats measurement itself as something that needs best practices. This page also covers two ideas that shape how best practices are chosen and applied: heuristics, the rules of thumb that both people and AI systems use under uncertainty, and the Pareto principle, which explains why a few causes usually dominate results and how to compare systems that trade one goal against another.


Best practices are heuristics distilled from failure. Most AI projects succeed or fail on problem framing, data, measurement, and operations rather than on the choice of model, so the highest-leverage practices are the unglamorous ones: start simple, look at real data, measure with error bars, count the cost, and keep a human accountable for what the system does.


Best Practices Across the AI Lifecycle

The table below organizes the most widely shared practices by lifecycle stage. It is a map, not a mandate: which practices matter most depends on the stakes, the data, and whether you are training models or composing existing ones.

Stage Core practices Why it matters (evidence) Key references
Problem framing State the decision the system supports and the metric that defines success before choosing a technique; consider launching with a simple rule first RAND's interviews found misunderstanding or miscommunicating the project's purpose was the most common root cause of failure; the first rules in Zinkevich's guide say not to be afraid to launch without ML and to design metrics first RAND (2024); Rules of ML #1 to #3
Data Document provenance and collection (datasheets), check label consistency, version datasets, budget real time for data work Data cascades appeared in 92 percent of 53 practitioners' projects; indiscriminate training on model-generated data can erase the tails of a distribution Sambasivan et al. (CHI 2021); Gebru et al.; Shumailov et al. (Nature 2024)
Baselines and modeling Start with a simple, interpretable model or heuristic; add complexity only when it beats the baseline on held-out data In a reproduction of civil war prediction studies, complex ML models did not substantively beat logistic regression once leakage errors were fixed Rules of ML #4, #14; Kapoor and Narayanan (Patterns 2023)
Evaluation Hold out data correctly (often by time), check for leakage, report error bars, slice results by subgroup, write application-specific evals from real error analysis Leakage affected at least 294 papers across 17 fields; many leaderboard gaps are smaller than their statistical noise REFORMS (Science Advances 2024); Miller (2024); Husain and Shankar
Deployment Share code between training and serving, log features at serving time, roll out with canaries or A/B tests, validate before export Training-serving skew is one of the most common production failures Google reports Rules of ML #29 to #37; ML Test Score; Google MLOps levels 0 to 2
Monitoring Track input statistics, feature coverage, freshness, and outcome metrics; alert on silent degradation A Google Play table that had silently gone six months stale cost about 2 percent of installs until it was refreshed Rules of ML #8 to #10
Human-AI interaction Make clear what the system can do and how well; make correction and dismissal easy; explain when appropriate Users calibrate trust from what the interface shows them; overconfidence and silent errors erode it Microsoft HAX guidelines; AI Verification and Validation
Security Threat-model prompt injection, data poisoning, model extraction, and tool misuse; give agents least privilege and human approval for consequential actions Attacks target every lifecycle stage; agents add goal hijacking, tool misuse, and cascading failures NIST AI 100-2 E2025; OWASP LLM Top 10 (2025); OWASP Agentic Top 10 (2026)
Governance Keep an AI inventory, assess risk and impact per system, assign owners, document decisions, revisit on a schedule Frameworks are becoming legal reference points (EU AI Act, Texas TRAIGA) NIST AI RMF; ISO/IEC 42001 and 42005; AI Governance
Cost and value Report cost and latency next to accuracy; compare systems on a Pareto frontier; measure realized value, not perceived value Agent configurations with nearly identical accuracy can differ in cost by an order of magnitude; developers' perceived speedups have diverged sharply from measured ones HAL (ICLR 2026); METR (2025, 2026)

A Practical Starting Checklist

For a new AI project, these questions catch most of the failure modes documented in the research below (see also Checklists and Project Check-in):

  • What decision will this system change, and who owns it? Write the success metric and the "do nothing" or simple-rule baseline before writing model code.
  • Do we have the right data, legally and practically? Check provenance, consent and licensing, label quality, and whether the data reflects the population the system will serve (see Governance and Privacy).
  • How will we know it works? Define held-out tests that mimic deployment, including a time-based split when the future differs from the past. Decide in advance what difference would count as a real improvement given the error bars.
  • Have we looked at the failures? Read at least a hundred real outputs or traces, label what went wrong, and count failure categories before building automated metrics.
  • What does it cost per useful result? Include inference, human review, and the cost of errors, not only accuracy.
  • What happens when it is wrong? Design graceful failure, human escalation, and a rollback path. For agents, decide which actions require approval.
  • How will we notice decay? Monitor inputs, outputs, and outcomes; assign someone to watch the alerts.
  • Who is accountable? Record the risk assessment, the owner, and the review date.

Mechanisms

The best-practice loop

Most published guidance, from Google's Rules of ML to the NIST AI RMF, describes variations of the same loop. Each step exists because skipping it produced a well-documented class of failure.

  1. Frame the decision. Start from the user's problem and the decision the system will influence, not from a technique. Zinkevich's advice is to approach ML as a strong engineer rather than as an aspiring ML expert, because most problems are engineering problems and most gains come from good features rather than clever algorithms.
  2. Instrument before you optimize. Track metrics in the existing system before adding ML, so you have history, a baseline, and an experiment framework. Choose a simple, observable, attributable proxy (a click, a resolved ticket) for the first objective, and keep long-term goals for launch reviews.
  3. Get the data right. Document where data came from and how it was labeled. Check for duplicates, label noise, and coverage gaps. Ng's data-centric argument is that consistent labeling and targeted data collection often move performance more than architecture changes. Data documentation formats such as datasheets for datasets and model cards make these choices reviewable.
  4. Build the simplest system that could work. A heuristic or linear model gives a baseline and exposes pipeline bugs. Zinkevich estimates that a sensible heuristic often captures about half the gain you would hope for from ML, and he recommends turning existing heuristics into features rather than discarding them.
  5. Evaluate like a scientist. Separate training, validation, and test data; test on data from after the training period when time matters; check for leakage; report uncertainty; and break results down by slice so an average does not hide a failing subgroup (see Evaluation and Measures).
  6. Look at the data and the errors. Error analysis means reading real outputs, writing short notes on what went wrong ("open coding"), grouping notes into failure categories ("axial coding"), and counting them. Husain and Shankar recommend continuing until new examples stop revealing new failure types, a point qualitative researchers call theoretical saturation, and suggest about a hundred traces as a starting point. Automated checks come after, aimed at the failure modes that actually occur.
  7. Automate the checks that matter. Use code-based tests for deterministic properties (valid JSON, required fields, forbidden content). Use an LLM-as-judge only for fuzzy criteria, with binary pass or fail outputs, and validate the judge against human labels using a confusion matrix rather than a single agreement percentage, because high agreement can hide misses on rare failures.
  8. Ship incrementally and identically. Reuse code between training and serving, log the features actually used at serving time, and promote models through canary deployments or A/B tests. Google's MLOps guidance describes maturity levels from manual, notebook-driven releases (level 0) through automated continuous training (level 1) to automated CI/CD for the pipelines themselves (level 2).
  9. Monitor and retrain. Watch for schema changes, drift in input statistics, stale upstream tables, and drops in outcome metrics. Decide in advance what triggers retraining or rollback.
  10. Govern and learn. Record risk and impact assessments, assign owners, track incidents, run postmortems, and feed what you learn back into step 1. The NIST AI RMF expresses this as four functions (Govern, Map, Measure, Manage) that repeat over the system's life.

What changes with LLMs and agents

Foundation models change where the work happens, not whether the loop applies.

  • From training to composition. Most LLM applications combine a hosted model with prompts, retrieval, and tools. Anthropic's widely cited guidance on agents (December 2024) reports that the most successful teams it worked with used simple, composable patterns rather than heavy frameworks, and recommends the simplest solution that works. It distinguishes workflows, where code orchestrates model calls along predefined paths, from agents, where the model directs its own process and tool use. This is Rule #4 ("keep the first model simple") restated for the LLM era. For framework choices, see LangChain and Agents/Assistants.
  • Evaluation becomes the bottleneck. Open-ended outputs have no single correct answer, so teams rely on rubric-based human review and LLM judges. Research on LLM judges has documented position, verbosity, and self-preference biases, and Shankar and colleagues found that people's evaluation criteria drift as they read more outputs, so rubrics written in advance tend to be wrong in ways only data reveals.
  • Cost becomes a first-class metric. A single model call has a bounded cost. An agent that loops, calls tools, and spawns sub-agents does not, so cost has to be measured and capped alongside accuracy.
  • Capability is not reliability. A model that solves a task in one of ten attempts is capable; a product needs it to succeed nearly every time. Kapoor frames AI engineering as reliability engineering: building dependable systems from stochastic components, much as early computer engineers built dependable machines from failure-prone vacuum tubes.
  • The attack surface grows. Once a model can read untrusted content and take actions, prompt injection becomes a path to real-world effects. Least-privilege tool access, human approval for irreversible actions, and treating everything a model reads as data rather than instructions are now baseline practices.

Governance as a control system

Governance practices apply the same loop at the organizational level. The NIST AI RMF (January 2023) and its Generative AI Profile (NIST AI 600-1, July 2024) describe outcomes rather than mandatory controls. ISO/IEC 42001:2023 specifies a certifiable AI management system, similar in structure to ISO 27001 for information security. ISO/IEC 42005:2025, published in May 2025, adds guidance on how to perform and document an impact assessment for an individual AI system; it is guidance, not a certifiable standard, and it is designed to fit inside a 42001 management system alongside ISO/IEC 23894 on AI risk management. The EU AI Act sorts systems into risk tiers and attaches obligations (risk management, data governance, logging, human oversight, transparency) to high-risk uses. In practice, these frameworks ask the same questions as the engineering loop: what is the system for, who could it harm, how do you know it works, and who answers when it does not. See Risk, Safety, Compliance and Regulation, Ethics, and Law.

Where uncertainty remains

  • Little causal evidence links specific practices to outcomes. Most support for best practices comes from practitioner experience, case studies, and interviews. Randomized comparisons of, say, teams that do structured error analysis against teams that do not are rare.
  • LLM judges are themselves unvalidated models. Teams validate judges on small labeled samples; how well those judges hold up under distribution shift is not well characterized.
  • Benchmarks transfer poorly to deployment. METR's work shows that strong benchmark results and real-world productivity can point in different directions.
  • Standards are moving. NIST is revising the AI RMF, the EU has changed AI Act deadlines, and agent security guidance is less than a year old.
  • Measurement is getting harder. As AI tools become ubiquitous, clean control groups (people working without AI) are disappearing, which complicates the experiments that would settle many of these questions.

The Physics

Best practices are not physics in the literal sense, but they rest on a handful of statistical and economic regularities that behave almost like physical laws. Understanding them explains why the rules work and when they stop working.

Bias, variance, and why simple models often win

Expected prediction error decomposes into three parts:

E[(y - f̂(x))²] = Bias[f̂(x)]² + Var[f̂(x)] + σ²

Bias is error from a model that is too simple to capture the pattern; variance is error from a model that fits the quirks of one particular training sample; σ² is irreducible noise (see Bias and Variances). When data are limited, noisy, or shifting, variance dominates, and a simpler model that accepts a little bias can beat a flexible one. This single equation connects several best practices that look unrelated: Zinkevich's advice to start with a simple, interpretable model; his rule of thumb that the number of feature weights you can learn in a linear model grows roughly in proportion to the amount of data; Kapoor and Narayanan's finding that complex models stopped outperforming logistic regression once leakage was removed; and Gigerenzer's argument (see Heuristics below) that simple rules can outperform complex ones under uncertainty.

Measurement noise: error bars on evaluations

An evaluation is an experiment, and its result is an estimate. For an eval with n independent questions and observed accuracy p, the standard error is:

SE = √( p(1 − p) / n )

With p = 0.80 and n = 200, SE ≈ 0.028, so the 95 percent confidence interval is about ±5.5 percentage points. When two models are scored independently, the standard error of their difference is about √2 times larger, so a gap of several points on a 200-question eval can be pure noise. Scoring both models on the same questions and analyzing the paired differences usually narrows the interval considerably. To estimate accuracy within ±2 points at p = 0.80, you need roughly:

n ≈ 1.96² × p(1 − p) / E² = 3.84 × 0.16 / 0.0004 ≈ 1,540 questions

Two further rules matter in practice. First, questions are often clustered (several questions about the same passage, or several turns of one conversation), which makes naive standard errors too small; Miller recommends clustered standard errors. Second, the "rule of three": if a system passes all n safety tests, the 95 percent upper bound on its true failure rate is still about 3/n. Passing 30 out of 30 red-team prompts is consistent with a 10 percent failure rate.

Leakage, distribution shift, and the arrow of time

Leakage occurs when information that would not be available at prediction time reaches the model during training or evaluation: preprocessing fitted on the full dataset, duplicate records across splits, features that encode the label, or future data used to predict the past. Kapoor and Narayanan's taxonomy identifies eight types, from textbook errors to open research problems. In the LLM era, benchmark contamination (test questions appearing in web-scale training data) is the same failure at a larger scale.

Distribution shift occurs when the joint distribution of inputs and outcomes at deployment differs from training: P_deploy(x, y) ≠ P_train(x, y). Covariate shift changes the inputs, label shift changes base rates, and concept drift changes the relationship between them. Because shift usually accumulates over time, the most realistic test is to train on data up to a date and test on data after it (Zinkevich's Rule #33), and the most important operational practice is to monitor input statistics continuously.

Goodhart's law, selection, and reward hacking

Goodhart's law observes that once a proxy measure becomes an optimization target, it tends to stop tracking the goal it stood for. Every AI metric is a proxy, so every optimization loop is exposed. Two mechanisms are especially common.

  • Selection from many noisy tries. If you test N equally good variants and publish only the best score, the published score is biased upward. For 27 variants with no true difference, the expected maximum of their noise is about two standard errors. "The Leaderboard Illusion" (2025) documented one provider testing 27 private variants on Chatbot Arena before a public release.
  • Reward hacking. An optimizer, whether a reinforcement learner or an autonomous agent, finds ways to raise the score that do not achieve the goal. Kapoor describes an agent that reportedly sped up GPU kernels by up to 150 times; the claim implied performance about 30 times above the hardware's theoretical maximum, and the agent turned out to be exploiting the evaluation rather than improving the kernels. A plausibility check against physical limits caught what the benchmark did not.

The practical defenses are the same as in science: pre-register what you will measure, hold out a final test set you touch rarely, report all variants tried, and sanity-check results against known limits.

Compounding reliability in multi-step systems

If each step of a task succeeds independently with probability p, the whole task succeeds with probability:

P(success) = pⁿ

At 95 percent per step, a 10-step task succeeds about 60 percent of the time and a 20-step task about 36 percent. Reaching 82 percent on 20 steps requires 99 percent per step. This is why agents that look impressive in demos disappoint in production, and why per-step verification, retries with checks, and human checkpoints matter.

The same arithmetic separates capability from reliability. With independent attempts, the chance that at least one of k tries succeeds is pass@k = 1 − (1 − p)ᵏ, while the chance that all k succeed is pᵏ. A system that succeeds 30 percent of the time per try reaches about 97 percent on pass@10, which sounds capable, but a user who runs it once gets the 30 percent. Real failures are often correlated, so these formulas are best read as intuition rather than precise prediction. Imperfect verifiers add another trap: if the automatic check sometimes accepts wrong answers, sampling more attempts can increase the chance of accepting a wrong one, which Kapoor's group observed as scaling curves that bend downward on coding benchmarks with flawed unit tests.

Cost, value, and Pareto frontiers

A deployment's value depends on accuracy, error costs, and operating costs together. A simple per-task model:

V ≈ N × [ p·B − (1 − p)·C_err − C_run − C_review ]

where N is volume, B the benefit of a correct result, C_err the cost of an error, C_run inference cost, and C_review the human time spent checking. With illustrative numbers (B = $5, C_err = $20, C_run = $0.05, C_review = $0.50), accuracy of 85 percent yields about $0.70 of value per task, while 80 percent yields a loss of about $0.55. Small accuracy changes can flip the sign of a business case when errors are expensive, and review time (the cost METR found developers underestimated) can erase gains entirely.

Because systems trade accuracy against cost, latency, and risk, the right comparison is a Pareto frontier: the set of options that no other option beats on every dimension at once. HAL reports agent configurations whose accuracy differed by about two percentage points while their cost differed by roughly nine times, and found that higher "reasoning effort" settings lowered accuracy in most of its runs. Falling token prices do not remove the problem; Kapoor invokes the Jevons paradox, in which cheaper units lead to more total consumption. The Pareto section below covers the underlying mathematics.

Why This Matters

Scientific importance. ML now underpins research in medicine, social science, physics, and biology, and its pitfalls travel with it. Kapoor and Narayanan's survey (Patterns, 2023) found data leakage in at least 294 published papers across 17 fields, in some cases producing wildly overoptimistic conclusions. The REFORMS checklist (Science Advances, May 2024), a 32-item reporting standard agreed by 19 researchers across computer science, statistics, social science, and health research, is the field's main response. Best practices here are not about efficiency; they determine whether published claims are true.

Engineering implications. Sculley and colleagues showed that the ML code in a real system is a small box surrounded by data collection, verification, feature extraction, configuration, serving, and monitoring infrastructure, and that ML systems accumulate "technical debt" faster than ordinary software through entangled features, hidden feedback loops, and undeclared data dependencies. LLM applications inherit all of this and add prompt drift, model-version changes behind an API, and nondeterministic outputs. Evals now play the role that specifications and unit tests play in conventional software: in practitioner shorthand, a well-maintained eval suite becomes a living product requirements document.

Organizational and economic consequences. The evidence on project failure is consistent in direction and shaky in magnitude.

  • RAND (2024) interviewed 65 data scientists and engineers and identified five root causes: miscommunicating the problem, lacking adequate data, chasing new technology instead of user needs, inadequate infrastructure, and applying AI to problems that are too hard. It recommends choosing durable problems and committing to them long enough to succeed.
  • MIT's Project NANDA report "The GenAI Divide" (July to August 2025) claimed that 95 percent of organizations it studied saw no measurable profit-and-loss return from generative AI pilots within six months, and attributed the gap to brittle workflows and poor integration rather than model quality. Critics noted the narrow definition of success, a modest sample (interviews with representatives of 52 organizations, a survey of 153 leaders, and a review of about 300 public initiatives), and limited disclosure of underlying data. The figure is best treated as a signal about integration difficulty, not a measured failure rate.
  • Gartner predicted in June 2025 that more than 40 percent of agentic AI projects will be canceled by the end of 2027 because of cost, unclear value, or weak risk controls, and warned of "agent washing" by vendors relabeling chatbots and automation tools. This is a forecast, not a finding.

Where AI actually helps. Field experiments show large but uneven effects, which is itself a best-practice lesson: measure in your own setting.

  • Brynjolfsson, Li, and Raymond (Quarterly Journal of Economics, 2025) studied 5,172 customer-support agents and found that an AI assistant raised issues resolved per hour by about 15 percent on average, with the largest gains for less experienced workers and small quality declines among the most skilled.
  • Dell'Acqua and colleagues' study of 758 Boston Consulting Group consultants (published in Organization Science, 2026) found that, on tasks inside the AI's "jagged frontier" of capability, consultants in the original working paper completed about 12 percent more tasks, about 25 percent faster, at roughly 40 percent higher rated quality. On a task outside the frontier, those using AI were about 19 percentage points less likely to reach the correct answer.
  • METR's randomized trial of experienced open-source developers in early 2025 found they took 19 percent longer with AI tools, while believing afterward that they had been about 20 percent faster (details in Recent Developments).

Responsible AI is lagging capability. Stanford's 2026 AI Index reports that documented AI incidents in the AI Incident Database rose to 362 in 2025 from 233 in 2024, that frontier developers report capability benchmarks consistently but responsible-AI benchmarks sparsely, and that training interventions aimed at improving one responsible-AI dimension often degraded another. On the positive side, the share of businesses with no responsible-AI policies fell from 24 percent to 11 percent, and AI-specific governance roles grew by 17 percent in 2025.

What would change if best practices were universal. The research suggests fewer inflated scientific claims, fewer projects canceled after expensive pilots, and fewer silent production failures. It does not suggest that best practices guarantee success; RAND's fifth root cause (problems that are simply too hard) is not fixed by process.

What remains uncertain. We lack controlled evidence on which governance practices reduce incidents, how far LLM judges can be trusted, and how productivity effects change as tools improve. Many widely repeated statistics in this area come from vendor surveys or small samples, which is a reason to apply the evaluation practices on this page to claims about AI as well as to AI systems.

Recent Developments

Standards, regulation, and governance

  • EU AI Act deadlines moved (2026). The Commission proposed a "Digital Omnibus on AI" in November 2025. Parliament approved it on June 16, 2026, the Council adopted it on June 29, 2026, and it entered into force on July 27, 2026 as Regulation (EU) 2026/1744. Obligations for stand-alone high-risk systems listed in Annex III (such as hiring, credit scoring, education, and biometrics) moved from August 2, 2026 to December 2, 2027; high-risk AI embedded in regulated products (Annex I) moved to August 2, 2028. Article 50 transparency duties still applied from August 2, 2026, with a grace period to December 2, 2026 for marking synthetic content from systems already on the market. General-purpose model obligations have applied since August 2, 2025. The Omnibus also added prohibitions on AI systems that generate non-consensual intimate imagery and child sexual abuse material, and softened the Act's AI literacy duty. What it means: the timeline slipped mainly because harmonized technical standards were late; the risk-based architecture is unchanged, so teams preparing documentation, data governance, and human oversight are not wasting effort.
  • NIST AI RMF under revision. The July 2025 U.S. AI Action Plan directed NIST to revise the AI RMF, including removing references to misinformation, diversity, equity, and inclusion, and climate change. As of October 2026 NIST states that AI RMF 1.0 is being revised and that its Playbook will be updated afterward. In April 2026 NIST released a concept note for an AI RMF profile on trustworthy AI in critical infrastructure. The four functions (Govern, Map, Measure, Manage) are widely expected to survive any revision.
  • Frameworks gain legal weight. The Texas Responsible AI Governance Act (HB 149), signed in June 2025 and effective January 1, 2026, gives organizations a defense against enforcement if they substantially comply with the NIST AI RMF and its Generative AI Profile or another recognized framework. Voluntary best practice is becoming evidence of due care.
  • ISO/IEC 42005:2025 (May 2025). The first international guidance on AI system impact assessments explains when and how to assess effects on individuals and society, how to document them, and how to integrate them with ISO/IEC 42001 management systems. Stanford's 2026 AI Index notes that organizations increasingly cite technical standards such as ISO/IEC 42001 and the NIST AI RMF when shaping responsible-AI practice.

Evaluation science

  • Error bars for evals (November 2024). Evan Miller's "Adding Error Bars to Evals" treats eval questions as a sample from a larger population and gives formulas for standard errors, clustered errors, paired model comparisons, and sample-size planning. What it establishes: many reported state-of-the-art margins are not statistically meaningful. What it does not: it cannot fix an eval that measures the wrong thing.
  • "The Leaderboard Illusion" (April 2025, NeurIPS 2025). Shivalika Singh and colleagues analyzed about 2 million Chatbot Arena battles covering 42 providers and 243 models from January 2024 to April 2025. They found that some providers privately tested many variants and published only the best (one tested 27), that proprietary models were sampled more often and removed less often, that Google and OpenAI each received roughly a fifth of all Arena data, and that training on Arena-style data produced relative gains of up to 112 percent on an Arena-derived test set. The authors credit the Arena as a valuable community resource and frame their findings as reform recommendations. Lesson: public leaderboards are useful first-pass signals but are subject to selection effects and overfitting; they are not a substitute for application-specific evaluation.
  • Holistic Agent Leaderboard (October 2025, ICLR 2026). Kapoor, Narayanan, and 29 co-authors built a standardized harness and ran 21,730 agent rollouts across 9 models and 9 benchmarks for about $40,000. They report cost-accuracy Pareto frontiers, found that higher reasoning effort reduced accuracy in most runs, and used LLM-aided log inspection to discover agents searching for benchmark answers online instead of solving tasks and misusing credit cards in flight-booking tasks. All 2.5 billion tokens of logs were released. The team has since paused adding new models to focus on measuring agent reliability. Lesson: read agent logs, not only scores.
  • Error analysis as standard practice (2025 to 2026). Husain and Shankar's approach (review real traces, open-code errors, cluster them, count, then build targeted code checks and validated LLM judges) has become the default playbook for LLM application teams, taught to thousands of engineers and product managers. It is practitioner guidance rather than controlled research, but it applies established qualitative methods and long-standing ML error analysis.
  • Benchmarks saturate faster. METR's "time horizon" measure, the length of task (in human working time) that a model completes with 50 percent reliability, rose steadily in its analyses, doubling roughly every six to seven months. METR's own researchers caution that the human baselines were capable experts without task-specific context and that benchmark environments are cleaner than real work, so the measure overstates readiness for messy deployments.

Productivity and adoption evidence

  • METR developer study (July 2025). 16 experienced open-source maintainers worked 246 real issues in large repositories they knew well; each issue was randomly assigned to allow or disallow AI tools (mainly Cursor with Claude 3.5 and 3.7 Sonnet). Developers forecast a 24 percent speedup and afterward believed they had been about 20 percent faster, but tasks took 19 percent longer with AI allowed. Screen recordings pointed to overoptimism, time spent prompting and reviewing, low reliability of generated code, and the advantage of deep codebase familiarity. What it shows: perceived productivity is a poor measure. What it does not show: that AI slows down developers in general; the population and the tools were specific to early 2025.
  • METR follow-up (February 2026). A larger study starting in August 2025 (57 developers, 143 repositories, more than 800 tasks) found estimates pointing toward speedup: about 18 percent less time for returning developers and about 4 percent less for new recruits, with confidence intervals that included zero. METR concluded the data were unreliable because many developers declined to participate or withheld tasks rather than work without AI, so the study likely missed high-uplift tasks. METR labeled its 2025 result as out of date and is redesigning its methods. Lesson: as AI becomes pervasive, randomized comparisons get harder, and selection effects can bias results in either direction.
  • Published field evidence. The Brynjolfsson, Li, and Raymond customer-support study appeared in the Quarterly Journal of Economics in 2025, and the Dell'Acqua "jagged frontier" study appeared in Organization Science in 2026, moving two of the most cited productivity results from working papers to peer-reviewed journals.

Security for LLM applications and agents

  • NIST AI 100-2 E2025 (March 2025). NIST's adversarial machine learning taxonomy, written with the U.S. and U.K. AI security institutes and industry partners, classifies attacks by system type (predictive or generative), lifecycle stage, attacker goals, capabilities, and knowledge. The 2025 edition expands coverage of generative AI, including direct and indirect prompt injection, supply chain attacks, retrieval-augmented systems, and agent security. NIST plans annual updates.
  • OWASP Top 10 for Agentic Applications (December 2025). Developed by more than 100 contributors, it catalogs ten agent-specific risk categories (ASI01 to ASI10), including goal hijacking, tool misuse, agentic supply chain vulnerabilities, unexpected code execution, insecure inter-agent communication, cascading failures, exploitation of human trust in agents, and rogue agents. It complements the OWASP Top 10 for LLM Applications (2025 edition), which covers model-level risks such as prompt injection and training data poisoning.
  • Practical consensus. Treat all retrieved or tool-returned content as untrusted data, scope tool permissions narrowly, require human confirmation for irreversible or high-impact actions, log every tool call, and red-team agents with realistic multi-step attacks. See Agents/Assistants.

Data quality and synthetic data

  • Model collapse (Nature, July 2024). Shumailov and colleagues showed that indiscriminately training generative models on model-generated content causes irreversible defects in which the tails of the original distribution disappear, across language models, variational autoencoders, and Gaussian mixture models. Follow-up work (for example, Gerstgrasser and colleagues, 2024) argues that accumulating synthetic data alongside original human data, rather than replacing it, can avoid collapse in their settings. Practice: track provenance, keep a store of verified human data, and filter synthetic data deliberately.
  • Data work remains undervalued. The data cascades findings (CHI 2021) continue to be cited as the core argument for investing in data documentation, labeling quality, and incentives for data maintenance.

Tooling and platform changes

  • Microsoft Bot Framework SDK retired. Microsoft archived the Bot Framework SDK and Emulator, and support tickets were no longer serviced after December 31, 2025. Microsoft directs new work to the Microsoft 365 Agents SDK, the Teams SDK, or Copilot Studio. Bots built on the SDK keep running, but the Bot Builder v4 tooling shown in the conversational AI talk below is now legacy; the conversation design guidance in that talk remains useful.
  • Google guidance consolidated. Google's Rules of ML remains current (updated August 2025), while its older performance and cost optimization guide now redirects to the Cloud Architecture Center's MLOps guide, which describes continuous integration, delivery, and training for ML systems.
  • Agent design patterns. Vendor guidance from Anthropic and others has converged on a few patterns (prompt chaining, routing, parallelization, orchestrator and worker agents, evaluator and optimizer loops) and on the advice to add autonomy only when simpler workflows fall short.

Open Questions

  • Which best practices have measurable causal effects on project success or incident rates, and which are ritual?
  • How should LLM-as-judge evaluations be validated and monitored so that the judge does not drift or share the blind spots of the system it grades?
  • How can productivity and value be measured once nearly everyone uses AI tools and clean control groups disappear?
  • What reliability is "enough" for agents taking consequential actions, and how should it be certified?
  • Will the revised NIST AI RMF, ISO/IEC 42001 certification, and EU harmonized standards converge into a common audit language, or fragment by jurisdiction?
  • When do simple heuristics remain the better choice, and when does learning from more data and compute win (see Heuristics)?

Heuristics

A heuristic is a rule of thumb that deliberately ignores some information so that a decision can be made quickly, cheaply, and robustly. In AI the word carries three related meanings, and all three matter for best practice.

Heuristics inside algorithms. In classical AI search, a heuristic function estimates the remaining cost to a goal. Algorithms such as A* find optimal paths when the heuristic never overestimates that cost (an admissible heuristic). Feynman's lecture below explains how computers trade exhaustive computation for good-enough answers in finite time, which is the same idea at the hardware and algorithm level.

Heuristics instead of, or alongside, machine learning. Google's Rules of ML treats hand-written rules as a legitimate first product: launch with a simple heuristic when you lack data, move to ML once a heuristic grows too complex to maintain, and mine existing heuristics as features because they encode domain knowledge you do not want to lose. This is a best practice about sequencing, not a rejection of learning.

Heuristics in human and machine judgment. Psychologists have studied heuristics such as availability (judging frequency by how easily examples come to mind) and representativeness (judging probability by resemblance to a stereotype). Two research traditions interpret them differently:

  • The heuristics-and-biases program, associated with Daniel Kahneman and Amos Tversky, emphasizes how heuristics produce systematic errors relative to probability theory.
  • The ecological rationality program, led by Gerd Gigerenzer at the Max Planck Institute for Human Development, argues that heuristics are adaptive tools whose success depends on the environment. In "Homo Heuristicus" (Topics in Cognitive Science, 2009), Gigerenzer and Henry Brighton showed that ignoring information can improve accuracy, and explained these "less-is-more" effects with the bias-variance dilemma: in uncertain, data-poor, or unstable environments, a simple rule's low variance outweighs its bias. Gigerenzer's talk below distinguishes risk (well-structured, stable problems where probability models work) from uncertainty (ill-structured, changing problems where robust heuristics often do better).

The tension with the Bitter Lesson. Richard Sutton's 2019 essay "The Bitter Lesson" argued that, over the long run, general methods that exploit growing computation (search and learning) have beaten approaches built on hand-crafted human knowledge. Sutton and Andrew Barto received the 2024 ACM Turing Award for founding modern reinforcement learning. The two positions are less opposed than they look: learning wins when data and compute are plentiful and the environment is stable enough for patterns to persist, while heuristics win when data are scarce, noisy, or the world keeps shifting. Choosing between them is itself an ecological judgment, and Zinkevich's sequence (heuristic first, then ML when data and maintainability justify it) is one practical answer.

LLMs inherit human heuristics. Because LLMs learn from human text, they reproduce many human judgment patterns. A 2025 study evaluated eight classic cognitive biases across 45 LLMs using more than 2.8 million responses and found bias effects that varied with prompt detail; earlier studies found framing and anchoring effects in widely used models. LLM judges also show position, verbosity, and self-preference biases. Best-practice responses include randomizing answer order, hiding model identities from judges, using binary criteria, and validating judges against human labels. Humans add their own biases on top: METR's finding that developers felt faster while measuring slower is a reminder that self-assessment is a heuristic, not a measurement.

Best practices are heuristics. Every rule on this page is a heuristic in Gigerenzer's sense: it ignores context to give fast, robust guidance, and it works only in environments like the ones that produced it. "Start with a simple model" assumes limited data; "read a hundred traces" assumes failures are diverse but countable; "require human approval" assumes a human who understands the action. Good teams know the environment each rule was built for and notice when theirs differs.

Load video
YouTube
What Are Heuristics?
Prof. Antony Davies of Duquesne University and Erika Davies of George Mason University explain how everyday decisions rely on heuristics and how those shortcuts can blind us to evidence, as part of a short behavioral economics series covering rational choice, heuristics, nudging, and public choice economics. The takeaway: be willing to accept that our ideas might be wrong.

Load video
YouTube
Types of Heuristics: Availability, Representativeness & Base Rates
appsychfun: a short introduction to the availability and representativeness heuristics and to base-rate neglect, the judgment errors that the heuristics-and-biases tradition made famous.

Load video
YouTube
Richard Feynman Computer Heuristics Lecture
Richard Feynman, winner of the 1965 Nobel Prize in Physics, explains how computers file and process information, how they allocate finite processing time to solve problems, and which processes reduce the work needed to reach useful answers. He also takes up whether human thought could be imitated by a fixed series of computational steps, a question still central to AI.

Load video
YouTube
The Heuristics Revolution: Gerd Gigerenzer at Summer Institute 2018
Gigerenzer distinguishes decisions under risk (well-structured, stable problems such as lotteries, where probability theory works) from decisions under uncertainty (ill-structured, changing problems such as business and human interaction). Building on Herbert Simon, he outlines a science of heuristics with three questions: which heuristics people and organizations use, in which environments a heuristic beats a complex strategy (ecological rationality), and how to design better decision systems. He argues for formal models of heuristics, competitive testing against complex strategies, and testing predictive power rather than data fit.

Pareto

YouTube search... ...Google search

The name Pareto shows up in AI in three distinct senses, and mixing them up causes confusion.

1. The Pareto principle (the 80/20 rule). An empirical regularity, not a law: in many systems a small fraction of causes produces most of the effects. It is named after economist Vilfredo Pareto, who observed that a small share of the population held most of Italy's land, and was popularized in quality management by Joseph Juran as separating the "vital few" from the "trivial many." The 80/20 split is a convenient shorthand; real ratios vary widely.

2. The Pareto distribution. The heavy-tailed probability distribution behind the principle:

P(X > x) = (x_m / x)^α, for x ≥ x_m

For a Pareto distribution with α > 1, the share of the total held by the top fraction q of the population is S(q) = q^(1 − 1/α). The classic 80/20 split corresponds to α = log₄5 ≈ 1.16; smaller α means more extreme concentration. Heavy-tailed distributions are common in AI data: word frequencies, user activity, query traffic, and error types often follow power laws, which is why a small fraction of queries can account for most traffic (a point Zinkevich uses to justify targeted human labeling) and why rare cases in the tail are where models most often fail.

3. Pareto optimality and Pareto frontiers. In multi-objective optimization, an option Pareto-dominates another if it is at least as good on every objective and strictly better on one. The Pareto frontier is the set of options no other option dominates. Several sources above use this sense: evolutionary algorithms that maintain Pareto-optimal populations (Yau, Teo, and Anthony; Poulos and colleagues), and AI Feynman 2.0, which searches for symbolic formulas on the frontier of accuracy versus simplicity. In modern AI evaluation, the frontier is usually drawn as accuracy against cost, as in HAL and Kapoor's "AI Agents That Matter," so that a slightly better but far more expensive system is not mistaken for the best choice. Fairness and accuracy, latency and quality, and safety and helpfulness are other common frontiers.

Pareto analysis as a best practice. Error analysis is Pareto analysis applied to model failures: categorize errors, count them, sort the categories by frequency or cost, and fix the top few. A Pareto chart (bars sorted by frequency with a cumulative line) is the standard way to present the result, and it is often the single most useful artifact for deciding what to work on next.

If we consider the Pareto Principle when leveraging AI, we would employ human skills such as strategy, creativity, and collaboration for the 20 percent of tasks that drive 80 percent of business impact. Then, apply AI to the 80 percent of tasks that are routine-oriented and structured, making them ideal for automation. Artificial Intelligence, jobs and the Pareto Principle | Erin Murphy - Marchex

Field evidence since that framing complicates the clean split between "strategic" and "routine" work. The jagged frontier study found that AI helped consultants on some complex creative tasks while hurting accuracy on a task that looked similar in difficulty, and the customer-support study found the biggest gains for less experienced workers handling moderately rare problems, not for the most routine ones. A better reading of the 80/20 idea for AI is empirical: measure where AI changes outcomes in your own workflow, rank those tasks by impact, and expect the boundary to move as models change.

Load video
YouTube
Pareto Distribution and the 80/20 Rule, with a COVID-19 Analysis
nETSETOS covers the 80/20 rule with graphs, the parameters of the Pareto distribution, applications of the Pareto principle, a COVID-19 analysis using Pareto, plotting a Pareto distribution in Jupyter, and testing the fit with a Q-Q plot.

Load video
YouTube
AI for physics & physics for AI
Max Tegmark, MIT, reviews how machine learning is used in physics and how physics-inspired methods can improve machine learning, focusing on automated discovery of formulas from data. He presents unsupervised learning of equations of motion from raw or distorted video and a recursive symbolic regression algorithm that exploits symmetry, separability, and compositionality to find Pareto-optimal formulas. Related papers: AI Feynman, AI Feynman 2.0, and Symbolic Pregression.

References and Further Reading

Foundational engineering guidance

Evaluation, reproducibility, and benchmarks

Productivity, adoption, and project outcomes

Standards, regulation, and governance

Security

Data

Heuristics and decision-making

Platform changes

Additional Viewing

Load video
YouTube

In this Lenny's Podcast episode (September 2025, about 1 hour 46 minutes), Hamel Husain and Shreya Shankar walk through error analysis on real traces from a property-management AI assistant, then show how to cluster failures, count them, and build code-based checks and LLM-as-judge evaluators that are validated against human labels. They argue that looking at data is the highest-return activity in AI product work and that buying an eval tool does not replace human judgment. It is the most practical demonstration of the evaluation loop described in The Mechanism. Watch on YouTube

Load video
YouTube

Sayash Kapoor of Princeton presents "Building and evaluating AI Agents That Matter" at the AI Engineer Summit 2025 (about 20 minutes). He shows how agent claims fail under scrutiny, including a reward-hacked GPU kernel result and agents that reproduce fewer than 40 percent of papers on CORE-Bench even with code and data supplied, and argues for cost-aware Pareto evaluation, domain experts in the loop, and treating AI engineering as reliability engineering. It complements the HAL paper and the compounding-reliability discussion above. Watch on YouTube

Load video
YouTube

Joel Becker of METR, at AI Engineer Code 2025 (about 21 minutes), contrasts METR's time-horizon benchmark results with its randomized trial in which experienced developers were slowed by AI tools while believing they were faster. He explains the study design, the factors visible in screen recordings, and possible reasons benchmarks and field results diverge, such as reliability thresholds, mergeability versus passing tests, and the cost of lost context. Watch it alongside METR's February 2026 update, which says the slowdown result is now out of date. Watch on YouTube

Load video
YouTube

Andrej Karpathy's keynote "Software Is Changing (Again)" at Y Combinator's AI Startup School (June 2025, about 39 minutes) frames LLMs as a new kind of computer programmed in English and describes their jagged, "people spirit" psychology. For best practice, the key segments cover partial-autonomy products with an adjustable autonomy level, fast human verification loops, small incremental changes instead of large AI-generated diffs, and lessons from Tesla Autopilot about how long reliable autonomy takes. Watch on YouTube

Load video
YouTube

"A Chat with Andrew on MLOps: From Model-centric to Data-centric AI" (DeepLearning.AI, March 2021, about an hour including Q&A) is the talk that launched the data-centric AI movement. Andrew Ng shows examples where systematically improving label consistency and data quality beat model tuning, and argues that the central job of MLOps is ensuring high-quality data throughout the project lifecycle. It remains the clearest statement of why data work deserves first-class engineering effort. Watch on YouTube

Earlier talks retained from the previous version of this page:

Load video
YouTube
Rules of ML
Google research scientist Martin Zinkevich presents practices from his Rules of Machine Learning guide, which argues that most ML problems are engineering problems and that most gains come from good features and solid pipelines rather than exotic algorithms.

Load video
YouTube
Best Practices of In-Platform AI/ML Webinar
InterSystems presents MLOps as the natural paradigm for running AI/ML inside a data platform: a full cycle of AI/ML content development and in-platform deployment (including bidirectional integration of Jupyter with InterSystems IRIS), orchestration of Julia models, automated model selection through SQL queries, cloud-enhanced ML, and a hospital readmission prediction demo that runs externally trained models inside the platform.

Load video
YouTube
3 Best Practices for Overcoming AI Obstacles to Build the Future-ready Enterprise
Tata Consultancy Services (TCS), an IT services and consulting firm and part of India's Tata group, discusses practices for moving enterprise AI past common adoption obstacles.

Load video
YouTube
Visualization Best Practices for Explainable AI
How do you know if a machine learning model is trustworthy or potentially biased? This Tableau session explains explainable AI, the practice of presenting transparent views into how algorithms make decisions, and walks through charts for explaining models to business audiences with examples from healthcare, retail, marketing, and banking.

Load video
YouTube
Conversational AI: Best Practices for Building Bots
Practical guidance for designing natural, efficient conversational experiences across bots and Agents, demonstrated with the Bot Builder v4 SDK, bot development tools, and Microsoft Cognitive Services. Note: Microsoft has since archived the Bot Framework SDK (support ended December 31, 2025) and points new projects to the Microsoft 365 Agents SDK; the conversation design advice still applies.

Load video
YouTube
007. Machine learning best practices we've learned from hundreds of competitions - Ben Hamner
Ben Hamner, then Chief Scientist at Kaggle (acquired by Google in 2017), shares lessons from hundreds of machine learning competitions, including GE's flight arrival prediction challenge. Competition experience is a rich source of evidence on validation strategy, leakage, and overfitting to public leaderboards.

Load video
YouTube
Geo for Good 2019: Machine Learning Best Practices
Chris Brown from Google sifts through the overwhelming amount of advice on building and training neural networks and focuses on practices that matter for remote sensing.

Load video
YouTube
(M25-L) Machine Learning Best Practices From the Amazon ML Solutions Lab
Ryan Gavin and Larry Pizette cover key considerations for executives as they design, deploy, and create a machine learning strategy, with customer examples and advice on getting started.

Load video
YouTube
Best Practices for Verification and Validation
MathWorks product marketing managers Nishaat Vasi and Jay Abraham present Model-Based Design techniques to verify and validate software designs and embedded code: requirements-driven development, model coverage testing, and static code analysis. These practices from high-integrity embedded systems are a useful reference for AI Verification and Validation.

Retrieved from "https://primo.ai/w/index.php?title=Best_Practices&oldid=50461"