Algorithm Administration

From PRIMO.ai
Jump to navigation Jump to search

YouTube ... Quora ...Google search ...Google News ...Bing News


Algorithm Administration is this wiki's umbrella term for everything that happens to a machine learning algorithm once you decide to rely on it. Training a model is one step. Around it sit many other jobs: governing the data the model learns from, versioning every artifact so results can be reproduced, choosing and tuning hyperparameters, automating parts of model and architecture search, shipping models through CI/CD pipelines, securing the model supply chain, monitoring behavior in production, and proving that a new version is actually better before it replaces the old one.

In industry, most of this work goes by the name MLOps (machine learning operations). Two newer variants extend it. LLMOps covers applications built on large language models, where the managed artifacts include prompts, retrieval indexes, and evaluation suites as well as weights. AgentOps covers AI agents that plan, call tools, and take actions, where you also have to administer what the system is allowed to do. AIOps is a related but different idea: using AI and machine learning to run IT operations.

The problem it addresses. A conventional program's behavior is determined by its code. A machine learning model's behavior is determined by code plus data plus configuration plus the training process, and the world it operates in keeps changing after deployment. A model can pass every test on launch day and quietly degrade a month later because customer behavior shifted. A team can lose the ability to rebuild its own model because nobody recorded which dataset snapshot and hyperparameters produced it. LLM-based systems add open-ended outputs, nondeterministic responses, per-token costs, and new attack surfaces such as prompt injection.

Origins and key contributors. The discipline grew out of DevOps and continuous delivery. A turning point was Google's 2015 NeurIPS paper Hidden Technical Debt in Machine Learning Systems by D. Sculley and colleagues, which argued that ML systems carry all the maintenance problems of ordinary code plus ML-specific ones such as boundary erosion, entanglement, hidden feedback loops, undeclared consumers, and data dependencies. Internal platforms such as Google's TFX and Uber's Michelangelo showed what end-to-end tooling could look like. Open-source projects such as Databricks' MLflow (2018), Google's Kubeflow (2018), and DVC (first released in 2017) made that tooling widely available. Academic work later formalized the field, for example Kreuzberger, Kühl, and Hirschl's definition of MLOps in IEEE Access. Andrew Ng's 2021 "data-centric AI" talk reframed the job around keeping a consistent, high-quality flow of data through every stage of a project. Gartner coined the term AIOps in 2016.

The central mechanism. Treat everything that determines model behavior as a versioned, testable, deployable artifact, then close the loop with monitoring, evaluation, and retraining. Data snapshots, feature definitions, training code, hyperparameters, model weights, prompts, retrieval indexes, and evaluation sets all get identifiers and lineage. Pipelines turn those artifacts into deployable models automatically, and production telemetry flows back to decide when to retrain, roll back, or redesign.

The paradigm shift. Algorithm administration has moved through several shifts: from "train once and ship" to a continuous lifecycle; from model-centric to data-centric improvement; from single trained models to composed systems (a foundation model plus prompts, retrieval, tools, and guardrails); and, most recently, from humans doing all of the ML engineering to AI agents that tune, search, and write ML pipelines under human supervision. That last shift means administration increasingly includes overseeing automated workers, not just automated pipelines.

How it differs from conventional software operations. DevOps tests logic and ships deterministic binaries. Algorithm administration also has to test data (schemas, distributions, leakage between training and test sets), test statistical performance (accuracy, calibration, and fairness across slices of the population), and accept that "correct" is often a probability rather than pass or fail. It adds a third automated pipeline, continuous training, next to continuous integration and continuous delivery.


A deployed model is not a finished artifact; it is a living system. Its data, code, configuration, prompts, and behavior must be versioned, tuned, tested, secured, and watched for as long as it runs, and increasingly the tools doing that work are AI systems that need administering too.


The Mechanism

The algorithm lifecycle runs as a loop. Here is the sequence most production teams follow:

  1. Data acquisition and governance. Raw data arrives from operational systems. Master Data Management resolves duplicate entities into trusted "golden records," governance rules decide who may use which data, and lineage tools record where every field came from.
  2. Validation and feature engineering. Pipelines check schemas, value ranges, and distributions, then compute features. A feature store keeps training-time and serving-time feature logic identical, which prevents training/serving skew.
  3. Versioning. Data snapshots, code commits, environment specifications, and configuration are pinned together so any model can be rebuilt and audited. See Versioning.
  4. Training and tuning. Training jobs run under an orchestrator, usually as DAG-based pipelines. Hyperparameter search, AutoML, or agentic search explores configurations while an experiment tracker logs every run.
  5. Evaluation and validation. Candidates are scored on held-out data, on critical data slices, and against the current production model. For LLM applications, this means curated evaluation datasets, rubric-based human review, and LLM-as-a-judge scoring.
  6. Registration and packaging. The approved model, its metadata, and its lineage go into a model registry. Packaging produces a container or a standard format (ONNX, MOJO, safetensors) and, increasingly, a cryptographic signature and an AI bill of materials.
  7. Deployment. Release strategies limit the blast radius of a bad model: shadow mode (score live traffic without acting on it), canary releases (a small share of traffic), blue/green switches, or A/B tests.
  8. Monitoring and observability. Operational metrics (latency, errors, throughput, cost) and model metrics (prediction drift, data drift, quality measured on delayed labels) are tracked continuously. LLM and agent systems emit traces of every model call, retrieval, and tool invocation.
  9. Feedback and retraining. Triggers (scheduled, drift-based, or performance-based) launch retraining or rollback. Human feedback and production failures become new evaluation cases, and the loop starts again.
Stage What gets administered Example tools Typical failure if neglected
Data and MDM Entities, schemas, lineage, access policies OpenLineage, Unity Catalog, MDM platforms, feature stores Duplicate or conflicting records poison training data
Versioning Data snapshots, code, configs, weights, prompts Git, DVC, lakeFS, MLflow Model Registry Models cannot be reproduced, debugged, or audited
Tuning and AutoML Hyperparameters, architectures, whole pipelines Optuna, Ray Tune, Katib, AutoGluon Wasted compute, or overfitting to the validation set
CI/CD/CT Tests, builds, retraining triggers GitHub Actions, GitLab CI, CML, Kubeflow Pipelines Untested models and prompt changes reach production
Serving Containers, runtimes, autoscaling, routing KServe, vLLM, llm-d, SageMaker AI, Vertex AI Latency spikes, runaway GPU or token cost
Monitoring Drift, quality, traces, cost Evidently, MLflow tracing, OpenTelemetry, Langfuse, Phoenix Silent degradation goes unnoticed for months
Security Signatures, SBOMs, access control, guardrails OpenSSF model signing, CycloneDX ML-BOM, OWASP checklists Tampered models, prompt injection, data leakage

Core components. Most platforms, open source or commercial, share the same building blocks: a metadata and lineage store; an experiment tracker; a model registry with stage transitions and approvals; a feature store; a pipeline orchestrator; a CI/CD system extended with data and model tests; a serving layer; an observability stack; and a policy layer that holds approvals, model cards, and audit logs.

Maturity levels. Google's widely used MLOps guidance describes three levels. At level 0, data scientists train and hand off models manually. At level 1, the training pipeline itself is automated, enabling continuous training on fresh data. At level 2, the pipeline's own code goes through automated CI/CD, so teams can change features, models, and pipeline steps quickly and safely. Most organizations run different models at different levels, and that is usually fine.

What changes for LLM and agent systems. When the model is a hosted foundation model, the team often does not train weights at all. The administered artifacts shift to prompts and prompt templates, retrieval indexes and embedding models, tool definitions and permissions, model routing rules, fine-tuning adapters, guardrail policies, and above all evaluation datasets. Monitoring shifts from aggregate accuracy toward per-request traces, token usage, and judged quality. Agents add one more layer: tool scopes, agent identities, spending limits, and human approval gates for irreversible actions.

Dependencies and conditions. The loop only works when certain conditions hold. Ground-truth labels must eventually arrive, or proxy metrics must be validated against them. Reproducibility requires pinned environments, recorded random seeds, and awareness that some GPU kernels are nondeterministic. Safe deployment depends on rollback paths and traffic splitting. And the economics depend on compute budgets, which for large models make naive grid search impossible.

Where uncertainty remains. Detecting harmful drift without labels is still partly guesswork, because distribution statistics flag change, not harm. Automated judges for open-ended outputs carry their own biases. Hosted LLM APIs can change behavior behind a stable model name. And there is no settled method for showing that an autonomous agent will stay within policy across every situation it may meet.

Why This Matters

Most of the cost comes after the first deployment. The Sculley et al. argument still holds: the model code is a small part of a production ML system, and the surrounding configuration, data collection, feature extraction, verification, serving, and monitoring infrastructure is where debt accumulates. Without deliberate administration, that debt compounds silently until a retrain breaks something nobody understands.

Reproducibility and accountability. Being able to say exactly which data, code, and settings produced a model is the foundation for debugging production incidents, answering auditors, and doing credible science. Regulators now expect it. The EU AI Act requires providers of high-risk systems to maintain risk management, data governance, technical documentation, logging, human oversight, and conformity assessment. The 2026 Digital Omnibus moved the application dates but did not change what the rules require, so lifecycle tooling remains the practical route to compliance.

Reliability and safety. Monitoring is the only way to know that a model still performs on today's data. For agents, administration is a safety control in its own right: tool permissions, approval gates, and audit trails are what keep an agent's mistakes recoverable.

Cost control. Hyperparameter search, retraining, and LLM inference consume expensive accelerators and tokens. Multi-fidelity tuning and hyperparameter transfer cut search costs sharply; serving optimizations such as cache-aware routing and batching cut inference costs; per-request cost tracking catches runaway agents before the monthly bill does.

Security. Model files can carry executable payloads, public model hubs can host impostors, and LLM applications can be hijacked through the content they read. Treating models, prompts, and tools as part of the software supply chain closes gaps that ordinary DevSecOps misses. See DevSecOps.

Engineering implications of automation. If AI agents keep getting better at ML engineering (see Recent Developments), the human role shifts toward specifying objectives, designing trustworthy evaluators, and reviewing results. Evaluator quality becomes the bottleneck: an automated search optimizes whatever the evaluator rewards, including its flaws.

Limitations. Full MLOps stacks impose real overhead; a small team with one batch model may need little more than Git, a tracked experiment log, and a scheduled evaluation job. Tool consolidation since 2024 means several products have changed owners, which affects roadmaps and lock-in risk. Many published productivity gains for AIOps and AI SRE tools come from vendors rather than independent studies.

What would change if current trends hold. If agentic ML engineering keeps improving at the pace seen on benchmarks, routine model building could become largely automated, and algorithm administration would mostly mean governing automated engineers: their budgets, evaluators, permissions, and audit trails. If tabular foundation models keep improving, per-dataset training and tuning may become the exception for small and mid-size tabular problems. Both remain open questions, and both make evaluation and governance more important, not less.

Recent Developments

This section covers notable developments from roughly late 2024 through October 2026. It labels peer-reviewed results, preprints, standards actions, and vendor announcements as such. Vendor-reported and self-reported benchmark numbers have generally not been independently replicated.

Consolidation of the MLOps tool market

Several standalone MLOps tools changed hands, a sign that lifecycle tooling is being absorbed into larger data and compute platforms:

What this means: the core open-source projects (MLflow, DVC, Kubeflow, KServe) remain healthy, but commercial ownership has shifted, so check roadmaps and licensing before standardizing on a tool.

MLflow 3 and GenAI-native lifecycle tooling

MLflow 3.0, announced at Databricks' Data + AI Summit in June 2025, redesigned the most widely used open-source MLOps tool around generative AI. It adds a LoggedModel entity that groups traces, metrics, and evaluation results for each version of an agent or model; production-scale tracing built on OpenTelemetry; a prompt registry; human feedback collection; and an evaluation harness that uses LLM judges. Autologging integrations capture prompts, responses, token usage, and cost across many GenAI libraries with one line of code. Community tutorials report the 3.x line reaching version 3.9 by early 2026. The design reflects a broader shift: for GenAI applications, the lifecycle is a loop of prompt design, model calls, evaluation, and feedback rather than a linear train, evaluate, deploy sequence.

Observability standards: OpenTelemetry GenAI semantic conventions

OpenTelemetry formed a GenAI special interest group in April 2024 to standardize attribute names for model calls, agent steps, tool calls, and evaluation results. Conventions for the Model Context Protocol (MCP) arrived in semantic conventions v1.39. In June 2026, the project moved its GenAI, provider-specific, and MCP conventions into a dedicated repository so they can version independently of the core specification. As of August 2026 the conventions were still marked as Development rather than stable; some commentary describing them as finalized overstates their status. The practical takeaway: instrument with OpenTelemetry now for portability across vendors, but expect attribute names to change.

Kubernetes-native training and serving

  • KServe joined the CNCF. KServe was accepted as a CNCF incubating project on September 29, 2025, and celebrated at KubeCon North America that November. It serves predictive models (scikit-learn, XGBoost) and generative models (Hugging Face, vLLM) through pluggable runtimes, and integrates with llm-d for disaggregated serving, prefix caching, and cache-aware scheduling. KServe Joins CNCF | The New Stack
  • Kubeflow Trainer v2. Kubeflow 1.10 (April 2025) introduced Kubeflow Trainer with a single TrainJob API in place of framework-specific resources such as PyTorchJob and TFJob. Kubeflow 1.11 ships Trainer v2.1 with Python-first workflows and preconfigured LLM fine-tuning runtimes, while Katib hyperparameter tuning stays compatible. A unified Kubeflow Python SDK shipped in November 2025 and passed one million PyPI downloads by August 2026. Kubeflow SDK Evolution | CNCF

AI agents that do ML engineering

What researchers tested. OpenAI's MLE-bench (ICLR 2025) packages 75 Kaggle competitions as an offline environment and asks AI agents to do the whole job: prepare data, train models, run experiments, and submit predictions. Submissions are graded against the real Kaggle private-leaderboard medal thresholds, which gives a direct human comparison.

What they found. In the original paper (October 2024), the best setup, OpenAI's o1-preview running inside the AIDE agent scaffold, earned at least a bronze medal in 16.9% of competitions. The authors also measured how results scale with more runtime, hardware, and attempts, and checked for contamination from pre-training data.

What happened next. Through 2025 and 2026, agent systems built on tree search, evolutionary search, and accumulated reusable "skills" pushed reported medal rates much higher. The open-source MLEvolve system reports a 65.3% medal rate on the full 75-task benchmark with a 12-hour budget (June 2026). On the 22-task "Lite" subset, several agents using frontier models with 24-hour budgets report roughly 75% to 80%.

What the results do and do not establish. Agents can now produce competitive solutions to well-specified, already-solved Kaggle-style problems. That does not show they can frame business problems, collect representative data, or operate models in messy production environments. Most recent numbers are self-reported by system authors and use differing budgets and subsets; research on agent search policies has documented a generalization gap between validation-selected and test-selected submissions; and contamination from public Kaggle solutions remains a live concern.

Evolutionary coding agents: AlphaEvolve

What was tested. Google DeepMind's AlphaEvolve (announced May 2025, white paper on arXiv in June 2025) pairs Gemini models with an evolutionary loop. The LLMs propose code changes, automated evaluators score each candidate, and a program database keeps a diverse population of strong programs for further mutation.

What was found. Applied to Google's own computing stack, AlphaEvolve found a more efficient data-center scheduling heuristic, simplified a circuit in a hardware accelerator design, and sped up training of the very LLM that powers it. Reported figures include a 23% speedup of a key matrix-multiplication kernel and about a 1% reduction in overall Gemini training time. It also found a way to multiply 4x4 complex-valued matrices with 48 scalar multiplications, improving on Strassen's 1969 approach for that setting. A November 2025 follow-up paper applied it to many open mathematical problems, at times matching or improving the best known constructions.

Limits. The method works only where a fast, trustworthy, machine-checkable evaluator exists. That is the core administration lesson: the evaluator, not the generator, defines what "better" means.

Tabular foundation models change the AutoML baseline

What was tested. TabPFN (Hollmann et al., Nature, January 2025) is a transformer pretrained on about 100 million synthetic datasets drawn from structural causal models. Instead of training on your data, it predicts in a single forward pass, using your training rows as context (in-context learning).

What was found. On datasets with up to 10,000 samples, the Nature paper reports TabPFN outperforming an ensemble of the strongest baselines tuned for four hours, in about 2.8 seconds for classification. TabPFN-2.5 (November 2025) extended the range to 50,000 rows and 2,000 features and reported matching AutoGluon 1.4's four-hour tuned ensemble on the TabArena benchmark. A May 2026 preprint on TabPFN-3 reports that its test-time-compute API variant beats AutoGluon 1.5's "extreme" preset while running about ten times faster.

How AutoML tools responded. Rather than compete, AutoML frameworks absorbed these models. AutoGluon 1.5 added RealTabPFN-2.5 and TabDPT as ensemble members, plus the Chronos-2 time series foundation model; AutoGluon 1.6 added TabPFN-3, TabICLv2, and other tabular foundation models.

Caveats. The newest claims come from the model developers, and independent benchmarking through TabArena is ongoing. Large datasets, strict latency budgets, and licensing (some checkpoints are non-commercial) still favor gradient-boosted trees in many deployments.

Hyperparameter transfer and modern HPO tooling

For large neural networks, the answer to "grid search is unaffordable" is increasingly hyperparameter transfer: tune a small proxy model and reuse its settings at full scale. Maximal Update Parametrization (μP), introduced in Yang et al.'s Tensor Programs V (2022), showed that hyperparameters transferred from a 40M-parameter proxy outperformed published numbers for the 6.7B-parameter GPT-3, with tuning cost about 7% of pretraining. Recent work extends and stress-tests the idea:

On the tooling side, Optuna v4.0 (September 2024) launched OptunaHub for sharing optimization algorithms. Version 4.4 extended its Gaussian-process sampler to multi-objective problems and added an MCP server so LLM agents can drive optimization studies, and version 4.6 added LLM integration to Optuna Dashboard. Preferred Networks reported Optuna in use by more than 16,000 software projects at the v4.0 release.

Evaluating open-ended systems: LLM-as-a-judge

LLM judges are now the default way to score GenAI outputs at scale, and tools such as MLflow 3, Opik, Weave, and Phoenix build them in. A widely cited survey (Gu et al., revised October 2025) treats reliability as the central open problem: judges show position bias, length bias, and preference for outputs resembling their own, and their scores shift when the judge model changes. The practical consensus: validate every judge against a human-labeled gold set, measure agreement with chance-corrected statistics, run position-swap and length-control checks, version judge prompts like code, and keep monitoring agreement after deployment.

Model supply chain security

  • Model signing. The OpenSSF Model Signing project released version 1.0 of its library and CLI in April 2025, then formalized the OpenSSF Model Signing (OMS) specification, built on Sigstore bundles, with Google, NVIDIA, and HiddenLayer among the contributors. NVIDIA has signed models it publishes to its NGC catalog since March 2025, and Kaggle serves signatures alongside hosted models. A signature proves the bytes are unchanged and that a particular key signed them; it does not prove the signer was authorized or that training was trustworthy, so provenance work continues. OpenSSF Model Signing for Safer AI Supply Chains | HiddenLayer
  • OWASP lists. The OWASP GenAI Security Project published the Top 10 for Agentic Applications on December 9, 2025, covering agent-specific risks such as goal hijacking, tool misuse, identity and privilege abuse, agentic supply chain vulnerabilities, and cascading multi-agent failures (ASI01 to ASI10). The 2026 edition of the Top 10 for LLM Applications followed in August 2026, keeping prompt injection first, moving excessive agency up to third, and cross-linking to the agentic list. OWASP's 2026 LLM Top 10 | Cloud Security Alliance
  • NIST. NIST released a preliminary draft of its Cyber AI Profile (NIST IR 8596) on December 16, 2025, applying Cybersecurity Framework 2.0 to securing AI systems, AI-enabled cyber defense, and AI-enabled attacks. It is being developed alongside SP 800-53 control overlays for securing AI systems (COSAiS), and a workshop summary (NIST IR 8607) appeared in August 2026. NIST has not released an "AI RMF 2.0"; updates to its AI risk guidance have come as profiles and companion resources. Cyber AI Profile | NIST NCCoE

Regulation and government acquisition

  • EU AI Act. Obligations for general-purpose AI models applied from August 2, 2025. The Digital Omnibus on AI (Regulation (EU) 2026/1744), in force since July 27, 2026, moved obligations for stand-alone high-risk systems listed in Annex III (hiring, credit scoring, education, biometrics, critical infrastructure) from August 2, 2026 to December 2, 2027, and for high-risk AI embedded in regulated products under Annex I to August 2, 2028. Article 50 transparency obligations still applied from August 2, 2026, with a grace period to December 2, 2026 for the generative AI marking requirement on systems already on the market. Digital AI Omnibus Delays Key Deadlines | Cooley
  • U.S. Department of Defense. The DoD CIO launched the Software Fast Track (SWFT) initiative with a 90-day sprint in spring 2025 to replace slow, document-driven Authority to Operate (ATO) reviews with faster, data-driven authorizations based on SBOMs, supply chain risk information, third-party assessments, and machine-readable security artifacts. Vendor analyses report that SWFT moved from pilot toward requirement, with self-attestation required in solicitations from November 10, 2025. This pushes contractors, including those delivering AI models in containers, toward pipelines that produce compliance evidence continuously. Arrington kicks off effort to eliminate RMF for DoD software | Federal News Network

From AIOps to AI SRE agents

Classic AIOps platforms use statistics and machine learning for anomaly detection, event correlation, and alert noise reduction. Since 2024, a newer category of "AI SRE" agents uses LLMs to investigate incidents: they query logs, metrics, traces, and deploy history, follow up on what they find, and propose ranked root-cause hypotheses, typically consuming what AIOps correlation produces. New Relic launched an SRE agent in February 2026 and cites its own report that users of its AI features resolved incidents 25% faster; industry coverage describes Gartner publishing its first market guide for AI SRE in January 2026. Independent evidence is still thin. Most products default to read-only access, and granting remediation authority is exactly where governance matters most.

Open questions

  • How well do agentic ML engineering results transfer from benchmarks to proprietary, poorly specified problems?
  • Can LLM judges become reliable enough to gate production releases without large human review budgets?
  • Will OpenTelemetry's GenAI conventions stabilize before teams lock into proprietary trace formats?
  • How should provenance extend beyond file signatures to cover training data, fine-tuning runs, and agent tool chains?
  • When an AI system improves the infrastructure that trains it, as AlphaEvolve did, what review and rollback processes are adequate?

Additional Viewing

Load video
YouTube

A Chat with Andrew on MLOps: From Model-centric to Data-centric AI (YouTube). Andrew Ng's hour-long DeepLearning.AI session from March 24, 2021 that launched the data-centric AI movement. Ng argues that for many practical applications, systematically improving data (labeling consistency, error analysis, feedback on drift) pays off more than changing the model, and that MLOps should ensure a consistent, high-quality flow of data through every project stage. It remains the clearest statement of why data administration sits at the center of this page.

Load video
YouTube

Andrej Karpathy: Software Is Changing (Again) (YouTube). A 39-minute keynote at Y Combinator's AI Startup School in June 2025. Karpathy describes "Software 3.0," in which natural-language prompts program LLMs, and argues for partially autonomous products built around fast human generate-and-verify loops rather than unsupervised agents. It explains why LLMOps and AgentOps administer prompts, autonomy levels, and human review, not only model weights.

Load video
YouTube

Google AlphaEvolve: Discovering New Science (YouTube). Machine Learning Street Talk's interview of more than an hour (May 2025) with Google DeepMind researchers Alexander Novikov and Matej Balog. They walk through AlphaEvolve's evolutionary loop, why automated evaluators are essential, compute costs, the human and AI collaboration loop, and the recursive angle of an agent improving the infrastructure that trains its own model. It pairs with the automated learning and AlphaEvolve material on this page.

Tools

This catalog preserves the page's original tool list. Notes marked 2026 status record ownership changes, renames, and maintenance status through October 2026. Several early AutoML libraries are no longer actively developed, so check repository activity before adopting any of them.

  • Google AutoML automatically build and deploy state-of-the-art machine learning models
  • SageMaker | Amazon
    • 2026 status: AWS renamed the core service Amazon SageMaker AI in December 2024 and introduced SageMaker Unified Studio for combined data, analytics, and AI work. Much of SageMaker Autopilot's AutoML functionality has moved into SageMaker Canvas.
  • MLOps | Microsoft ...model management, deployment, and monitoring with Azure
    • How can we improve Azure Data Catalog?
      • 2026 status: Azure Data Catalog has been superseded by Microsoft Purview, and the old feedback forum is no longer active.
    • AutoML
    • 2026 status: Azure Machine Learning remains Microsoft's classic MLOps service, while generative AI and agent development has moved to Microsoft's Foundry platform (formerly Azure AI Studio).
  • Ludwig - a Python toolbox from Uber that allows to train and test deep learning models
    • 2026 status: Ludwig is hosted by the Linux Foundation AI & Data and supports declarative fine-tuning of large language models.
  • TPOT a Python library that automatically creates and optimizes full machine learning pipelines using genetic programming. Not for NLP, strings need to be coded to numerics.
  • H2O Driverless AI for automated Visualization, feature engineering, model training, hyperparameter optimization, and explainability.
  • alteryx: Feature Labs, Featuretools
  • MLBox Fast reading and distributed data preprocessing/cleaning/formatting. Highly robust feature selection and leak detection. Accurate hyper-parameter optimization in high-dimensional space. State-of-the art predictive models for classification and regression (Deep Learning, Stacking, LightGBM,…). Prediction with models interpretation. Primarily Linux.
    • 2026 status: Development appears inactive; verify maintenance before adopting.
  • auto-sklearn algorithm selection and hyperparameter tuning. It leverages recent advantages in Bayesian optimization, meta-learning and ensemble construction.is a Bayesian hyperparameter optimization layer on top of scikit-learn. Not for large datasets.
    • Note: Auto-sklearn 2.0 added portfolio-based warm starting and automatic selection of its own validation policy, reducing the configuration burden on users.
  • Auto Keras is an open-source Python package for neural architecture search.
  • ATM -auto tune models - a multi-tenant, multi-data system for automated machine learning (model selection and tuning). ATM is an open source software library under the Human Data Interaction project (HDI) at MIT.
    • 2026 status: A historically important research system; development appears inactive.
  • Auto-WEKA is a Bayesian hyperparameter optimization layer on top of Weka. Weka is a collection of machine learning algorithms for data mining tasks. The algorithms can either be applied directly to a dataset or called from your own Java code. Weka contains tools for data pre-processing, classification, regression, clustering, association rules, and visualization.
  • TransmogrifAI - an AutoML library for building modular, reusable, strongly typed machine learning workflows. A Scala/SparkML library created by Salesforce for automated data cleansing, feature engineering, model selection, and hyperparameter optimization
  • RECIPE - a framework based on grammar-based genetic programming that builds customized scikit-learn classification pipelines.
  • AutoMLC Automated Multi-Label Classification. GA-Auto-MLC and Auto-MEKAGGP are freely-available methods that perform automated multi-label classification on the MEKA software.
  • Databricks MLflow an open source framework to manage the complete Machine Learning lifecycle using Managed MLflow as an integrated service with the Databricks Unified Analytics Platform... ...manage the ML lifecycle, including experimentation, reproducibility and deployment
    • 2026 status: MLflow 3 (June 2025) added GenAI tracing, a prompt registry, LLM-judge evaluation, and the LoggedModel entity. See Recent Developments.
  • SAS Viya automates the process of data cleansing, data transformations, feature engineering, algorithm matching, model training and ongoing governance.
  • Comet ML ...self-hosted and cloud-based meta machine learning platform allowing data scientists and teams to track, compare, explain and optimize experiments and models
    • 2026 status: Comet also maintains Opik, an open-source tool for LLM tracing and evaluation.
  • Domino Model Monitor (DMM) | Domino ...monitor the performance of all models across your entire organization
  • Weights and Biases ...experiment tracking, model optimization, and dataset versioning
    • 2026 status: Acquired by CoreWeave (completed May 2025). W&B's Weave toolkit covers LLM tracing and evaluation.
  • SigOpt ...optimization platform and API designed to unlock the potential of modeling pipelines. This fully agnostic software solution accelerates, amplifies, and scales the model development process
    • 2026 status: Acquired by Intel in 2020; confirm current availability before planning around it.
  • DVC ...Open-source Version Control System for Machine Learning Projects
    • 2026 status: Stewarded by lakeFS since November 2025; still open source.
  • ModelOp Center | ModelOp
  • Moogsoft and Red Hat Ansible Tower
    • 2026 status: Dell acquired Moogsoft in 2023. Ansible Tower's functionality continues as the automation controller in Red Hat Ansible Automation Platform.
  • DSS | Dataiku
  • Model Manager | SAS
  • Machine Learning Operations (MLOps) | DataRobot ...build highly accurate predictive models with full transparency
    • 2026 status: DataRobot acquired Algorithmia in 2021 to strengthen its model deployment and MLOps capabilities.
  • Metaflow, Netflix and AWS open source Python library
    • 2026 status: Outerbounds, founded by Metaflow's creators, offers a managed Metaflow platform.

Additional Tools and Platforms (2024 to 2026)

  • Optuna ...define-by-run hyperparameter optimization from Preferred Networks with TPE, CMA-ES, and Gaussian-process samplers, pruning, multi-objective search, OptunaHub, and an MCP server for agent-driven studies
  • Ray Tune ...distributed tuning with ASHA, Population Based Training, and integrations with most search libraries
  • Katib ...Kubernetes-native hyperparameter tuning and neural architecture search inside Kubeflow
  • AutoGluon ...AutoML for tabular, time series, and multimodal data; recent versions ensemble tabular foundation models such as TabPFN
  • TabPFN ...tabular foundation model that predicts in a single forward pass on small and mid-size tables
  • KServe ...CNCF incubating model serving platform for predictive and generative AI on Kubernetes
  • vLLM ...high-throughput LLM inference engine with paged attention and continuous batching
  • llm-d ...Kubernetes-native distributed LLM inference with cache-aware routing and disaggregated serving
  • lakeFS ...Git-like branches, commits, and rollbacks for data lakes on object storage
  • Feast ...open-source feature store with offline and online stores
  • Evidently ...open-source data drift, model quality, and LLM evaluation reports and monitors
  • Langfuse ...open-source LLM tracing, prompt management, and evaluation
  • Arize Phoenix ...open-source LLM and agent observability built on OpenTelemetry
  • OpenLineage ...open standard for data lineage events, hosted by LF AI & Data
  • Unity Catalog ...open-source catalog for data and AI assets, open-sourced by Databricks in 2024

Master Data Management (MDM)

YouTube search... ...Google search Feature Store / Data Lineage / Data Catalog

Master Data Management is the discipline of creating one trusted, governed version of an organization's core entities (customers, products, suppliers, locations, assets) across many source systems. It matters for machine learning because duplicate or conflicting entities become duplicate or conflicting training examples. A customer split across three IDs looks like three customers; the same entity can land in both training and test sets and inflate accuracy; and features computed per entity silently disagree between systems.

Core MDM operations. Matching finds records that refer to the same real-world entity. Merging applies survivorship rules that decide which source wins for each attribute. Stewardship workflows route ambiguous matches to people. Publishing distributes the resulting "golden records" to consumers, including ML pipelines.

How AI is changing MDM. ML-based matching replaced brittle hand-written rules years ago, and Tony Brownlee's video below covers early "cognitive" approaches to spotting data quality issues. More recently, embedding similarity and LLMs are used for entity resolution on messy text fields, for mapping schemas between systems, and for drafting stewardship decisions. These methods raise match recall but make decisions harder to explain, so many teams keep people in the loop for low-confidence merges and log the model version behind every automated merge.

Feature stores. A feature store is MDM's counterpart for model inputs: a governed registry of feature definitions with an offline store for training and a low-latency online store for serving, both computed from the same logic so models see identical features in each place. Open-source Feast and commercial platforms fill this role. Databricks' 2025 acquisition of Tecton shows real-time feature serving being repositioned as live "context" for AI agents.

Data lineage and catalogs. Lineage records how every dataset and feature was derived, which is what lets you answer "which models consumed this corrupted table?" OpenLineage defines an open standard for lineage events emitted by orchestrators and processing engines. Data catalogs such as Unity Catalog and Microsoft Purview tie lineage, access policies, and documentation together, and increasingly register models and AI functions alongside tables.

Data contracts. A growing practice is to define explicit, versioned contracts (schema, meaning, freshness, quality thresholds) between data producers and ML consumers, and to enforce them in CI so upstream changes cannot silently break features.

Load video
YouTube
How is AI changing the game for Master Data Management?
Tony Brownlee talks about the ability to inspect and find data quality issues as one of several ways cognitive computing technology is influencing master data management.

Load video
YouTube
Introducing Roxie. Data Management Meets Artificial Intelligence.
Introducing Roxie, Rubrik's Intelligent Personal Assistant. A hackathon project by Manjunath Chinni. Created in 10 hours with the power of Rubrik APIs.

Load video
YouTube
DAS Webinar: Master Data Management - Aligning Data, Process, and Governance
Getting MDM “right” requires a strategic mix of Data Architecture, business process, and Data Governance.

Load video
YouTube
How to manage Artificial Intelligence Data Collection [Enterprise AI Governance Data Management ]
Mind Data AI AI researcher Brian Ka Chan's AI ML DL introduction series. Collecting Data is an important step to the success of Artificial intelligence Program in the 4th industrial Revolution. In the current advancement of Artificial Intelligence technologies, machine learning has always been associated with AI, and in many cases, Machine Learning is considered equivalent of Artifical Intelligence. Machine learning is actually a subset of Artificial Intelligence, this discipline of machine learning relies on data to perform AI training, supervised or unsupervised. On average, 80% of the time that my team spent in AI or Data Sciences projects is about preparing data. Preparing data includes, but not limited to: Identify Data required, Identify the availability of data, and location of them, Profiling the data, Source the data, Integrating the data, Cleanse the data, and prepare the data for learning

Load video
YouTube
What is Data Governance?
Understand what problems a Data Governance program is intended to solve and why the Business Users must own it. Also learn some sample roles that each group might need to play.

Load video
YouTube
Top 10 Mistakes in Data Management
Come learn about the mistakes we most often see organizations make in managing their data. Also learn more about Intricity's Data Management Health Check which you can download here: https://www.intricity.com/intricity101/ To Talk with a Specialist go to: https://www.intricity.com/intricity101/ www.intricity.com

Versioning

YouTube search... ...Google search

Versioning answers one question: can you rebuild exactly this model, and explain exactly why it behaves the way it does? Answering it requires versioning far more than code:

Artifact Why it must be versioned Common approaches
Source code and pipeline definitions Training logic and preprocessing change behavior Git, pull-request review
Data and labels The same code on different data yields a different model DVC, lakeFS, Delta Lake or Apache Iceberg snapshots ("time travel")
Environment Library, driver, and hardware versions change numerical results Lock files, container images, pinned CUDA versions
Configuration and hyperparameters Small changes can swing results Config files in Git, experiment trackers
Model weights and adapters The deployable artifact itself Model registries (MLflow, SageMaker AI, Vertex AI), Hugging Face Hub
Prompts, retrieval indexes, tool schemas In LLM apps these define behavior as much as weights do Prompt registries, index snapshots, Git
Evaluation datasets and judges Scores are only comparable on the same yardstick Versioned evaluation sets, judge prompts under version control

Why Git alone falls short. Git is built for text diffs, not multi-gigabyte datasets and model files. DVC keeps small pointer files in Git and stores the content in object storage, so a single commit pins code and data together. lakeFS takes a different route, adding Git-like branches, commits, and merges directly on top of a data lake, which suits large shared datasets and lets teams test pipeline changes on an isolated branch of production data. Table formats such as Delta Lake and Apache Iceberg add snapshot "time travel" so a training job can record exactly which table version it read.

Ownership changes worth knowing. Hewlett Packard Enterprise acquired Pachyderm in 2023, and DVC moved from Iterative.ai to lakeFS in November 2025. Both projects remain available, but buyers should confirm current roadmaps and support options.

Load video
YouTube
How to manage model and data versions
Raj Ramesh Managing data versions and model versions is critical in deploying machine learning models. This is because if you want to re-create the models or go back to fix them, you will need both the data that went into training the model and as well as the model hyperparameters itself. In this video I explained that concept. Here's what I can do to help you. I speak on the topics of architecture and AI, help you integrate AI into your organization, educate your team on what AI can or cannot do, and make things simple enough that you can take action from your new knowledge. I work with your organization to understand the nuances and challenges that you face, and together we can understand, frame, analyze, and address challenges in a systematic way so you see improvement in your overall business, is aligned with your strategy, and most importantly, you and your organization can incrementally change to transform and thrive in the future. If any of this sounds like something you might need, please reach out to me at dr.raj.ramesh@topsigma.com, and we'll get back in touch within a day. Thanks for watching my videos and for subscribing. www.topsigma.com www.linkedin.com/in/rajramesh

Load video
YouTube
Version Control for Data Science Explained in 5 Minutes (No Code!)
In this code-free, five-minute explainer for complete beginners, we'll teach you about Data Version Control (DVC), a tool for adapting Git version control to machine learning projects.

- Why data science and machine learning badly need tools for versioning - Why Git version control alone will fall short - How DVC helps you use Git with big datasets and models - Cool features in DVC, like metrics, pipelines, and plots

Check out the DVC open source project on GitHub: https://github.com/iterative/dvc

Load video
YouTube
How to easily set up and version your Machine Learning pipelines, using Data Version Control (DVC) and Machine Learning Versioning (MLV)-tools | PyData Amsterdam 2019
Stephanie Bracaloni, Sarah Diot-Girard Have you ever heard about Machine Learning versioning solutions? Have you ever tried one of them? And what about automation? Come with us and learn how to easily build versionable pipelines! This tutorial explains through small exercises how to setup a project using DVC and MLV-tools. www.pydata.org

Load video
YouTube
Alessia Marcolini: Version Control for Data Science | PyData Berlin 2019
Track:PyData Are you versioning your Machine Learning project as you would do in a traditional software project? How are you keeping track of changes in your datasets? Recorded at the PyConDE & PyData Berlin 2019 conference. https://pycon.de

Load video
YouTube
Introduction to Pachyderm
Joey Zwicker A high-level introduction to the core concepts and features of Pachyderm as well as a quick demo. Learn more at: pachyderm.io github.com/pachyderm/pachyderm docs.pachyderm.io

Load video
YouTube
E05 Pioneering version control for data science with Pachyderm co-founder and CEO Joe Doliner
5 years ago, Joe Doliner and his co-founder Joey Zwicker decided to focus on the hard problems in data science, rather than building just another dashboard on top of the existing mess. It's been a long road, but it's really payed off. Last year, after an adventurous journey, they closed a $10m Series A led by Benchmark. In this episode, Erasmus Elsner is joined by Joe Doliner to explore what Pachyderm does and how it scaled from just an idea into a fast growing tech company. Listen to the podcast version https://apple.co/2W2g0nV

Model Versioning - ModelDB

ModelDB began as Manasi Vartak's research project at MIT CSAIL, one of the first open-source systems for tracking model versions, metadata, and experiments. Vartak's company Verta commercialized it as a model management and operations platform, which Cloudera acquired in June 2024. The ideas ModelDB pioneered (unique model identities, captured training context, and rollback) are now standard features of model registries such as MLflow's, which in MLflow 3 also groups traces and evaluations under a LoggedModel for each version. Teams considering the open-source ModelDB repository today should check its recent activity.

Load video
YouTube
Model Versioning Done Right: A ModelDB 2.0 Walkthrough
In a field that is rapidly evolving but lacks infrastructure to operationalize and govern models, ModelDB 2.0 provides the ability to version the full modeling process including the underlying data and training configurations, ensuring that teams can always go back and re-create a model, whether to remedy a production incident or to answer a regulatory query.

Load video
YouTube
Model Versioning Why, When, and How
Models are the new code. While machine learning models are increasingly being used to make critical product and business decisions, the process of developing and deploying ML models remain ad-hoc. In the “wild-west” of data science and ML tools, versioning, management, and deployment of models are massive hurdles in making ML efforts successful. As creators of ModelDB, an open-source model management solution developed at MIT CSAIL, we have helped manage and deploy a host of models ranging from cutting-edge deep learning models to traditional ML models in finance. In each of these applications, we have found that the key to enabling production ML is an often-overlooked but critical step: model versioning. Without a means to uniquely identify, reproduce, or rollback a model, production ML pipelines remain brittle and unreliable. In this webinar, we draw upon our experience with ModelDB and Verta to present best practices and tools for model versioning and how having a robust versioning solution (akin to Git for code) can streamlining DS/ML, enable rapid deployment, and ensure high quality of deployed ML models.

Hyperparameter

YouTube search... ...Google search

In machine learning, a hyperparameter is a parameter whose value is set before the learning process begins. By contrast, the values of other parameters are derived via training. Different model training algorithms require different hyperparameters, some simple algorithms (such as ordinary least squares regression) require none. Given these hyperparameters, the training algorithm learns the parameters from the data. Hyperparameter (machine learning) | Wikipedia

Machine learning algorithms train on data to find the best set of weights for each independent variable that affects the predicted value or class. The algorithms themselves have variables, called hyperparameters. They’re called hyperparameters, as opposed to parameters, because they control the operation of the algorithm rather than the weights being determined. The most important hyperparameter is often the learning rate, which determines the step size used when finding the next set of weights to try when optimizing. If the learning rate is too high, the gradient descent may quickly converge on a plateau or suboptimal point. If the learning rate is too low, the gradient descent may stall and never completely converge. Many other common hyperparameters depend on the algorithms used. Most algorithms have stopping parameters, such as the maximum number of epochs, or the maximum time to run, or the minimum improvement from epoch to epoch. Specific algorithms have hyperparameters that control the shape of their search. For example, a Random Forest (or) Random Decision Forest Classifier has hyperparameters for minimum samples per leaf, max depth, minimum samples at a split, minimum weight fraction for a leaf, and about 8 more. Machine learning algorithms explained | Martin Heller - InfoWorld

Refinements from later practice. The learning-rate description above is a useful simplification. A learning rate that is too high more often makes the loss oscillate or diverge outright, while one that is too low makes progress so slow that training ends in a poor region of the loss surface within the available budget. Modern training rarely uses a single fixed value: a short warmup followed by decay (cosine or linear schedules) is standard for Transformer models, and the best learning rate interacts with batch size, weight decay, optimizer choice, and the network's parametrization. The line between parameters and hyperparameters has also blurred. Hypernetworks generate weights for other networks, Population Based Training changes hyperparameters during a run, and in-context learners such as TabPFN have effectively learned their "training algorithm" in advance.

HPO1.png


Hyperparameter Tuning

Hyperparameters are the variables that govern the training process. Your model parameters are optimized (you could say "tuned") by the training process: you run data through the operations of the model, compare the resulting prediction with the actual value for each data instance, evaluate the accuracy, and adjust until you find the best combination to handle the problem. These algorithms automatically adjust (learn) their internal parameters based on data. However, there is a subset of parameters that is not learned and that have to be configured by an expert. Such parameters are often referred to as “hyperparameters” and they have a big impact ...For example, the tree depth in a decision tree model and the number of layers in an artificial Neural Network are typical hyperparameters. The performance of a model can drastically depend on the choice of its hyperparameters. Machine learning algorithms and the art of hyperparameter selection - A review of four optimization strategies | Mischa Lisovyi and Rosaria Silipo - TNW

There are four commonly used optimization strategies for hyperparameters:

  1. Bayesian optimization
  2. Grid search
  3. Random search
  4. Hill climbing

Bayesian optimization tends to be the most efficient. You would think that tuning as many hyperparameters as possible would give you the best answer. However, unless you are running on your own personal hardware, that could be very expensive. There are diminishing returns, in any case. With experience, you’ll discover which hyperparameters matter the most for your data and choice of algorithms. Machine learning algorithms explained | Martin Heller - InfoWorld

What later research added. The claim that Bayesian optimization tends to be most efficient needs context:

  • Bergstra and Bengio (JMLR 2012) showed that random search beats grid search whenever only a few hyperparameters really matter, which is common. Random search remains a strong, embarrassingly parallel baseline.
  • For deep learning, multi-fidelity methods often matter more than the choice of search model. They evaluate many configurations cheaply (a few epochs, a data subset, a smaller model) and promote only promising ones. Successive halving, Hyperband (Li et al.), and its asynchronous variant ASHA (Li et al.) are the workhorses; BOHB (Falkner, Klein, and Hutter) combines Hyperband with Bayesian optimization.
  • Population Based Training (Jaderberg et al., DeepMind) trains a population in parallel, copies weights from strong members, and perturbs their hyperparameters, so good schedules are discovered during training rather than fixed in advance.
  • Hyperparameter transfer (μP and its successors) avoids tuning at full scale altogether; see Recent Developments above.
  • Multi-objective tuning (accuracy versus latency, memory, cost, or fairness) produces a Pareto front of trade-offs rather than a single best configuration.
  • Running thousands of trials can overfit the validation set. Keep a final test set the tuner never sees, and consider nested cross-validation for small datasets.
Method How it works Strengths Weaknesses
Grid search Tries every combination on a fixed grid Simple and exhaustive in low dimensions Cost grows exponentially; wastes trials on unimportant dimensions
Random search Samples configurations at random Strong baseline; trivially parallel Does not learn from earlier trials
Bayesian optimization (Gaussian process, TPE) A surrogate model predicts promising configurations Sample-efficient when each trial is expensive Sequential bottleneck; struggles in very high dimensions
Hill climbing and local search Moves to better neighboring configurations Cheap and simple Gets stuck in local optima
Evolutionary and genetic methods Mutates and selects a population of configurations Handles mixed and conditional search spaces Needs many evaluations
Multi-fidelity (Hyperband, ASHA, BOHB) Stops weak trials early on cheap budgets Large savings for deep learning Low-budget rankings can mislead
Population Based Training Exploits and explores during training Learns schedules; uses parallel hardware well Requires many simultaneous workers
Hyperparameter transfer (μP) Tunes a small proxy, reuses settings at scale Near-zero tuning cost at full scale Requires a specific parametrization; known limits, such as changing mixture-of-experts granularity

Hyperparameter Optimization libraries:

Tuning:

  • Optimizer type
  • Learning rate (fixed or not)
  • Learning-rate schedule (warmup length, decay shape)
  • Epochs
  • Regularization rate (or not)
  • Type of Regularization - L1, L2, ElasticNet
  • Weight decay (decoupled, as in AdamW)
  • Dropout rate
  • Gradient clipping threshold
  • Search type for local minima
    • Gradient descent
    • Simulated annealing
    • Evolutionary
  • Decay rate (or not)
  • Momentum (fixed or not)
  • Nesterov Accelerated Gradient momentum (or not)
  • Batch size
  • Parametrization (standard or μP) and initialization scale
  • Fine-tuning settings: LoRA rank and scaling, number of trainable layers
  • Inference-time settings for LLM applications: sampling temperature, top-p, maximum tokens, retrieval top-k (tuned against evaluation sets rather than training loss)
  • Fitness measurement type
  • Stop criteria

Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube

Automated Learning

YouTube search... ...Google search

Several production machine-learning platforms now offer automatic hyperparameter tuning. Essentially, you tell the system what hyperparameters you want to vary, and possibly what metric you want to optimize, and the system sweeps those hyperparameters across as many runs as you allow. (Google Cloud hyperparameter tuning extracts the appropriate metric from the TensorFlow model, so you don’t have to specify it.)

Where automated learning stands in 2026. Automation now operates at four levels, each building on the one before:

  1. Hyperparameter tuning services (the paragraph above) remain standard in every major cloud ML platform and in open-source tools such as Optuna, Ray Tune, and Katib.
  2. AutoML systems search over whole pipelines: preprocessing, model family, hyperparameters, and ensembling. AutoGluon and auto-sklearn lead open-source tabular benchmarks, and since 2025 AutoGluon also ensembles pretrained tabular foundation models.
  3. Foundation models reduce the need for per-task training. An LLM can classify text zero-shot, and TabPFN predicts on a small table in one forward pass. Here "automated learning" becomes model selection and evaluation rather than training.
  4. Agentic ML engineering: LLM agents write and run complete ML pipelines, as measured by MLE-bench, and evolutionary coding agents such as AlphaEvolve discover new algorithms wherever a reliable automated evaluator exists.

The business-analyst vision quoted below (from 2018) has partly arrived by a different route than expected, through conversational assistants and agents rather than drag-and-drop AutoML. Piotr Plonski's warning that automatic machine learning can be "broken" still applies: automation can produce a model fast, but it cannot decide whether the problem is framed correctly, whether the data represents the population you care about, or whether the metric matches the business goal.

An emerging class of data science toolkit that is finally making machine learning accessible to business subject matter experts. We anticipate that these innovations will mark a new era in data-driven decision support, where business analysts will be able to access and deploy machine learning on their own to analyze hundreds and thousands of dimensions simultaneously. Business analysts at highly competitive organizations will shift from using visualization tools as their only means of analysis, to using them in concert with AML. Data visualization tools will also be used more frequently to communicate model results, and to build task-oriented user interfaces that enable stakeholders to make both operational and strategic decisions based on output of scoring engines. They will also continue to be a more effective means for analysts to perform inverse analysis when one is seeking to identify where relationships in the data do not exist. 'Five Essential Capabilities: Automated Machine Learning' | Gregory Bonnette

H2O Driverless AI automatically performs feature engineering and hyperparameter tuning, and claims to perform as well as Kaggle masters. AmazonML SageMaker supports hyperparameter optimization. Microsoft Azure Machine Learning AutoML automatically sweeps through features, algorithms, and hyperparameters for basic machine learning algorithms; a separate Azure Machine Learning hyperparameter tuning facility allows you to sweep specific hyperparameters for an existing experiment. Google Cloud AutoML implements automatic deep transfer learning (meaning that it starts from an existing Deep Neural Network (DNN) trained on other data) and neural architecture search (meaning that it finds the right combination of extra network layers) for language pair translation, natural language classification, and image classification. Review: Google Cloud AutoML is truly automated machine learning | Martin Heller

Status update. Several services named above have changed since these reviews were written. Google Cloud AutoML now lives inside Vertex AI; AWS renamed its core service SageMaker AI and moved most Autopilot AutoML features into SageMaker Canvas; Azure Machine Learning AutoML continues alongside Microsoft's generative AI tooling. Claims that a tool "performs as well as Kaggle masters" now have an independent yardstick for AI agents in MLE-bench, which grades complete solutions against real Kaggle leaderboards.

Load video
YouTube
Hyperparameter Tuning with Amazon SageMaker's Automatic Model Tuning - AWS Online Tech Talks
Learn how to use Automatic Model Tuning with Amazon SageMaker to get the best machine learning model for your dataset. Training machine models requires choosing seemingly arbitrary hyperparameters like learning rate and regularization to control the learning algorithm. Traditionally, finding the best values for the hyperparameters requires manual trial-and-error experimentation. Amazon SageMaker makes it easy to get the best possible outcomes for your machine learning models by providing an option to create hyperparameter tuning jobs. These jobs automatically search over ranges of hyperparameters to find the best values. Using sophisticated Bayesian optimization, a meta-model is built to accurately predict the quality of your trained model from the hyperparameters. Learning Objectives: 

- Understand what hyperparameters are and what they do for training machine learning models
- Learn how to use Automatic Model Tuning with Amazon SageMaker for creating hyperparameter tuning of your training jobs
- Strategies for choosing and iterating on tuning ranges of a hyperparameter tuning job with Amazon SageMaker

Load video
YouTube
Automatic Hyperparameter Optimization in Keras for the MediaEval 2018 Medico Multimedia Task
Rune Johan Borgli, Pål Halvorsen, Michael Riegler, Håkon Kvale Stensland, Automatic Hyperparameter Optimization in Keras for the MediaEval 2018 Medico Multimedia Task. Proc. of MediaEval 2018, 29-31 October 2018, Sophia Antipolis, France. Abstract: This paper details the approach to the MediaEval 2018 Medico Multimedia Task made by the Rune team. The decided upon approach uses a work-in-progress hyperparameter optimization system called Saga. Saga is a system for creating the best hyperparameter finding in Keras, a popular machine learning framework, using Bayesian optimization and transfer learning. In addition to optimizing the Keras classifier configuration, we try manipulating the dataset by adding extra images in a class lacking in images and splitting a commonly misclassified class into two classes. Presented by Rune Johan Borgli


AutoML

YouTube search... ...Google search


New cloud software suite of machine learning tools. It’s based on Google’s state-of-the-art research in image recognition called Neural Architecture Search (NAS). NAS is basically an algorithm that, given your specific dataset, searches for the most optimal neural network to perform a certain task on that dataset. AutoML is then a suite of machine learning tools that will allow one to easily train high-performance deep networks, without requiring the user to have any knowledge of deep learning or AI; all you need is labelled data! Google will use NAS to then find the best network for your specific dataset and task. AutoKeras: The Killer of Google’s AutoML | George Seif - KDnuggets

A qualification. The promise that AutoML needs no knowledge of deep learning, only labelled data, overstated the case. In practice AutoML shifted the required expertise rather than removing it: someone still has to frame the task, build representative training and test splits, prevent leakage, choose metrics, and judge whether results are good enough to deploy. Neural architecture search also proved less decisive than early coverage suggested. Li and Talwalkar (Random Search and Reproducibility for Neural Architecture Search) found random search with early stopping to be a competitive NAS baseline, and as Transformer architectures became standard, most practitioners moved from searching architectures to fine-tuning pretrained foundation models. Today's AutoML frontier is less about architectures and more about ensembling strong pretrained models, automating feature and pipeline construction, and, increasingly, letting LLM agents write and debug the whole pipeline.



Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube

Automatic Machine Learning (AML)

Automatic Machine Learning (AML) is used here as a near-synonym for AutoML, with emphasis on commercial end-to-end platforms that automate feature engineering directly from relational business data, such as dotData, DataRobot, and H2O Driverless AI. Tom Davenport's 2019 article captures the boom in these tools. The years since brought consolidation and a pivot: most AML vendors now also offer generative AI and agent tooling, and governance features (documentation, bias checks, monitoring) have become as important to buyers as raw automation.

Load video
YouTube
Load video
YouTube

Self-Learning

"Self-learning" covers several different mechanisms that are often lumped together:

  • Self-supervised learning creates training signals from the data itself, for example predicting the next token or a masked patch. It is how foundation models are pretrained.
  • Self-play has an agent learn by competing against copies of itself, as in game-playing systems.
  • Self-training (pseudo-labeling) uses a model's confident predictions on unlabeled data as new training labels.
  • Self-improving systems use a model's own outputs to improve the model or its infrastructure. AlphaEvolve's speedup of the training of its own underlying LLM is a concrete, limited example.

For administrators, the main caution is data provenance. Shumailov et al. (Nature, 2024) showed that models trained recursively on generated data can suffer "model collapse," progressively losing the tails of the original distribution. Track which training data is synthetic, keep fresh human-generated data in the mix, and evaluate on data the loop never produced.

Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube

DARTS: Differentiable Architecture Search

YouTube search... ...Google search

What happened after DARTS. DARTS made architecture search affordable on a single GPU, but follow-up work exposed weaknesses. The search often collapses toward parameter-free operations such as skip connections, and Zela et al. linked many failures to sharp minima in the architecture loss landscape, proposing early stopping and regularization as fixes. Other variants reduced memory use and corrected biases in how candidate operations compete. NAS research has since shifted toward hardware-aware search (architectures that meet latency or energy limits on a target device), cheap "zero-cost" proxies for ranking candidates, and searching over training recipes or code rather than cell structures, an idea that LLM-driven evolutionary systems such as AlphaEvolve generalize.

Load video
YouTube

AIOps/MLOps

Youtube search... ...Google search


Machine learning capabilities give IT operations teams contextual, actionable insights to make better decisions on the job. More importantly, AIOps is an approach that transforms how systems are automated, detecting important signals from vast amounts of data and relieving the operator from the headaches of managing according to tired, outdated runbooks or policies. In the AIOps future, the environment is continually improving. The administrator can get out of the impossible business of refactoring rules and policies that are immediately outdated in today’s modern IT environment. Now that we have AI and machine learning technologies embedded into IT operations systems, the game changes drastically. AI and machine learning-enhanced automation will bridge the gap between DevOps and IT Ops teams: helping the latter solve issues faster and more accurately to keep pace with business goals and user needs. How AIOps Helps IT Operators on the Job | Ciaran Byrne - Toolbox

A reality check on that vision. AIOps has delivered real gains in alert noise reduction, event correlation, and faster triage, but runbooks and policies have not disappeared. They have become the guardrails that define what automated remediation is allowed to do. The newest wave, LLM-based "AI SRE" agents that investigate incidents by querying logs, metrics, and deploy history, mostly runs with read-only access, because handing remediation authority to an automated system is a governance decision, not just a technical one.

Untangling the terms.

  • DevOps unifies software development and operations through automation, CI/CD, and shared ownership.
  • MLOps applies DevOps to machine learning and adds data versioning, experiment tracking, model registries, continuous training, and model monitoring.
  • AIOps uses AI and ML to operate IT systems (anomaly detection, event correlation, root-cause analysis). It is about running infrastructure with AI, not running AI.
  • LLMOps adapts MLOps to applications built on large language models: prompts, retrieval, evaluation of open-ended outputs, token costs, and guardrails.
  • AgentOps extends LLMOps to agents that take actions: tool permissions, agent identity, step and spending limits, action audit logs, and trajectory-level evaluation.
  • DataOps applies similar automation and quality discipline to data pipelines that feed all of the above.

MLOps maturity. Google's maturity model is a practical way to place each system. Level 0 means manual training and hand-off. Level 1 automates the training pipeline so models retrain continuously on fresh data. Level 2 adds CI/CD for the pipeline code itself. Higher is not always better; match the level to how often the model must change and how costly a failure would be.


Load video
YouTube
MLOps #28 ML Observability // Aparna Dhinakaran - Chief Product Officer at Arize AI
MLOps.community As more and more machine learning models are deployed into production, it is imperative we have better observability tools to monitor, troubleshoot, and explain their decisions. In this talk, Aparna Dhinakaran, Co-Founder, CPO of Arize AI (Berkeley-based startup focused on ML Observability), will discuss the state of the commonly seen ML Production Workflow and its challenges. She will focus on the lack of model observability, its impacts, and how Arize AI can help. This talk highlights common challenges seen in models deployed in production, including model drift, data quality issues, distribution changes, outliers, and bias. The talk will also cover best practices to address these challenges and where observability and explainability can help identify model issues before they impact the business. Aparna will be sharing a demo of how the Arize AI platform can help companies validate their models performance, provide real-time performance monitoring and alerts, and automate troubleshooting of slices of model performance with explainability. The talk will cover best practices in ML Observability and how companies can build more transparency and trust around their models. Aparna Dhinakaran is Chief Product Officer at Arize AI, a startup focused on ML Observability. She was previously an ML engineer at Uber, Apple, and Tubemogul (acquired by Adobe). During her time at Uber, she built a number of core ML Infrastructure platforms including Michaelangelo. She has a bachelors from Berkeley's Electrical Engineering and Computer Science program where she published research with Berkeley's AI Research group. She is on a leave of absence from the Computer Vision PhD program at Cornell University.

Load video
YouTube
DevOps for AI - Microsoft
DOES18 Las Vegas: Because the AI field is young compared to traditional software development, best practices and solutions around life cycle management for these AI systems have yet to solidify. This talk will discuss how we did this at Microsoft in different departments (one of them being Bing). DevOps for AI - Microsoft Gabrielle Davelaar, Data Platform Solution Architect/A.I., Microsoft Jordan Edwards, Senior Program Manager, Microsoft Gabrielle Davelaar is a Data Platform Solution Architect specialized in Artificial Intelligence solutions at Microsoft. She was originally trained as a computational neuroscientist. Currently she helps Microsoft’s top 15 Fortune 500 customers build trustworthy and scalable platforms able to create the next generation of A.I. applications. While helping customers with their digital A.I. transformation, she started working with engineering to tackle one key issue: A.I. maturity. The demand for this work is high, and Gabrielle is now working on bringing together the right people to create a full offering. Her aspirations are to be a technical leader in the healthcare digital transformation. Empowering people to find new treatments using A.I. while insuring privacy and taking data governance in consideration. Jordan Edwards is a Senior Program Manager on the Azure AI Platform team. He has worked on a number of highly performant, globally distributed systems across Bing, Cortana and Microsoft Advertising and is currently working on CI/CD experiences for the next generation of Azure Machine Learning. Jordan has been a key driver of dev-ops modernization in AI+R, including but not limited to: moving to Git, moving the organization at large to CI/CD, packaging and build language modernization, movement from monolithic services to microservice platforms and driving for a culture of friction free devOps and flexible engineering culture. His passion is to continue driving Microsoft towards a culture which enables our engineering talent to do and achieve more. DOES18 Las Vegas DOES 2018 US DevOps Enterprise Summit 2018 https://events.itrevolution.com/us/

Load video
YouTube
Productionizing Machine Learning with AI Platform Pipelines (Cloud AI Huddle)
In this virtual edition of AI Huddle, Yufeng introduces Cloud AI Platform Pipelines, a newly launched product from GCP. We'll first cover the basics of what it is and how to get it set up, and then dive deeper into how to operationalize your ML pipeline in this environment, using the Kubeflow Pipelines SDK. All the code we'll be using is already in the open, so boot up your GCP console and try it out yourself!

Load video
YouTube
How AIOps is transforming customer experience by breaking down DevOps silos at KPN
DevOps means dev fails fast and ops fails never. The challenge is that the volume of data exceeds human ability to analyze it and take action before user experience degrades. In this session, Dutch telecom KPN shares how AIOps from Broadcom is helping connect operational systems and increase automation across their toolchain to breakdown DevOps silos and provide faster feedback loops by preventing problems from getting into production.

Load video
YouTube
Getting to Sub-Second Incident Response with AIOps (FutureStack19)
Machine learning and artificial intelligence are revolutionizing the way operations teams work. In this talk, we’ll explore the incident management practices that can get you toward zero downtime, and talk about how New Relic can help you along that path. Speaker: Dor Sasson, Senior Product Manager AIOps, New Relic

Load video
YouTube
AWS re:Invent 2018: [REPEAT 1] AIOps: Steps Towards Autonomous Operations (DEV301-R1)
In this session, learn how to architect a predictive and preventative remediation solution for your applications and infrastructure resources. We show you how to collect performance and operational intelligence, understand and predict patterns using AI & ML, and fix issues. We show you how to do all this by using AWS native solutions: Amazon SageMaker and Amazon CloudWatch.

Load video
YouTube
Machine Learning & AIOps: Why IT Operations & Monitoring Teams Should Care
In this webinar, we break down what machine learning is, what it can do for your organization, and questions to ask when evaluating ML tools. For more information - please visit out our product pages: https://www.bigpanda.io/our-product/

Load video
YouTube
Anthony Bulk @ NTT DATA, "AI/ML Democratization"
North Technology People would like to welcome you to the CAMDEA Digital Forum for Tuesday 15th September Presenter: Anthony Bulk Job title: Director AI & Data Presentation: AI/ML Democratization

Load video
YouTube
AIOps: The next 5 years
A Conversation with Will Cappelli, CTO of EMEA, Moogsoft & Rich Lane, Senior Analyst, Forrester In this webinar, the speakers discuss the role that AIOps can and will play in the enterprise of the future, how the scope of AIOps platforms will expand, and what new functionality may be deployed. Watch this webinar to learn how AIOps will enable and support the digitalization of key business processes, and what new AI technologies and algorithms are likely to have the most impact on the continuing evolution of AIOps. For more information, visit our website www.moogsoft.com

Load video
YouTube
Building an MLOps Toolchain The Fundamentals
Artificial intelligence and machine learning are the latest “must-have” technologies in helping organizations realize better business outcomes. However, most organizations don’t have a structured process for rolling out AI-infused applications. Data scientists create AI models in isolation from IT, which then needs to insert those models into applications, and ensure their security, to deliver any business value. In this ebook/webinar, we examine the best way to set up an MLOps process to ensure successful delivery of AI-infused applications.

Load video
YouTube
Webinar: MLOps automation with Git Based CI/CD for ML
Deploying AI/ML based applications is far from trivial. On top of the traditional DevOps challenges, you need to foster collaboration between multidisciplinary teams (data-scientists, data/ML engineers, software developers and DevOps), handle model and experiment versioning, data versioning, etc. Most ML/AI deployments involve significant manual work, but this is changing with the introduction of new frameworks that leverage cloud-native paradigms, Git and Kubernetes to automate the process of ML/AI-based application deployment. In this session we will explain how ML Pipelines work, the main challenges and the different steps involved in producing models and data products (data gathering, preparation, training/AutoML, validation, model deployment, drift monitoring and so on). We will demonstrate how the development and deployment process can be greatly simplified and automated. We’ll show how you can: a. maximize the efficiency and collaboration between the various teams, b. harness Git review processes to evaluate models, and c. abstract away the complexity of Kubernetes and DevOps. We will demo how to enable continuous delivery of machine learning to production using Git, CI frameworks (e.g. GitHub Actions) with hosted Kubernetes, Kubeflow, MLOps orchestration tools (MLRun), and Serverless functions (Nuclio) using real-world application examples. Presenter: Yaron Haviv, Co-Founder and CTO @Iguazio

Load video
YouTube
How RealPage Leveraged Full Stack Visibility and Integrated AIOps for SaaS Innovation and Customer S
Development and operation teams continue to struggle with having a unified view of their applications and infrastructure. In this webinar, you'll learn how RealPage, the industry leader in SaaS-based Property Management Solutions, leverages an integrated AppDynamics and Virtana AIOps solution to deliver a superior customer experience by managing the performance of its applications as well as its infrastructure. RealPage must ensure its applications and infrastructure are always available and continuously performing, while constantly innovating to deliver new capabilities to sustain their market leadership. The combination of their Agile development process and highly virtualized infrastructure environment only adds to the complexity of managing both. To meet this challenge, RealPage is leveraging the visibility and AIOps capabilities delivered by the integrated solutions from AppDynamics and Virtana.

Load video
YouTube
Machine Learning on Kubernetes with Kubeflow
Google Cloud Platform Join Fei and Ivan as they talk to us about the benefits of running your TensorFlow models in Kubernetes using Kubeflow. Working on a cool project and want to get in contact with us? Fill out Don't forget to subscribe to the channel! → https://goo.gl/UzeAiN this form → https://take5.page.link/csf1 Watch more Take5 episodes here → https://bit.ly/2MgTllk

Load video
YouTube
PipelineAI End-to-End TensorFlow Model Training + Deploying + Monitoring + Predicting (Demo)
100% open source and reproduce-able on your laptop through Docker - or in production through Kubernetes! Details at https://pipeline.io and https://github.com/fluxcapacitor/pipeline End-to-end pipeline demo: Train and deploy a TensorFlow model from research to live production. Includes full metrics and insight into the offline training and online predicting phases.

Load video
YouTube
Deep Learning Pipelines: Enabling AI in Production
Deep learning has shown tremendous successes, yet it often requires a lot of effort to leverage its power. Existing deep learning frameworks require writing a lot of code to run a model, let alone in a distributed manner. Deep Learning Pipelines is an Apache Spark Package library that makes practical deep learning simple based on the Spark MLlib Pipelines API. Leveraging Spark, Deep Learning Pipelines scales out many compute-intensive deep learning tasks. In this talk, we discuss the philosophy behind Deep Learning Pipelines, as well as the main tools it provides, how they fit into the deep learning ecosystem, and how they demonstrate Spark's role in deep learning. About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business. Website: https://databricks.com

Load video
YouTube
PipelineAI: High Performance Distributed TensorFlow AI + GPU + Model Optimizing Predictions
We will each build an end-to-end, continuous TensorFlow AI model training and deployment pipeline on our own GPU-based cloud instance. At the end, we will combine our cloud instances to create the LARGEST Distributed TensorFlow AI Training and Serving Cluster in the WORLD! Pre-requisites Just a modern browser and an internet connection. We'll provide the rest! Agenda Spark ML TensorFlow AI Storing and Serving Models with HDFS Trade-offs of CPU vs. *GPU, Scale Up vs. Scale Out CUDA + cuDNN GPU Development Overview TensorFlow Model Checkpointing, Saving, Exporting, and Importing Distributed TensorFlow AI Model Training (Distributed TensorFlow) TensorFlow's Accelerated Linear Algebra Framework (XLA) TensorFlow's Just-in-Time (JIT) Compiler, Ahead of Time (AOT) Compiler Centralized Logging and Visualizing of Distributed TensorFlow Training (Tensorboard) Distributed TensorFlow AI Model Serving/Predicting (TensorFlow Serving) Centralized Logging and Metrics Collection (Prometheus, Grafana) Continuous TensorFlow AI Model Deployment (TensorFlow, Airflow) Hybrid Cross-Cloud and On-Premise Deployments (Kubernetes) High-Performance and Fault-Tolerant Micro-services (NetflixOSS) https://pipeline.ai

Load video
YouTube
Bringing Your Data Pipeline into The Machine Learning Era - Chris Gaun & Jörg Schad, Mesosphere
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (https://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (https://bit.ly/kccncchina18). Bringing Your Data Pipeline into The Machine Learning Era - Chris Gaun, Mesosphere (Intermediate Skill Level) Kubeflow is a new tool that makes it easy to run distributed machine learning solutions (e.g. Tensorflow) on Kubernetes. However, much of the data that can feed machine learning algorithms is already in existing distributed data stores. This presentation shows how to connect existing distributed data services running on Apache Mesos to Tensorflow on Kubernetes using the Kubeflow tool. Chris Gaun will show you how this existing data can now leverage machine learning, such as Tensorflow, on Kubernetes using the Kubeflow tool. These lessons can be extrapolated to any local distributed data. Chris Gaun is a CNCF ambassador and product marketing manager at Mesosphere. He has presented at Kubecon in 2016 and has put on over 40 free Kubernetes workshops across US and EU in 2017. About Jörg He is a technical lead at Mesosphere https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

Load video
YouTube
Deep Dive into Deep Learning Pipelines - Sue Ann Hong & Tim Hunter
"Deep learning has shown tremendous successes, yet it often requires a lot of effort to leverage its power. Existing deep learning frameworks require writing a lot of code to run a model, let alone in a distributed manner. Deep Learning Pipelines is a Spark Package library that makes practical deep learning simple based on the Spark MLlib Pipelines API. Leveraging Spark, Deep Learning Pipelines scales out many compute-intensive deep learning tasks. In this talk we dive into - the various use cases of Deep Learning Pipelines such as prediction at massive scale, transfer learning, and hyperparameter tuning, many of which can be done in just a few lines of code. - how to work with complex data such as images in Spark and Deep Learning Pipelines. - how to deploy deep learning models through familiar Spark APIs such as MLlib and Spark SQL to empower everyone from machine learning practitioners to business analysts. Finally, we discuss integration with popular deep learning frameworks. About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business. Connect with us: Website: https://databricks.com

Load video
YouTube
How to Build Machine Learning Pipelines in a Breeze with Docker
Docker Let's review and see in action a few projects based on docker containers that could help you prototype ML based projects (detect faces, nudity, evaluate sentiment, building a smart bot) within a few hours. Let´s see in practice where containers could help in this type of projects.

Load video
YouTube
End to End Streaming ML Recommendation Pipeline Spark 2 0, Kafka, TensorFlow Workshop
End to End Streaming ML Recommendation Pipeline Spark 2 0, Kafka, TensorFlow Workshop Presented at Bangalore Apache Spark Meetup by Chris Fregly on 10/12/2016. Connect with Chris Fregly at https://www.linkedin.com/in/cfregly https://twitter.com/cfregly https://www.slideshare.net/cfregly

Load video
YouTube
Machine Learning with caret: Building a pipeline
Building a Machine Learning model with RandomForest and caret in R.

Load video
YouTube
Let’s Write a Pipeline - Machine Learning Recipes #4
Google Developer In this episode, we’ll write a basic pipeline for supervised learning with just 12 lines of code. Along the way, we'll talk about training and testing data. Then, we’ll work on our intuition for what it means to “learn” from data. Check out TensorFlow Playground: https://goo.gl/cv7Dq5

Load video
YouTube
Kevin Goetsch | Deploying Machine Learning using sklearn pipelines
PyData Chicago 2016 Sklearn pipeline objects provide an framework that simplifies the lifecycle of data science models. This talk will cover the how and why of encoding feature engineering, estimators, and model ensembles in a single deployable object.

Load video
YouTube
Building online churn prediction ML model using XGBoost, Spark, Featuretools, Python and GCP
Mariusz Jacyno This video shows step by step how to build, evaluate and deploy churn prediction model using Python, Spark, automated feature engineering (Featuretools), extreme gradient boosting algorithm (XGBoost) and Google ML service.

Load video
YouTube
MLOps #34 Owned By Statistics: How Kubeflow & MLOps Can Help Secure ML Workloads // David Aronchick
While machine learning is spreading like wildfire, very little attention has been paid to the ways that it can go wrong when moving from development to production. Even when models work perfectly, they can be attacked and/or degrade quickly if the data changes. Having a well understood MLOps process is necessary for ML security! Using Kubeflow, we demonstrated how to the common ways machine learning workflows go wrong, and how to mitigate them using MLOps pipelines to provide reproducibility, validation, versioning/tracking, and safe/compliant deployment. We also talked about the direction for MLOps as an industry, and how we can use it to move faster, with less risk, than ever before. David leads Open Source Machine Learning Strategy at Azure. This means he spends most of his time helping humans to convince machines to be smarter. He is only moderately successful at this. Previously, he led product management for Kubernetes on behalf of Google, launched Google Kubernetes Engine, and co-founded the Kubeflow project. He has also worked at Microsoft, Amazon and Chef and co-founded three startups. When not spending too much time in service of electrons, he can be found on a mountain (on skis), traveling the world (via restaurants) or participating in kid activities, of which there are a lot more than he remembers than when he was that age.

Load video
YouTube
An introduction to MLOps on Google Cloud
The enterprise machine learning life cycle is expanding as firms increasingly look to automate their production ML systems. MLOps is an ML engineering culture and practice that aims at unifying ML system development and ML system operation enabling shorter development cycles, increased deployment velocity, and more dependable releases in close alignment with business objectives. Learn how to construct your systems to standardize and manage the life cycle of machine learning in production with MLOps on Google Cloud. Speaker: Nate Keating Watch more: Google Cloud Next ’20: OnAir → https://goo.gle/next2020

Load video
YouTube
MLOps: Planning for the new normal
Join us for a conversation with Diego Oppenheimer, CEO of Algorithmia, and Kahini Shah, Investor at Gradient Ventures, as they discuss the impacts that the global pandemic is having on AI/ML.

Load video
YouTube
Artificial Intelligence (AI) Operations in ServiceNow | Share The Wealth
Mike Desmond of GlideFast Consulting explains AI Operations in ServiceNow in this Share the Wealth session.

Load video
YouTube
GitOps for MLOps - David Aronchick
GitOps Days 2020 Day 1, Session 8 (May 20) While DevOps and GitOps have made huge traction in recent years, many customers struggle to apply these practices to ML workloads. This talk will focus on some best practices to bring GitOps to MLOps for production-grade applications and possibilities for the future. David Aronchick, Head of Open Source ML Strategy, Microsoft David Aronchick leads open source machine learning strategy at Azure. He spends most of his time helping humans convince machines to be smarter. (He’s only moderately successful at this.) Previously, he led product management for Kubernetes on behalf of Google, launched Google Kubernetes Engine, and cofounded the Kubeflow project. He’s also worked at Microsoft, Amazon, and Chef and cofounded three startups. When not spending too much time in service of electrons, he can be found on a mountain (on skis), traveling the world (via restaurants), or participating in kid activities, of which there are a lot more than he remembers when he was that age.

Load video
YouTube
MLOps Demo: A Day in the Life
DataRobot MLOps is the way to manage all your production models. In this demo, we will walk through a typical day-in-a-life of a financial services company running a Center of Excellence for delivering ML in Production. We'll see the types of administrative activities our MLOps engineer George performs, and understand how he works with Mary, our lead Data Scientist. George and Mary rely on each other to effectively build, deploy, and manage models for their company over their full lifecycle. Visit the MLOps product page to find out more: datarobot.com/platform/mlops/

Load video
YouTube
MLOPs topics in the cloud
Pragmatic AI Labs A lecture on emerging topics on MLOps in the cloud including Azure, GCP and Sagemaker.

Load video
YouTube
Commit Virtual 2020: MLOps DevOps for Machine Learning
GitLab Speaker: Monmayuri Ray The practice of Devops - developing software and operationalizing the development cycle has been evolving for over a decade. Now, a new addition has joined this holistic development cycle: Machine Predictions. The emerging art and science of machine learning algorithms integrated into current operational systems is opening new possibilities for engineers, scientists, and architects in the tech world. This presentation will take the audience on a journey in understanding the fundamentals of orchestrating machine predictions using MLOps in this ever-changing, agile world of software development. You’ll learn how to excel at the craft of DevOps for Machine Learning (ML). Monmayuri will unpack the theoretical constructs and show they apply to real-world scenarios. Get in touch with Sales: https://bit.ly/2IygR7z

Load video
YouTube
AIDevFest20: Machine Learning Design Patterns for MLOps
Speaker: Valliappa Lakshmanan (LAK), Google Design patterns are formalized best practices to solve common problems when designing a software system. As machine learning moves from being a research discipline to a software one, it is useful to catalog tried-and-proven methods to help engineers tackle frequently occurring problems that crop up during the ML process. In this talk, I will cover five patterns (Workflow Pipelines, Transform, Multimodal Input, Feature Store, Cascade) that are useful in the context of adding flexibility, resilience and reproducibility to ML in production. For data scientists and ML engineers, these patterns provide a way to apply hard-won knowledge from hundreds of ML experts to your own projects. Anyone designing infrastructure for machine learning will have to be able to provide easy ways for the data engineers, data scientists, and ML engineers to implement these, and other, design patterns. Website: https://gdgdevfest20.xnextcon.com

Load video
YouTube
Python Lunch Break presents MLOps
Deploy your Chatbot using CI/CD by William Arias Leverage DevOps practices in your next Chatbot deployments, create pipelines for training and testing your Chatbot before automating the deployment to staging or production. NLP developers can focus on the Chatbot development and worry less about infrastructure and environments configuration Bio: Born and raised in Colombia, studied Electronic Engineering and specialized in Digital Electronics, former exchange student in Czech Technical University in the faculty of Data Science. I started my career working in Intel as Field Engineer, then moved to Oracle in Bogota. After 5 years in Oracle as Solution Architect I decided to get deeper in my knowledge of Machine Learning and Databases, relocated to Prague in Czech Republic, where I studied Data Science for a while from the academia perspective, later jumped to Solution Architect in CA technologies, later moved on to Developer of Automations and NLP in Adecco. Currently I work at Gitlab as Technical Marketing Engineer, teaching and evangelizing developers about DevOps

Large Language Model Operations (LLMOps)


Large Language Model Operations (LLMOps) is a set of practices and tools for the operational management of large language models (LLMs) in production environments. LLMOps encompasses the entire lifecycle of LLMs, from development and deployment to monitoring and maintenance. LLMOps is important because LLMs are complex and resource-intensive models that require careful management to ensure that they are performing as expected and meeting the needs of their users. LLMOps tools and practices can help to streamline and automate the LLM lifecycle, making it easier to deploy and manage LLMs in production.

Here are some of the key components of LLMOps:

  • Model development: LLMOps tools and practices can help to streamline the process of developing new LLMs, from data collection and preparation to model training and evaluation.
  • Model deployment: LLMOps tools and practices can help to deploy LLMs to production environments in a safe and efficient manner.
  • Model monitoring: LLMOps tools and practices can help to monitor the performance of LLMs in production and identify any problems early on.
  • Model maintenance: LLMOps tools and practices can help to maintain LLMs in production, including fine-tuning models to improve their performance and updating models with new data.


LLMOps is still a relatively new field, but it is rapidly growing as more and more organizations adopt LLMs. There are a number of LLMOps tools and platforms available, both open source and proprietary. Here are some of the benefits of using LLMOps:

  • Improved efficiency and productivity: LLMOps tools and practices can help to automate and streamline the LLM lifecycle, freeing up data scientists and engineers to focus on more creative and strategic tasks.
  • Reduced costs: LLMOps can help to reduce the costs of developing and managing LLMs by improving efficiency and reducing the risk of errors.
  • Increased reliability and performance: LLMOps can help to ensure that LLMs are performing reliably and meeting the needs of their users by providing tools and practices for monitoring and maintaining LLMs in production.

A note on cost. LLMOps reduces waste, but LLM applications often cost more per request to run than classic models, and agent workflows multiply calls. The more reliable benefit is control: knowing what each feature costs, catching runaway loops, and choosing the cheapest model that meets a quality bar.

What LLMOps administers in practice (2026). Most teams no longer train their own LLMs. They build on hosted or open-weight foundation models, so day-to-day LLMOps centers on:

  • Prompt and configuration versioning. Prompt templates, system instructions, model names, and sampling settings live in a registry or in Git, and every change triggers evaluation.
  • Retrieval pipelines. For retrieval-augmented generation (RAG): embedding model versions, chunking settings, index snapshots, and freshness of source documents.
  • Evaluation. Offline evaluation sets, LLM judges calibrated against human labels, and regression suites run in CI on every prompt, model, or index change.
  • Tracing and observability. Per-request traces of model calls, retrieved context, tool calls, tokens, latency, and cost, ideally using OpenTelemetry-compatible instrumentation.
  • Guardrails. Input and output filters, personal data redaction, prompt-injection defenses, and policy checks.
  • Gateways and routing. Choosing among models by cost, latency, and quality, with fallbacks when a provider fails.
  • Serving and capacity. For self-hosted models: engines such as vLLM, KServe with llm-d for cache-aware routing, quantization, batching, and autoscaling.
  • Fine-tuning and adapters. LoRA and similar adapters get their own versioning, evaluation, and rollback.
  • Provider change management. Pin model versions where providers allow it, and re-run evaluations when a provider updates or retires a model.

AgentOps. Agents that plan and act add another administrative layer: least-privilege tool permissions, a distinct identity for each agent, step and spending limits, human approval gates for irreversible actions, complete action audit logs, and evaluation of whole trajectories rather than only final answers. The OWASP Top 10 for Agentic Applications (December 2025) is a useful checklist of what can go wrong, from goal hijacking to cascading failures across multiple agents.

How is LLMOps different from AIOps/MLOps?


LLMOps is a subset of MLOps that focuses specifically on the operational management of large language models (LLMs). MLOps is a set of practices and tools for managing the entire lifecycle of machine learning models, from development to deployment to production. AIOps is a broader field that focuses on using AI and machine learning to improve IT operations and management. Here is a table that summarizes the key differences between LLMOps, AIOps, and MLOps:

Updated perspective. Many practitioners now describe LLMOps as an extension of MLOps rather than a strict subset. When a team consumes a hosted model through an API, it may never train or deploy weights at all; the managed artifacts become prompts, retrieval indexes, tool configurations, and evaluation suites. Chip Huyen's AI Engineering (2025) describes this as starting from a product idea and readily available models rather than from data collection and model training. The table below keeps the original three columns and adds AgentOps.

Characteristic LLMOps AIOps MLOps AgentOps
Focus Operational management of LLMs Use of AI and ML to improve IT operations and management Operational management of machine learning models Operational management of AI agents that plan, call tools, and act
Use cases Language-related tasks, such as chatbots, language translation, sentiment analysis, and content generation IT operations tasks, such as incident management, capacity planning, and performance monitoring Machine learning tasks, such as fraud detection, product recommendation, and predictive maintenance Multi-step tasks such as customer service resolution, coding, research, and back-office workflows
Team composition NLP specialists, linguists, data scientists with language modeling expertise, and software engineers experienced in building language-related applications; increasingly, software engineers working as "AI engineers" on hosted models Data scientists, analytics professionals, data engineers, domain experts, and IT professionals Data scientists, machine learning engineers, software engineers, and DevOps engineers AI engineers, platform engineers, security and identity specialists, and domain owners who approve agent permissions
Primary artifacts Prompts, retrieval indexes, adapters, evaluation sets, guardrail policies Telemetry pipelines, correlation models, runbooks Datasets, features, trained models, pipelines Tool definitions, permissions, plans, memory stores, action logs
Key metrics Judged quality, groundedness, latency, tokens, cost per request Mean time to detect and resolve, alert noise, availability Accuracy, drift, latency, business KPIs Task success rate, steps per task, tool error rate, policy violations, cost per task
Characteristic risks Hallucination, prompt injection, data leakage, provider changes Missed incidents, unsafe automated remediation Drift, training/serving skew, bias Goal hijacking, tool misuse, privilege abuse, cascading failures


Here are some examples of how LLMOps is different from AIOps/MLOps:

  • Model development: LLMOps tools and practices are specifically designed for developing and training LLMs. For example, LLMOps tools may help to automate the process of collecting and cleaning text data, or they may provide specialized libraries for training LLMs.
  • Model deployment: LLMOps tools and practices can help to deploy LLMs to production environments in a way that is optimized for performance and reliability. For example, LLMOps tools may help to scale LLMs to handle large volumes of traffic, or they may provide specialized infrastructure for hosting LLMs.
  • Model monitoring and maintenance: LLMOps tools and practices can help to monitor the performance of LLMs in production and identify any problems early on. For example, LLMOps tools may track the accuracy of LLMs on real-world data, or they may detect potential biases in LLMs.
  • Evaluation: Classic MLOps compares predictions with known labels. LLMOps usually has no single correct answer, so it relies on rubrics, human review, and LLM judges, which must themselves be validated.
  • Change management: In MLOps, the team controls when a model changes. In LLMOps on hosted models, the provider may change or retire a model, so pinned versions and regression evaluations become essential.

Continuous Machine Learning (CML)

CML runs model training and evaluation inside standard CI systems such as GitHub Actions and GitLab CI, then posts metrics, tables, and plots back to the pull request, so model changes get reviewed the same way code changes do. CML was developed by Iterative alongside DVC; after DVC moved to lakeFS in November 2025, check the CML repository for its current maintenance status. The underlying pattern has become standard regardless of tool: run an evaluation suite on every change to code, data, prompts, or model version, compare against the main branch, and block merges that regress key metrics.

Load video
YouTube
MLOps Tutorial #1: Intro to Continuous Integration for ML
DVCorg Learn how to use one of the most powerful ideas from the DevOps revolution, continuous integration, in your data science and machine learning projects. This hands-on tutorial shows you how to create an automatic model training & testing setup using GitHub Actions and Continuous Machine Learning (CML), two free and open-source tools in the Git ecosystem. Designed for total beginners! We'll be using: GitHub Actions: https://github.com/features/actions CML: https://github.com/iterative/cml Resources: Code: https://github.com/andronovhopf/wine GitLab support: https://github.com/iterative/cml/wiki

Load video
YouTube
MLOps Tutorial #3: Track ML models with Git & GitHub Actions
DVCorg In this tutorial, we'll compare ML models across two different Git branches of a project- and we'll do it in a continuous integration system (GitHub Actions) for automation superpowers! We'll cover:

- Why comparing model metrics takes more than a git diff - How pipelines, a method for making model training more reproducible, help you standardize model comparisons across Git branches - How to display a table comparing model performance to the main branch in a GitHub Pull Request

Helpful links: Dataset: Data on farmers’ adoption of climate change mitigation measures, individual characteristics, risk attitudes and social influences in a region of Switzerland https://www.sciencedirect.com/science/article/pii/S2352340920303048 Code: https://github.com/elleobrien/farmer DVC pipelines & metrics documentation: https://dvc.org/doc/start/data-pipelines#data-pipelines CML project repo: https://github.com/iterative/cml DVC Discord channel: https://discord.gg/bzA6uY7

DevSecOps

Youtube search... ...Google search

DevSecOps (also known as SecDevOps and DevOpsSec) is the process of integrating secure development best practices and methodologies into continuous design, development, deployment and integration processes

Extending DevSecOps to models and AI systems (MLSecOps). Machine learning adds assets that ordinary security scanners do not understand. A secure AI pipeline adds controls such as these:

  • Treat model files as code. Pickle-based model formats can execute arbitrary code when loaded. Prefer non-executable formats such as safetensors, scan third-party models before use, and load untrusted artifacts only in sandboxes.
  • Sign and verify models. Sign models when training finishes and verify the signature every time they are loaded or deployed. The OpenSSF Model Signing (OMS) specification, built on Sigstore, is the emerging standard. Remember the limit: a valid signature proves integrity and the identity of a key, not that the signer was authorized or that the training process was sound.
  • Produce an AI bill of materials. CycloneDX's ML-BOM profile and SPDX 3.0's AI and dataset profiles extend SBOMs to record base checkpoints, datasets, training frameworks, and fine-tuning runs.
  • Protect training data. Version and hash datasets, restrict who can write to them, and watch for anomalous contributions that could indicate poisoning.
  • Threat-model the application layer. Use the OWASP Top 10 for LLM Applications (2026 edition) and the Top 10 for Agentic Applications as checklists for prompt injection, sensitive data disclosure, excessive agency, supply chain risks, and unbounded resource consumption.
  • Follow secure development guidance. NIST SP 800-218A (July 2024) extends the Secure Software Development Framework with practices for generative AI and dual-use foundation models, and NIST's draft Cyber AI Profile maps Cybersecurity Framework 2.0 outcomes to AI systems.
  • Red-team before and after release. Run adversarial testing before launch and after major model, prompt, or tool changes, and turn every successful attack into a regression test.
Dull, Dangerous, and/or Dirty

Load video
YouTube
DevSecOps : What, Why and How
Black Hat In this talk, we shall focus on how a DevOps pipeline can easily be metamorphosed into a DevSecOps and the benefits which can be achieved with this transformation. The talk (assisted with various demos) will focus on developing a DevSecOps pipeline using free/open-source tools in various deployment platforms, i.e. on-premise, cloud native and hybrid scenarios. By Anant Shrivastava Full Abstract & Presentation Material

Load video
YouTube
DevSecOps State of the Union
Clint Gibler, Research Director, NCC Group It’s tough to keep up with the DevSecOps resources out there, or even know where to start. This talk will summarize and distill the unique tips and tricks, lessons learned, and tools discussed in dozens of blog posts and more than 50 conference talks over the past few years, and combine it with knowledge gained from in-person discussions with security leaders at companies with mature security programs. Pre-Requisites: General understanding of the fundamental areas of modern application security programs, including threat modeling, secure code reviews, security training, building security culture/developing security champions, security scanning (static and dynamic analysis tools), monitoring and logging in production, etc. Understanding of how software generally moves from development to production in agile environments that embrace CI/CD practices. Basic understanding of the principles of network/infrastructure and cloud security.

Load video
YouTube
Webinar: Critical DevSecOps considerations for multi-cloud Kubernetes
Cloud Native Computing Foundation (CNCF) The distributed nature of Kubernetes has turned both legacy infrastructure and traditional cybersecurity approaches on their heads. Organizations building cloud-native environments in their own data centers grapple with operationalizing and scaling Kubernetes clusters, and then ensuring system-wide security from the infrastructure layer all the way up to each container. In this webinar, you’ll hear from two cloud-native experts in infrastructure and security who will offer up valuable insights on: - How containerized applications use compute, storage, and network resources differently than do legacy applications - Why hyper-converged infrastructure is suited for Kubernetes - How a Kubernetes stack should be instrumented for observability - Best practices for implementing system-wide security for multi-cloud Kubernetes Presented by Nutanix and Sysdig Presenters: Sylvain Huguet, Sr. Product Manager - Karbon/Kubernetes @Nutanix Loris Digioanni, CTO & Founder @Sysdig.

Load video
YouTube
DevOps for Beginners Where to Start
DevOpsTV Tim Beattie Global Head of Product, Open Innovation Labs, RedHat If you're beginning to learn more about DevOps, you may be confused about where to start or where to focus. The term “DevOps” refers to this idea that we no longer have pure and separated development streams and operations streams, and to a set of DevOps practices that optimizes and improves both functions. Join this webinar to explore how to get going with this cross-functional way of working that breaks down walls, removes bottlenecks, improves speed of delivery, and increases experimentation.

Load video
YouTube
DevSecOps Days DC 2020
DevSecOps Days is a global series of one and two day conferences helping to educate, evolve, and explore concepts around developing security as code. Join us online to meet fellow practitioners integrating security into their DevOps practices, learn about their journeys, share ideas on integrating security into your teams, and trade insights on automating security tools in your DevOps pipeline.

Welcome (Hasan Yasar) 00:00--10:47

The Need for Threat Modeling in a DevSecOps World (Simone Curzi) 10:48--44:50

Close the Gap: Bringing Engineering & Security/Compliance Together (Sare Ehmann) 49:15--1:20:29

Delivering DevSecOps Value Faster? Use Metrics and Standards (Steven Woodward) 1:24:15--1:54:13

Navigating DevOps Requirements with Risk Management (Victoria (Vicky) Hailey) 2:02:03--2:25:05

Make it Personal to Make it Happen (Ruth Lennon) 02:34:11--03:03:00

Keynote: Successful DevSecOps Implementation for Production Environments is a Journey Not a Milestone (Christopher Brazier) 03:44:11--04:29:24

Evolve Your DevSecOps to Manage Speed and Risk (Altaz Valani) 04:34:09--05:03:12

How Software Acquisition and DevSecOps Can Improve the Lethality of the DOD (Sean Brady) 05:14:05--05:42:48

Why Organizations Need DevSecOps Now More Than Ever (Krishna Guru) 05:47:41--06:14:05

Wrap-up (HasanYasar) 06:14:17--06:26:29

Load video
YouTube
DevSecOps Days Pittsburgh 2020
Software Engineering Institute | Carnegie Mellon University Speaker Presentations (with timestamps)

Welcome and Introductions Hasan Yasar 00:00:32 - 00:17:13

Keynote: Core Values and Customer-Focused Delivery: DevOps in Special Operations Jason Turse 00:27:57 - 01:11:05

Are We Building Just Containers or Secure Containers? Arun Chaudhary 01:15:28 - 01:43:48 https://resources.sei.cmu.edu/library...

Are your Pipelines Secure? Lock-down Unsecured CI/CD Pipelines Angel Rivera 01:47:25 - 02:20:05

Risk: A Bias Toward Action Jason Lamb 02:30:25 - 02:55:33

The Innovation Ninja Amber Vanderburg 03:00:30 - 03:30:34

Security is an Awesome Product Feature Mark P. Hahn 04:16:57 - 04:43:08 https://resources.sei.cmu.edu/library...

Exhilarating Journey of Transformation into Digital Edge Himanshu Patel 04:47:57 - 05:16:12 https://resources.sei.cmu.edu/library...

Invited Speaker: Intentional Culture Architecture Jeff Shupack 05:20:11 - 05:40:17

Service Mess to Service Mesh Rob Richardson 05:59:56 - 06:26:35 https://resources.sei.cmu.edu/library...

Code to Kubernetes: Deployment Shouldn’t be an Afterthought Lakmal Warusawithana 06:32:25 - 06:53:30

Design Thinking, Security, and Your Life Alex Cowan 06:59:56 - 07:23:28 https://resources.sei.cmu.edu/library...

Keynote: How to Use Hacker Personas to Successfully Build DevSecOps Pipeline Robin Yeman 07:29:54 - 08:00:45 https://resources.sei.cmu.edu/library/...

Outbrief and Closing Remarks Suzanne Miller • Hasan Yasar 08:12:18 - 08:26:42

Meet fellow practitioners integrating security into their DevOps practices. Learn about their journeys, share ideas on integrating security into your teams, and trade insights on automating security tools in your DevOps pipeline.

DevSecOps in Government

Update (2025 to 2026). The 2019 Reference Design above was the starting point for a series of later DoD CIO DevSecOps documents, and policy has kept moving toward continuous authorization. In spring 2025 the DoD CIO launched the Software Fast Track (SWFT) initiative with a 90-day sprint and three requests for information. SWFT aims to replace slow, checklist-driven Authority to Operate reviews with faster, government-led risk determinations based on SBOMs, supply chain risk management data, third-party assessments, and machine-readable security artifacts processed with automation and AI. Vendor analyses report that SWFT moved from a pilot toward a requirement, with self-attestation required in solicitations from November 10, 2025, and that containerized software delivered to DoD software factories must meet hardened-container expectations. For AI programs, the implication is direct: models shipped in containers inherit the same SBOM and continuous-evidence expectations as any other software, which makes AI bills of materials and signed model artifacts practical necessities.

Load video
YouTube
DOD Enterprise DevSecOps Initiative
CSIAC The current Department of Defense (DOD) software acquisition process is not responsive to the needs of our warfighters. Therefore, it is difficult for the DOD to keep pace with our potential adversaries and avoid falling behind them. To address this situation, the DOD is pursuing a new software development activity called the DOD Enterprise DevSecOps Initiative. This webinar will present the vision for transforming DOD software acquisition into secure, responsive software factories. It will examine and explore the utilization of modern software development processes and tools to revolutionize the Department’s ability to provide responsive, timely, and secure software capabilities for our warfighters. The focus of the effort involves exploiting automated software tools, services, and standards so warfighters can rapidly create, deploy, and operate software applications in a secure, flexible, and interoperable manner. Slides

Load video
YouTube
DevSecOps Implementation in the DoD: Barriers and Enablers
Software Engineering Institute | Carnegie Mellon University Today's DOD software development and deployment is not responsive to warfighter needs. As a result, the DOD's ability to keep pace with potential adversaries is falling behind. In this webcast, panelists discuss potential enablers of and barriers to using modern software development techniques and processes in the DOD or similar segregated environments. These software development techniques and processes are as commonly known as DevSecOps.

Load video
YouTube
Andrew Weiss/Harshal Dharia: Government DevSecOps with Azure and GitHub
Andrew Weiss, Federal Solutions Engineer, GitHub and Harshal Dharia, Senior Azure Specialist, Microsoft Federal presents Government DevSecOps with Azure and GitHub at the Azure Government February 2020 meetup.

Load video
YouTube
Infrastructure: Foundations of the Future
GovernmentCIO Media & Research As agencies continue to modernize and move away from their legacy systems, how are they reshaping their IT infrastructure and leveraging data to transform their enterprise? Hear from top government IT leaders at the VA, DOD, and HHS about how they’re adopting cloud and data center infrastructure, strategies and capabilities to move their organizations toward the future.

Strangler Fig / Strangler Pattern

Strangulation of a legacy or undesirable solution is a safe way to phase one thing out for something better, cheaper, or more expandable. You make something new that obsoletes a small percentage of something old, and put them live together. You do some more work in the same style, and go live again (rinse, repeat). Strangler Applications | Paul Hammant ...case studies

Applying the pattern to ML and LLM systems. The strangler fig maps naturally onto model replacement. Put a facade, such as an API gateway or model router, in front of the legacy decision logic. Route a thin slice of traffic to the new model in shadow mode first, comparing its decisions with the legacy system without acting on them. Then let it serve a small, low-risk segment, expand as evidence accumulates, and retire the old path last. The same approach works for replacing a rules engine with a model, a classic ML pipeline with an LLM-based component, or one model provider with another. The facade is what makes each step cheap to reverse.

Load video
YouTube
"Choking the Monolith - The Strangler Pattern Applied" by Tobias Goeschel (@w3ltraumpirat)
The so called "Strangler Fig" (aka "Strangler") pattern is a much cited strategy for replacing legacy systems with new, often microservice-based architectures. However, it's not actually a microservice pattern, and there are several, quite different, ways to implement it. Are you confused yet? Fear not: We will have a look at the theory, and then explore together, how Strangler Fig could be used to improve and replace a project worthy being called "The Most Horrible Piece Of Code in the World". As a principal consultant at codecentric, Tobias has seen, improved and survived enough of "the code that earns our money" to enjoy it. Slides

Load video
YouTube
AWS New York Summit 2019: Migrating Monolithic Applications with the Strangler Pattern (FSV303)
Learn more about Amazon AWS Global Summits at - “Lifting and shifting” an enterprise-scale application will yield some of the benefits of the cloud, but elasticity and agility may still be limited. Conversely, rewriting that application to be cloud-native can be costly in terms of both time and money and could cause you to miss market opportunities. This session explores the challenges financial institutions face when migrating their existing portfolio of applications to the cloud. Then, we share practical tips to migrate applications to realize cloud-native architectural benefits incrementally using the strangler pattern.

Model Monitoring

YouTube search... ...Google search

Monitoring production systems is essential to keeping them running well. For ML systems, monitoring becomes even more important, because their performance depends not just on factors that we have some control over, like infrastructure and our own software, but also on data, which we have much less control over. Therefore, in addition to monitoring standard metrics like latency, traffic, errors and saturation, we also need to monitor model prediction performance. An obvious challenge with monitoring model performance is that we usually don’t have a verified label to compare our model’s predictions to, since the model works on new data. In some cases we might have some indirect way of assessing the model’s effectiveness, for example by measuring click rate for a recommendation model. In other cases, we might have to rely on comparisons between time periods, for example by calculating a percentage of positive classifications hourly and alerting if it deviates by more than a few percent from the average for that time. Just like when validating the model, it’s also important to monitor metrics across slices, and not just globally, to be able to detect problems affecting specific segments. ML Ops: Machine Learning as an Engineering Discipline | Cristiano Breuel - Towards Data Science

Types of change to watch. "Drift" covers several different problems, and each needs a different detector:

Change What shifts How it shows up Typical detection
Data (covariate) drift The input distribution Feature statistics move away from training Population Stability Index, Kolmogorov-Smirnov tests, Jensen-Shannon distance, embedding drift
Prior (label) shift The class balance The mix of predicted classes changes Monitor predicted-class rates; confirm with delayed labels
Concept drift The relationship between inputs and outcomes Accuracy falls while inputs look normal Performance on delayed labels, proxy outcomes, backtesting
Training/serving skew Feature computation differs between training and production Strong offline metrics, weak live results Compare logged serving features with training features
Upstream data breakage Schemas, units, null rates, pipeline bugs Sudden, sharp metric changes Schema and range checks, data contracts
LLM behavior change Provider model updates, prompt edits, retrieval index changes Different answers to the same questions Regression evaluation suites, judged quality on sampled traces

When labels are late or missing. Labels often arrive weeks later (fraud chargebacks) or never. Use input and prediction drift as early warnings, track proxy outcomes such as clicks or human reviewer overrides, sample production cases for human labeling, and measure performance by slice once labels arrive. Remember that drift alone does not prove harm; many drift alerts are benign, so tie alerts to business impact wherever possible to avoid alert fatigue.

Monitoring LLM and agent systems. Trace every request: prompt, retrieved context, model and parameters, tool calls, outputs, token counts, latency, and cost. OpenTelemetry-compatible instrumentation keeps those traces portable across vendors. Sample traces for LLM-judge scoring and human review, and track refusal rates, guardrail triggers, and flagged hallucinations. For agents, also track task success, steps per task, tool errors, and policy violations. Because traces capture user content, apply redaction and retention policies; OpenTelemetry's GenAI instrumentation treats capture of message content as an opt-in setting for this reason.


Load video
YouTube
Josh Wills: Visibility and Monitoring for Machine Learning Models
Josh started our Meetup with a short talk on deploying machine learning models into production. He's worked as the Director of Data Science at Cloudera, he wrote the Java version of Google's AB testing framework, and he recently held the position of Director of Data Engineering at Slack. In his opinion the most important question is: "How often do you want to deploy this?" You should never deploy a machine learning model once. If the problem is not important enough to keep working on it and deploy new models, then its not important enough to pay the cost of putting it into production in the first place. Watch his talk to get his thoughts on testing machine learning models in production.

Load video
YouTube
Concept Drift: Monitoring Model Quality in Streaming Machine Learning Applications
Most machine learning algorithms are designed to work on stationary data. Yet, real-life streaming data is rarely stationary. Models lose prediction accuracy over time if they are not retrained. Without model quality monitoring, retraining decisions are suboptimal and costly. Here, we review the monitoring methods and evaluate them for applicability in modern fast data and streaming applications.

Load video
YouTube
Monitoring models in production - Jannes Klaas
PyData Amsterdam 2018 A Data Scientists work is not done once machine learning models are in production. In this talk, Jannes will explain ways of monitoring Keras neural network models in production, how to track model decay and set up alerting using Flask, Docker and a range of self-built tools. www.pydata.org

Load video
YouTube
Machine Learning Models in Production
Data Scientists and Machine Learning practitioners, nowadays, seem to be churning out models by the dozen and they continuously experiment to find ways to improve their accuracies. They also use a variety of ML and DL frameworks & languages , and a typical organization may find that this results in a heterogenous, complicated bunch of assets that require different types of runtimes, resources and sometimes even specialized compute to operate efficiently. But what does it mean for an enterprise to actually take these models to "production" ? How does an organization scale inference engines out & make them available for real-time applications without significant latencies ? There needs to be different techniques for batch ,offline, inferences and instant, online scoring. Data needs to be accessed from various sources and cleansing, transformations of data needs to be enabled prior to any predictions. In many cases, there maybe no substitute for customized data handling with scripting either. Enterprises also require additional auditing and authorizations built in, approval processes and still support a "continuous delivery" paradigm whereby a data scientist can enable insights faster. Not all models are created equal, nor are consumers of a model - so enterprises require both metering and allocation of compute resources for SLAs. In this session, we will take a look at how machine learning is operationalized in IBM Data Science Experience, DSX, a Kubernetes based offering for the Private Cloud and optimized for the HortonWorks Hadoop Data Platform. DSX essentially brings in typical software engineering development practices to Data Science, organizing the dev-test-production for machine learning assets in much the same way as typical software deployments. We will also see what it means to deploy, monitor accuracies and even rollback models & custom scorers as well as how API based techniques enable consuming business processes and applications to remain relatively stable amidst all the chaos. Speaker: Piotr Mierzejewski, Program Director Development IBM DSX Local, IBM

A/B Testing

YouTube search... ...Google search

A/B testing (also known as bucket testing or split-run testing) is a user experience research methodology. A/B tests consist of a randomized experiment with two variants, A and B. It includes application of statistical hypothesis testing or "two-sample hypothesis testing" as used in the field of statistics. A/B testing is a way to compare two versions of a single variable, typically by testing a subject's response to variant A against variant B, and determining which of the two variants is more effective. A/B testing | Wikipedia

A randomized controlled trial (or randomized control trial; RCT) is a type of scientific (often medical) experiment that aims to reduce certain sources of bias when testing the effectiveness of new treatments; this is accomplished by randomly allocating subjects to two or more groups, treating them differently, and then comparing them with respect to a measured response. One group, the experimental group, receives the intervention being assessed, while the other, usually called the control group, receives an alternative treatment, such as a placebo or no intervention. The groups are monitored under conditions of the trial design to determine the effectiveness of the experimental intervention, and efficacy is assessed in comparison to the control. There may be more than one treatment group or more than one control group. Randomized controlled trial | Wikipedia

Getting A/B tests right for models.

  • Avoid the peeking problem. Checking a classical p-value repeatedly while a test runs inflates false positives. Use always-valid (sequential) inference, as in the Stanford seminar below, or a pre-registered group-sequential design.
  • Reduce variance. CUPED, developed at Microsoft, uses each user's pre-experiment behavior as a covariate to shrink variance, so tests reach a conclusion with less traffic.
  • Use interleaving for ranking. For search and recommendation models, interleaving mixes results from both rankers in one list and is far more sensitive than a split test.
  • Consider bandits when regret matters. Multi-armed bandits shift traffic toward the better variant during the test, trading statistical clarity for fewer users exposed to the worse option.
  • Watch guardrail metrics. Track latency, error rates, cost, and complaint rates alongside the primary metric, so a "winning" model cannot quietly degrade the service.
  • Account for novelty and interference. Early enthusiasm for anything new fades, and in marketplaces or social products one user's treatment can affect another's outcome.
  • Test offline first. Shadow deployments and replay of logged data catch obvious regressions before any user is exposed.
  • For LLM features, response-level variance is high and "quality" is multidimensional. Combine online outcomes (task completion, retention, explicit feedback) with judged quality on sampled traffic, and do not let an LLM judge be the only success metric.

Load video
YouTube
Stanford Seminar: Peeking at A/B Tests - Why It Matters and What to Do About It
Ramesh Johari Stanford University I'll describe a novel statistical methodology that has been deployed by the commercial A/B testing platform Optimizely to communicate experimental results to their customers. Our methodology addresses the issue that traditional p-values and confidence intervals give unreliable inference. This is because users of A/B testing software are known to continuously monitor these measures as the experiment is running. We provide always valid p-values and confidence intervals that are provably robust to this effect. Not only does this make it safe for a user to continuously monitor, but it empowers her to detect true effects more efficiently. I'll cover why continuous monitoring is a problem, and how our solution addresses it. The talk will be presented at a level that does not presume much statistical background. Ramesh Johari is an Associate Professor at Stanford University, with a full-time appointment in the Department of Management Science and Engineering (MS&E), and courtesy appointments in the Departments of Computer Science (CS) and Electrical Engineering (EE). He is a member of the Operations Research group and the Social Algorithms Lab (SOAL) in MS&E, the Information Systems Laboratory (ISL) in EE, and the Institute for Computational and Mathematical Engineering (ICME). Learn more about Stanford's Human-Computer Interaction Group: https://hci.stanford.edu

Load video
YouTube
Facebook's A B Platform Interactive Analysis in Realtime - @Scale 2014 - Data
Itamar Rosenn, Engineering Manager at Facebook This talk presents Facebook¹s platform for streamlined and automated A/B test analysis. We will discuss the system architecture, how it enables interactive analysis on near-realtime results, and the challenges we've faced in scaling the system to meet our customer needs.

Scoring Deployed Models

YouTube search... ...Google search

"Scoring" means running a deployed model on new data to produce predictions; the word is also used for grading those predictions afterward. The right scoring pattern depends on how fresh predictions must be:

  • Batch scoring runs on a schedule over large tables (nightly churn scores, weekly demand forecasts). It is cheap and simple, but predictions can be stale.
  • Online (real-time) scoring answers individual requests through an API within milliseconds, as in fraud checks at payment time.
  • Streaming scoring attaches models to event streams (Kafka, Flink), as in the video below, so predictions update as events arrive.
  • Edge scoring runs compact models on devices or gateways using portable formats such as H2O MOJOs, ONNX Runtime, or LiteRT (formerly TensorFlow Lite), trading model size for latency, privacy, and offline operation.

Serving infrastructure. Common choices include KServe on Kubernetes (for predictive models and, through its LLM inference service and llm-d integration, large language models), NVIDIA Triton, vLLM for LLM inference, and managed endpoints in SageMaker AI, Vertex AI, and Azure Machine Learning.

What to measure. For classic models, track latency percentiles (p50, p95, p99), throughput, error rates, hardware utilization, and cost per thousand predictions. For LLMs, add time to first token, inter-token latency, tokens per second, and cost per request. Batching, quantization, distillation, caching, and autoscaling (including scaling to zero for spiky workloads) are the main levers. A champion/challenger setup, where a challenger model scores the same traffic in shadow, keeps a ready replacement measured against the live champion at all times.

Load video
YouTube
ML Model Deployment and Scoring on the Edge with Automatic ML & DF / Flink2Kafka
recorded on June 18, 2020. Machine Learning Model Deployment and Scoring on the Edge with Automatic Machine Learning and Data Flow Deploying Machine Learning models to the edge can present significant ML/IoT challenges centered around the need for low latency and accurate scoring on minimal resource environments. H2O.ai's Driverless AI AutoML and Cloudera Data Flow work nicely together to solve this challenge. Driverless AI automates the building of accurate Machine Learning models, which are deployed as light footprint and low latency Java or C++ artifacts, also known as a MOJO (Model Optimized). And Cloudera Data Flow leverage Apache NiFi that offers an innovative data flow framework to host MOJOs to make predictions on data moving on the edge. Speakers: James Medel (H2O.ai - Technical Community Maker) Greg Keys (H2O.ai - Solution Engineer) Kafka 2 Flink - An Apache Love Story This project has heavily inspired by two existing efforts from Data In Motion's FLaNK Stack and Data Artisan's blog on stateful streaming applications. The goal of this project is to provide insight into connecting an Apache Flink applications to Apache Kafka. Speaker: Ian R Brooks, PhD (Cloudera - Senior Solutions Engineer & Data)

Load video
YouTube
Shawn Scully: Production and Beyond: Deploying and Managing Machine Learning Models
PyData NYC 2015 Machine learning has become the key component in building intelligence-infused applications. However, as companies increase the number of such deployments, the number of machine learning models that need to be created, maintained, monitored, tracked, and improved grow at a tremendous pace. This growth has lead to a huge (and well-documented) accumulation of technical debt. Developing a machine learning application is an iterative process that involves building multiple models over a dataset. The dataset itself evolves over time as new features and new data points are collected. Furthermore, once deployed, the models require updates over time. Changes in models and datasets become difficult to track over time, and one can quickly lose track of which version of the model used which data and why it was subsequently replaced. In this talk, we outline some of the key challenges in large-scale deployments of many interacting machine learning models. We then describe a methodology for management, monitoring, and optimization of such models in production, which helps mitigate the technical debt. In particular, we demonstrate how to: Track models and versions, and visualize their quality over time Track the provenance of models and datasets, and quantify how changes in data impact the models being served Optimize model ensembles in real time, based on changing data, and provide alerts when such ensembles no longer provide the desired accuracy.

1*ldWxdWDzEYnSbvchuL5k1w.png

Retrieved from "https://primo.ai/w/index.php?title=Algorithm_Administration&oldid=51940"