...find outliers
Jump to navigation
Jump to search
| AI Solver map |
|---|
|
Your current page is in bold
0 · Start
1 · Predict & analyze
tables, numbers, events
Predict values
Predict categories Forecast time series Cluster / segment Find outliers Make recommendations Find key drivers 2 · Understand unstructured data
text, documents, images, audio
3 · Generate & converse
4 · Decide & act
5 · Build it right
applies to every path
|
YouTube ... Quora ...Google search ...Google News ...Bing News
- Embedding ... Fine-tuning ... RAG ... Clustering ... Recommendation ... Anomaly Detection ... Classification ... Dimensional Reduction. ...find outliers
- Cybersecurity ... OSINT ... Frameworks ... References ... Offense ... NIST ... DHS ... Screening ... Law Enforcement ... Government ... Defense ... Lifecycle Integration ... Products ... Evaluating
- Signal Processing ... Pathology
Use this page when you want to flag what is unusual: fraud, intrusions, machine faults, defective parts, data-entry errors. Usually you have few or no labeled anomalies, so the model learns what "normal" looks like and flags departures.
Outliers don't necessarily represent abnormal behavior
Start here
- Do you have enough labeled examples of past anomalies (confirmed fraud, known failures)?
- Yes → it is imbalanced classification, which usually beats unsupervised detection: ...predict categories
- No or very few → learn "normal" and flag deviations (below); use the alerts people confirm to build labels over time
- What kind of data?
- A single metric → robust statistics first: z-score, interquartile range, median absolute deviation; remove seasonality first (e.g., STL decomposition) and threshold the remainder
- Time series or event streams (sensors, metrics, logs)
- Forecast, then compare: flag points far outside the forecast range (statistical models or foundation models such as Chronos, TimesFM or Datadog's Toto) → ...forecast time series
- Unusual shapes or subsequences → Matrix Profile (STUMPY)
- Looking for event based patterns? → Bidirectional Long Short-Term Memory (BI-LSTM) with Attention Mechanism
- High-speed streams, one pass → online detectors such as Half-Space Trees (River library)
- Tabular records
- Default → Isolation Forest (fast, scales well); Local Outlier Factor for local density; ECOD or COPOD when you want no tuning. The PyOD library offers dozens of detectors behind one interface
- Do you have > 100 features? Yes, then try One-class Support Vector Machine (SVM) (best on modest data sizes)
- No, need fast training, then try Principal Component Analysis (PCA)-based Anomaly Detection (flag rows the main components reconstruct poorly)
- Images (scratches, cracks, missing parts) → train on "good" images only: PatchCore or PaDiM (Anomalib library), or an Autoencoder (AE) / Encoder-Decoder / Variational Autoencoder (VAE) reconstruction error
- Text and logs → embed messages, flag those far from known clusters; use an LLM to summarize and triage alerts
- Networks (fraud rings, money laundering, bot accounts) → graph features (shared devices, cards, addresses) or graph neural networks
- Also consider...
Quick picks
| Data | Try first | Then compare |
|---|---|---|
| One metric | Robust z-score / IQR after removing seasonality | Forecast residuals |
| Many metrics over time | Forecast residuals (statistical or foundation model) | Matrix Profile, BI-LSTM |
| Table of records | Isolation Forest | LOF, ECOD, one-class SVM, PCA reconstruction |
| Images of products | PatchCore / PaDiM | Autoencoder |
| Labeled past incidents | Gradient-boosted classifier | Anomaly score as an extra feature |
Real-world examples
- Card-testing fraud — Stripe's payments foundation model turns each transaction into an embedding; clusters of suspicious small charges stand out, and Stripe reported a 64% jump in detecting card-testing attacks on large businesses. Stripe unveils AI foundation model for payments | TechCrunch
- Product change to note — Microsoft retired its Azure AI Anomaly Detector, Metrics Advisor and Personalizer services by 1 October 2026 and points Anomaly Detector users to Microsoft Fabric and an open-source time-series anomaly detector. Prefer portable, open-source detectors for long-lived systems. Microsoft is retiring several Azure AI services | MSPoweruser ... Migration answer from the Azure team | Microsoft Q&A
Watch out
- Outliers are not always errors — the record customer, the viral product and the rare disease are real. Investigate before deleting.
- Alert fatigue — tune thresholds so the alert volume fits the team that reviews it.
- Normal changes — retrain as "normal" drifts (new products, seasons, software releases).
Measure success
Precision of the top-k alerts that people review, recall on known past incidents, false-alarm rate per day, and time to detect. More: ...evaluate and trust AI
Learn more
- PyOD: Python library for outlier and anomaly detection
- Novelty and outlier detection | scikit-learn
- Anomalib: visual anomaly detection library
- Outlier | Wikipedia
- Machine Learning: Trying to detect outliers or unusual behavior | Stacey Ronaghan - Medium
- Outlier Detection and Removal Algorithm in K-Means and Hierarchical Clustering | A. Barai and L. Dey
An Outlier is a rare chance of occurrence within a given data set. In Data Science, an Outlier is an observation point that is distant from other observations. An Outlier may be due to variability in the measurement or it may indicate experimental error. Outliers, being the most extreme observations, may include the sample maximum or sample minimum, or both, depending on whether they are extremely high or low. However, the sample maximum and minimum are not always outliers because they may not be unusually far from other observations. - Outlier Detection and Anomaly Detection with Machine Learning | Mehul Ved - Medium
Where next? ◀ Predict & analyze · ...predict categories · ...forecast time series · ...cluster · ▶ ...evaluate and trust AI