Capabilities

From PRIMO.ai
Jump to navigation Jump to search

YouTube ... Quora ...Google search ...Google News ...Bing News


In computer science and cognitive engineering, artificial intelligence capabilities define the functional transformations, perceptual discriminations, generative syntheses, and autonomous control actions that a computational system can execute across distinct data modalities. Rather than treating Artificial Intelligence (AI) as a monolithic entity, capability engineering decomposes machine intelligence into measurable mappings from an input space $\mathcal{X}$ (such as numeric tables, text sequences, acoustic waveforms, spoken language, static images, or continuous video streams) to an output space $\mathcal{Y}$ (such as a predicted scalar value, a classification label, a synthesized voice waveform, a 3D geometric mesh, or an executable database query).

The theoretical and architectural foundations governing AI capabilities were established across several generations of research:

  • Foundational Computability, Representation, and Causality: Alan Turing formulated the limits of mechanical computation and behavioral evaluation; Claude Shannon established the information-theoretic bounds of encoding, channel capacity, and cross-modal compression; David Marr defined the three levels of analysis for any information-processing system (the computational level specifying the input-output goal, the algorithmic/representational level specifying how inputs and outputs are encoded and transformed, and the implementation level specifying the physical substrate); and Judea Pearl formalized the Ladder of Causation, separating statistical association $P(y \mid x)$ from active intervention $P(y \mid \text{do}(x))$ and counterfactual reasoning.
  • Deep Representation Learning and Unified Architectures: Geoffrey Hinton, Yann LeCun, and Yoshua Bengio demonstrated that multi-layer neural networks trained via backpropagation can automatically discover hierarchical representations from raw sensory data, eliminating hand-crafted feature extractors in Vision and Speech Recognition. In 2017, Ashish Vaswani and colleagues at Google introduced the Transformer self-attention architecture, which provided a universal computational backbone capable of processing any modality that can be decomposed into discrete or continuous vector tokens.
  • Foundation Models and Rigorous Evaluation: Percy Liang, Rishi Bommasani, and colleagues at the Stanford Center for Research on Foundation Models (CRFM) formalized the paradigm of foundation models (large-scale models pre-trained on broad data that can be adapted to a wide range of downstream capabilities) and established the Holistic Evaluation of Language Models (HELM) framework. In parallel, François Chollet (creator of Keras and co-founder of the ARC Prize Foundation with Mike Knoop and Greg Kamradt) established a formal mathematical distinction between memorized task skill and general fluid intelligence (defined as skill-acquisition efficiency on novel, out-of-distribution problems).

Historically, every cell in the AI input-output capability matrix required a separate, incompatible software architecture. Converting speech to text required Hidden Markov Models (HMMs) coupled with Gaussian Mixture Models; classifying images required specialized convolutional neural networks (CNNs); forecasting numeric time series required ARIMA or gradient-boosted decision trees; and translating text required statistical phrase tables. If an engineer wanted to build a system that watched a video and answered a spoken question, they had to bolt together four brittle, single-modality pipelines where errors compounded at every handoff.

Today, the field has undergone a two-fold paradigm shift. First, native multimodal foundation models project numeric, text, audio, image, video, and 3D tokens into a shared high-dimensional latent manifold, enabling a single model backbone to perform any-to-any cross-modal transformations. Second, researchers have unlocked test-time compute scaling (inference-time reasoning, tree search, and program synthesis), allowing models to move beyond instantaneous "System 1" pattern matching and allocate variable computational effort to verify complex math, code, and multi-step agentic workflows before emitting an output.


Every artificial intelligence capability reduces to a mathematical transformation from an input modality to an output modality across a learned representation space; while unified foundation models and test-time reasoning now saturate static human knowledge benchmarks, true capability evaluation requires distinguishing interpolative pattern matching from sample-efficient adaptation to novel, out-of-distribution tasks.


Recent Developments

Between 2024 and 2026, empirical research, open-frontier training experiments, and new evaluation benchmarks have reshaped our understanding of what AI systems can do, how capabilities scale, and where fundamental limitations remain.

1. Benchmark Saturation, Leaderboards, and the Jagged Frontier (2025 to 2026)

The ninth edition of the Stanford HAI AI Index Report (2026) documents that AI capabilities are outpacing the static benchmarks designed to measure them, while simultaneously exposing severe reliability and contamination flaws in older test suites:

  • Rapid Saturation of Expert Knowledge and Coding Exams: On SWE-bench Verified (which evaluates an AI system's ability to resolve real GitHub issues across Python repositories), frontier model performance jumped from roughly 60% in 2024 to nearly 100% by 2025/2026. On Humanity's Last Exam (HLE), a benchmark specifically constructed by domain experts to resist saturation, frontier models gained 30 percentage points in a single year.
  • Convergence at the Frontier (U.S.-China and Open-Closed Parity): As of March 2026, human preference evaluations on the Chatbot Arena Leaderboard show the top four model developers clustered within just 25 Elo points: Anthropic (1,503), xAI (1,495), Google (1,494), and OpenAI (1,481), followed closely by Alibaba (1,449) and DeepSeek (1,424). After DeepSeek-R1 matched the top U.S. reasoning model in February 2025, U.S. and Chinese models traded the top ranking multiple times, with the U.S. lead standing at a narrow 2.7% in March 2026 and the performance gap between closed-weight and open-weight models standing at 3.3%.
  • Benchmark Error Rates and Gaming: Audits of widely used evaluations revealed invalid or mislabeled question rates ranging from 2% on MMLU Math to 42% on GSM8K. In addition, open-data curation pipelines (such as Stanford's Marin project) found that public web-crawl datasets routinely contain leaked copies of evaluation test sets, requiring strict $n$-gram decontamination before pre-training.
  • Olympiad Gold vs. ClockBench (Jagged Capabilities): Illustrating the gap between symbolic pattern reasoning and basic visual perception, Google's Gemini Deep Think scored 35 points (reaching the Gold Medal threshold) at the 2025 International Mathematical Olympiad (IMO), solving complex mathematical proofs end-to-end in natural language within the 4.5-hour time limit (up from a 28-point Silver performance in 2024). Yet on ClockBench, the top frontier multimodal model read analog clock faces correctly only 50.6% of the time, compared to 90.1% for human test-takers. In scientific research workflows, while AI accelerates specific sub-tasks, frontier models still score roughly half of what PhD experts achieve on end-to-end research benchmarks and achieve only ~17% accuracy on complex, real-world Bioinformatics analysis.

2. Fluid Intelligence vs. Memorization: ARC-AGI-1, ARC-AGI-2, and ARC-AGI-3 (2024 to 2026)

To test whether AI models possess genuine fluid intelligence (the ability to synthesize novel rules from 2 to 4 examples without relying on memorized internet templates), François Chollet introduced the Abstraction and Reasoning Corpus (ARC-AGI-1) in 2019:

  • The Test-Time Compute Breakthrough on ARC-AGI-1 (2024): From 2019 to early 2024, pure large language models failed on ARC-AGI-1, with GPT-3 scoring 0% and base direct-prompted LLMs scoring between 8% and 21%. In late 2024, two test-time compute paradigms broke the stalemate: (1) open-source Kaggle competitors (such as Jack Cole's MindsAI and the ARChitects) combined deep-learning-guided program search with test-time training (fine-tuning a model dynamically at inference time on synthetic variations of each test task) to reach 46% to 55.5% on the private holdout; and (2) in December 2024, OpenAI's o3-preview reasoning system scored 75.7% on the semi-private evaluation under low compute (~$20 per task) and 87.5% under high compute (170x more compute, costing thousands of dollars per task).
  • ARC-AGI-2 Resets the Reasoning Baseline (2025 to 2026): Recognizing that single-rule tasks in ARC-AGI-1 could be brute-forced by massive test-time search, Chollet, Greg Kamradt, and the ARC Prize Foundation launched ARC-AGI-2 in March 2025 (calibrated across 400+ human participants in San Diego, where 100% of tasks were solved by at least two humans in $\le 2$ attempts, with an average individual human score of 66%). ARC-AGI-2 requires multi-step compositional reasoning, contextual rule switching, and in-context symbolic interpretation. At launch in March 2025, ARC-AGI-2 re-collapsed frontier scores: pure LLMs scored 0%, and o3-preview-low dropped from 75.7% on ARC-AGI-1 to just 4% on ARC-AGI-2. Over the following 12 months (through April 2026), advances in adaptive test-time reasoning pushed lab-reported frontier scores on ARC-AGI-2 into the 75% to 85% range (with GPT-5.5 reporting 85%, GPT-5.4 Pro at 83.3%, Gemini 3.1 Pro at 77.1%, and Claude Opus 4.7 Adaptive at 75.8%), though no open-source system had yet claimed the audited $700,000 Kaggle Grand Prize under strict compute-cost limits.
  • ARC-AGI-3 and Interactive Agentic Efficiency (2026): In March and April 2026, the ARC Prize Foundation released ARC-AGI-3 (arXiv:2603.24621), moving beyond static input-output grids to interactive, multi-level environments where an agent must explore unknown rules, formulate goals, and complete tasks with minimal wasted actions, graded by Relative Human Action Efficiency (RHAE). On the initial April 2026 official ARC-AGI-3 leaderboard, frontier reasoning models scored under 1% (Google Gemini 3.1 Pro Preview at 0.37%, OpenAI GPT-5.4 High at 0.26%, Anthropic Claude Opus 4.6 Max at 0.25%, and xAI Grok-4.20 Beta at 0.00%), proving that autonomous, sample-efficient world modeling in unfamiliar interactive environments remains an open scientific challenge.

3. Generative Video Models as Zero-Shot Visual Learners and World Simulators (2025 to 2026)

For decades, computer Vision capabilities were fragmented across dozens of bespoke models: one model for Canny edge detection, another (such as Segment Anything) for object segmentation, another for optical flow, and another for image restoration. In September 2025, researchers at Google DeepMind (Thaddäus Wiedemer, Robert Geirhos, and colleagues) published Video models are zero-shot learners and reasoners (arXiv:2509.20328), demonstrating that large-scale generative video models are undergoing the same generalist unification that GPT-3 brought to Natural Language Processing (NLP).

Evaluating Veo 3 (and comparing it against Veo 2) across 62 qualitative tasks and 7 quantitative benchmarks totaling 18,384 generated 8-second videos, the researchers used a strictly zero-shot protocol: feeding the model only an initial image frame and a natural-language prompt describing a visual transformation, with zero task-specific training or weight modification. Veo 3 demonstrated emergent capabilities across four hierarchical visual tiers:

  1. Perception: Zero-shot edge detection, semantic instance segmentation, depth estimation, and super-resolution Enhancement simply by prompting the video model to fade away backgrounds or highlight boundaries over consecutive frames.
  2. Physical World Modeling: Simulating buoyancy, fluid dynamics, rigid-body collisions, lighting reflections, and material deformations.
  3. Scene Manipulation: Executing 3D novel-view synthesis (rotating the camera around a static object in a 2D photo to reveal occluded geometry) and object insertion/removal.
  4. Sequential Visual Reasoning: Solving 2D mazes by animating a path from start to finish and completing visual symmetry analogies across space and time. While specialized vision models still achieve higher precision on narrow benchmarks, the jump from Veo 2 to Veo 3 confirmed that generative spatiotemporal prediction acts as a general-purpose visual reasoning engine.

4. Digital Computer-Use Agents vs. Physical Robotics Manipulation (2025 to 2026)

Recent capability evaluations highlight a sharp divergence between digital action spaces and physical embodied action spaces (a modern manifestation of Moravec's Paradox):

  • Operating System and Terminal Agents: In digital environments where Agents/Assistants perceive screenshots or accessibility trees and execute mouse clicks, keystrokes, and bash shell commands, capabilities surged between 2024 and 2026. On OSWorld (a benchmark testing real multi-app computer tasks across operating systems), frontier agent accuracy more than quintupled from ~12% to 66.3%, coming within 6 percentage points of human performance (~72%). In software engineering, multi-agent teams demonstrated the capability to write working compilers from scratch and autonomously patch complex repository vulnerabilities, though agents still fail roughly 1 in 3 structured multi-step workplace tasks when unexpected UI states occur.
  • Physical Household vs. Simulated Robotics: In contrast, while robotic manipulation in deterministic software simulations (such as RLBench) reached 89.4% success by 2026, physical robots operating in unstructured real-world household environments succeeded in only 12% of tasks. Deformable objects (such as folding laundry), variable friction, partial visual occlusion, and contact-rich force control continue to bottleneck physical embodiment compared to purely digital capabilities.

5. Scientific Refutation and Regulatory Prohibition of Automated Lie Detection and Physiognomy

Several entries in historical AI capability lists, most notably Lie Detection (attempting to infer truthfulness from facial micro-expressions, eye tracking, or vocal pitch) and physiognomic Voice Vectors (attempting to infer a person's income level, character, or health from short voice clips), have been rigorously evaluated and refuted by primary scientific and regulatory bodies:

  • Psychological and Acoustic Evidence Against Automated Deception Detection: Comprehensive meta-analyses in experimental psychology (such as DePaulo et al.'s review of 206 studies and 24,483 judges, and Aldert Vrij's nonverbal deception research) establish that there is no universal "Pinocchio's nose": human beings do not exhibit a unique, invariant facial micro-expression or vocal tremor exclusively when lying. Instead, nonverbal cues measured by cameras and microphones reflect physiological arousal, anxiety, cognitive load, or fear, which innocent travelers, neurodivergent individuals, and stressed interviewees exhibit at high rates. Audits by the U.S. National Research Council (NRC) concluded that computerized Voice Stress Analysis (VSA) has "little or no scientific basis" and performs at roughly 50% accuracy (chance level) in ecological settings.
  • Failure of Border-Control Micro-Expression Pilots and LLM Veracity Judges: The European Union's experimental iBorderCtrl project (which tested an artificial neural network subsystem called Silent Talker to score 38 facial micro-expressions of travelers at borders in Hungary, Latvia, and Greece) was discontinued after independent evaluations and journalist tests demonstrated high false-positive rates and lack of scientific validity. More recently, a September 2025 peer-reviewed study in the Journal of Communication tested multimodal large language model personas across 12 deception-detection experiments: while the AI achieved 57.7% accuracy on low-stakes laboratory prompts (barely above the 54% human baseline), it exhibited severe truth-bias (71.7%) in casual settings, flipped to extreme lie-bias (judging over 75% of subjects as liars) in interrogation contexts, and saw its accuracy plummet to 15.9% under realistic ecological base rates.
  • Legal Restrictions Under the EU AI Act: Reflecting this scientific consensus, the European Union's Artificial Intelligence Act (Regulation (EU) 2024/1689, with prohibitions taking effect in 2025/2026) explicitly bans the deployment of AI emotion-recognition systems in workplaces and educational institutions, and classifies automated biometric/deception assessment in law enforcement, border control, and justice administration as unacceptable or strictly regulated high-risk applications. In legitimate clinical research, acoustic Voice Vectors are restricted to validated neurological or respiratory biomarkers (such as tracking dysarthria progression in Parkinson's disease) under informed medical consent, rather than socioeconomic or behavioral profiling.

Cross-Modal Capability Matrix (Input to Output)

The matrix below maps AI capabilities by their Input Modality (columns across the top: Numeric, Text, Sound, Audio, Speech, Image, and Video) and their Target Output or Functional Operation (rows down the left). To identify the appropriate capability for an engineering problem, locate the raw data format your sensors or databases provide along the top header, and trace down to the desired output format or analytical action.

Input --->
Output v 		Numeric Text Sound Audio Speech Image Video
Numeric ...predict values .. .. .. .. .. Count Customers
Text .. SynthPub ..Create AI .. Speech Recognition Character Recognition Captioning Storytelling
Audio/Sound .. .. .. Make music .. .. ..
Speech .. Synthesize Speech .. Voice Vectors Video Dubbing .. ..
Image .. Generated Image .. .. .. Reconstruction ..
            Colorize,Image Classification  
Video .. .. .. .. .. Animate Photo Colorize
Generate Video .. .. .. Facial Animate .. Video Synthesis Fake
Model .. .. .. .. .. 3D Model 3D Model
Database Query .. Robot Writing .. .. .. Recipes Immersive Reality
Classification .. News Signal Processing SoundHound .. Colorize Vision
Cluster ...cluster Recommendation Bird Identification Spotify .. .. ..
Summary .. Headlines .. .. .. Trailer Video Intelligence
Add Detail .. .. .. .. .. Enhancement Enhancement
Calendar .. Scheduling .. .. .. .. ..
Emotion .. Sentiment .. .. Sentiment .. Sentiment
          Lie Detection   Lie Detection
Prediction ...predict values Autocomplete .. .. .. .. Predict image
Object Revelation .. Drug Discovery .. .. .. Vision#Image Retrieval / Object Detection ..
    Astronomy          
Conversation .. Assistants .. .. Assistants .. Immersive Reality Transhumanism
Companionship .. Personal Companions .. .. Personal Companions .. ..
Segmentation .. .. .. .. .. Recognize Object Recognize Object
Anomaly Detection Intruder Cybersecurity Signal Processing .. .. Pathology ..

Modality-to-Modality Capability Directory & Architectural Breakdown

The directory below details the specific input-to-output transformations indexed in the matrix above, pairing each capability with its underlying algorithmic mechanism and domain links:

  • Numeric data to Numeric data - Regression - ...predict values (Continuous value estimation and time-series extrapolation using linear/logistic regression, gradient-boosted decision trees such as XGBoost/LightGBM, or temporal transformers).
  • Numeric data to Numeric list - classify each data point into a specific group - ...cluster (Unsupervised partitioning and manifold grouping via $k$-means, DBSCAN, Gaussian Mixture Models, or deep autoencoders).
  • Numeric data to Abnormality detection - Intruder (Out-of-distribution telemetry and network packet detection using Isolation Forests, One-Class SVMs, and variational autoencoder reconstruction error).
  • Text to Abnormality detection - Cybersecurity (Log parsing, phishing/prompt-injection detection, and threat intelligence extraction using fine-tuned encoder transformers and graph anomaly detectors).
  • Text to Speech - Synthesize Speech (Converting grapheme/phoneme sequences into natural acoustic waveforms via neural codec language models, flow-matching acoustic decoders, and neural vocoders).
  • Text to Classification (pre-defined classes) - News sections/filtering, Sentiment Analysis (Mapping documents to discrete taxonomy labels or polarity scores via bidirectional encoders such as RoBERTa/DeBERTa or zero-shot LLM classifiers).
  • Text to Cluster - Recommendation engines (Embedding text items and user histories into dense vector spaces for approximate nearest-neighbor retrieval and collaborative filtering).
  • Text to Calendar - Scheduling (Temporal entity extraction, constraint satisfaction, and agentic calendar API orchestration from natural-language requests).
  • Text to Image - Synthesize Generated Image based on a description (Iterative latent denoising using Diffusion Transformers, Rectified Flow models, and autoregressive visual tokenizers conditioned on text embeddings).
  • Text to Prediction - Autocomplete (Causal next-token and next-phrase sequence modeling $p(w_t \mid w_{<t})$ in editors, search bars, and IDEs).
  • Text/Image/Video to Sentiment - Emotion detection (Multimodal affective computing that estimates expressed sentiment polarity in product reviews, call-center transcripts, and media; requires careful calibration for cultural and demographic variance).
  • Text to Video - Unscripted storytelling; story-as-a-service video storytelling (Spatiotemporal diffusion transformers generating coherent multi-second video clips, camera trajectories, and synchronized audio from narrative prompts).
  • Text to Text - Headlines, summaries ... ChatGPT | OpenAI (Abstractive and extractive text compression, translation, and dialogue using instruction-tuned and reinforcement-learning-aligned foundation models).
  • Data (Text) to Text - Robot Writing Natural Language Generation (NLG) database analysis, News content (Transforming structured SQL tables, financial filings, and sports box scores into grounded narrative prose).
  • Text to (Presentation) - SynthPub synthesize publishing content, formats, web, social media, and presentations (Multi-format layout generation, slide structuring, and cross-channel content adaptation).
  • Script (Text) Generation - Create AI synthesize test data, software code, scripts (Repository-aware code synthesis, synthetic training data generation, and automated unit-test authoring paired with compiler/execution verifiers).
  • Speech to Text - Speech End-to-End Speech, Natural Language Processing (grammar, syntax and semantics), transcripts ... Speech Recognition (Conformer and Whisper-style encoder-decoder transformers mapping log-mel spectrograms directly to punctuated, multilingual text transcripts).
  • (Speech to) Text to Process to Text (to Speech) - Chatbots, Virtual Assistants, Language Translation, Conversation Modeling, Voice recognition (Both cascaded ASR-LLM-TTS pipelines and native speech-to-speech multimodal transformers for low-latency dialogue and live interpretation).
  • (Speech to) Text to Process to Text (to Speech) - Your Personal Companion (Long-term memory-augmented conversational agents with persistent persona state and emotional prosody synthesis).
  • (Speech to) Data - obtaining Voice Vectors to construct 3D image of face, income level, health (Speaker diarization and acoustic biomarker extraction via x-vectors and ECAPA-TDNNs; note that while vocal biomarkers have clinical utility for screening respiratory or neurodegenerative conditions like Parkinson's, claims of inferring facial bone structure or socioeconomic income from voice carry severe confounding biases and low ecological validity).
  • (Speech to) Speech - Video audio dubbing/translation (Cross-lingual voice conversion that translates spoken dialogue into a target language while preserving the original speaker's timbre, cadence, and lip-sync alignment).
  • Sensor Signals/Speech/Video to Lie Detection - Micro-expression changes in the eyes, voice, posture, etc. determine truthfulness (Scientific Qualification: While laboratory classifiers have attempted to correlate pupil dilation, vocal pitch, and facial micro-expressions with deception, peer-reviewed consensus from the National Research Council and experimental psychology demonstrates that nonverbal arousal cues do not reliably distinguish deception from anxiety or cognitive load, performing near chance in real-world settings and facing strict prohibitions under the EU AI Act).
  • Image to Classification - Colorize, Image Classification (Assigning whole-image semantic labels or predicting chrominance channels $(a^*, b^*)$ from grayscale luminance $L^*$ using CNNs and Vision Transformers).
  • Audio to Classification (predefined classes) - SoundHound, genre (Acoustic fingerprinting via constellation-map hashing and spectrogram classification for song identification and musical genre tagging).
  • Sound to Classification (predefined classes) - Signal Processing, genre (Environmental audio event detection, industrial bearing-fault acoustic classification, and sonar/radar signature identification).
  • Video to Classification (predefined classes) - Computer Vision identification, Screening; Passenger, Luggage, & Cargo Image classification, Video analysis (3D convolutional and spatiotemporal transformer models for human action recognition, X-ray/CT threat screening, and industrial quality control).
  • Sound to Cluster - grouping related items together e.g. Bird Identification (Bioacoustic spectrogram embedding and unsupervised clustering to catalog avian and marine species vocalizations).
  • Audio to Cluster - grouping related songs together e.g. Spotify, Pandora (Audio representation learning combined with collaborative-filtering graph embeddings for music discovery and playlist generation).
  • Sound/Audio to Abnormality detection - Signal Processing, Transmission troubleshooting, Electronic warfare (Detecting anomalous RF spectrum emissions, acoustic pipeline leaks, and mechanical cavitation in real time).
  • Audio Generation - Synthesize songs, Make music (Multi-track musical arrangement, vocal synthesis, and instrumental waveform generation via latent audio diffusion and autoregressive token models).
  • Audio to Video - Facial Animate (Audio-driven 3D facial mesh animation and lip-synchronized talking-head video synthesis conditioned on phoneme timing and prosody).
  • Image to Video - Synthesize video by animating a photo, e.g. family photo (Conditioning a spatiotemporal video diffusion model on a static reference anchor frame to synthesize plausible motion and camera parallax).
  • Image to Abnormality detection - retina screening, cancer detection, Pathology (High-sensitivity medical imaging triage across diabetic retinopathy fundus photographs, mammography, and gigapixel whole-slide histopathology scans).
  • Image to Text (Classification) Image classification (Dense visual-to-textual tagging and natural-language scene description).
  • Image/Video to Image/Video - Colorize B/W movies and photos (Temporally consistent chrominance restoration across historical film frames using reference-guided diffusion and optical-flow warping).
  • Image/Video to Image/Video - Image Enhancement, age regression (Super-resolution upscaling, motion deblurring, low-light denoising, and latent facial attribute manipulation).
  • Image/Video to Segmentation - Recognize Object, Collision avoidance (Pixel-level semantic, instance, and panoptic segmentation using architectures such as Mask R-CNN, YOLO, and Segment Anything Model (SAM 2) for autonomous navigation).
  • Image to Text - Digit recognition, Handwriting recognition, Optical Character Recognition (OCR) (Document layout analysis, scene-text detection, and vision-language transcription of printed and cursive script).
  • Image to Image - Image Reconstruction, complete obstructed image (Context-aware semantic inpainting and outpainting of occluded or damaged image regions).
  • Image to Image - Photo restoration Generative Facial Prior-Generative Adversarial Network (GFP-GAN) (Blind face restoration combining pre-trained facial geometry priors with generative adversarial and diffusion refinement).
  • Image to Database Query - Recipes (Inverse cooking and visual search: extracting ingredient entities and preparation states from food photography to query structured recipe databases).
  • Image/Video to 3D Model/Video/Image (Photogrammetry, Neural Radiance Fields (NeRFs), and 3D Gaussian Splatting that reconstruct navigable 3D meshes and radiance fields from 2D multi-view images or monocular video).
  • Generate Image/Video - Art/Image/Actors generation from scratch, Fake (Unconditional and prompt-conditioned synthesis of photorealistic synthetic humans, digital avatars, and deepfakes; requires cryptographic provenance standards such as C2PA watermarking for verification).
  • Video to Numeric - Count Customers (Multi-object tracking, crowd-density map estimation, and retail foot-traffic analytics across camera feeds).
  • Video to Text - Captioning, Real-time sign language (Dense video event captioning and continuous sign-language gloss/sentence translation from skeletal and hand-pose trajectories).
  • Video to Text - Video Intelligence, scenes are classified, searchable, & summarized (Shot-boundary detection, temporal action localization, and natural-language indexing of long-form video archives).
  • Video to Video (Hologram) - Immersive Reality (Volumetric video capture, real-time neural relighting, and light-field rendering for spatial computing displays).
  • Video to Database Query - Immersive Reality (Spatial localization, visual SLAM, and real-time object querying linking live camera feeds to enterprise Digital Twin databases).
  • Video to Prediction - Predict image next movie frames (Autoregressive and diffusion world models that predict future visual frames conditioned on current observations and agent action vectors).
  • Video to Database to Video - Personal information preservation with Transhumanism (Indexing multimodal life-logging archives to build interactive, voice-and-video-cloned biographical avatars).

Architectural & Benchmark Summary Across Capability Families

Technical Comparison of Core AI Capability Families (2026 State of the Art)
Capability Family Modality Mapping ($\mathcal{X} \to \mathcal{Y}$) Dominant Neural Architecture Primary Evaluation Benchmarks 2026 Performance & Key Bottleneck
Tabular & Time-Series Prediction Numeric $\to$ Numeric / Class Gradient-Boosted Decision Trees (XGBoost, CatBoost), TabPFN, Temporal Fusion Transformers UCI Tabular, Monash Time Series, Kaggle Benchmarks High production reliability and sub-millisecond latency; bottlenecked by data drift, missing causal confounders, and schema shifts.
Language, Code & Symbolic Reasoning Text $\to$ Text / Code / SQL Dense & Mixture-of-Experts (MoE) Transformers with RL Test-Time Compute HELM, SWE-bench Verified, Humanity's Last Exam, ARC-AGI-2 Near 100% on SWE-bench Verified; 75% to 85% on ARC-AGI-2; bottlenecked by hallucination under false presuppositions and verification cost.
Speech & Acoustic Processing Audio/Speech $\leftrightarrow$ Text / Speech Conformer ASR, Neural Audio Codecs (RVQ-VAE), Native Multimodal Transformers LibriSpeech, FLEURS, AudioSet, SUPERB Near-human transcription in high-resource languages; native speech-to-speech preserves prosody; bottlenecked by overlapping cocktail-party noise and low-resource dialects.
Computer Vision & 2D/3D Perception Image/Video $\to$ Class / Mask / 3D Vision Transformers (ViT), SAM 2, 3D Gaussian Splatting, Diffusion World Models (Veo 3) ImageNet, COCO, ClockBench, NuScenes Superhuman on medical triage and object segmentation; emergent zero-shot physics in video models; brittle on fine-grained spatial counting (50.6% on ClockBench).
Multimodal Media Synthesis Text/Image/Audio $\to$ Image / Video / 3D Diffusion Transformers (DiT), Rectified Flow Matching, Autoregressive Tokenizers GenEval, VBench, FVD (Fréchet Video Distance), Human Arena Photorealistic 1080p/4K synthesis with synchronized audio; bottlenecked by $\mathcal{O}((T \cdot H \cdot W)^2)$ attention compute and long-horizon physical consistency.
Interactive Computer & Embodied Agents Multimodal State $\to$ Action Sequence Vision-Language-Action (VLA) Models, Hierarchical RL, Neuro-Symbolic Planners OSWorld (Computer Use), RLBench (Sim), ARC-AGI-3 (Agentic) 66.3% on OSWorld digital tasks; 89.4% in simulated robotics, but only 12% in real household robotics and <1% on ARC-AGI-3 novel environment efficiency.

More

Additional Viewing

Load video
YouTube

In this 64-minute colloquium lecture at UC Berkeley EECS (September 2026, YouTube link), Stanford Professor Percy Liang (Director of the Center for Research on Foundation Models and leader of the Marin open-frontier project) examines the engineering mechanics required to build and predict frontier AI capabilities in the open. He details how parametric scaling recipes (incorporating QK-normalization, Frobenius-norm Hyperball optimization, and Mixture-of-Experts quantile load balancing) allow researchers to pre-register and extrapolate model losses across a 300x increase in compute up to 535-billion-parameter architectures, and explains how LLM-curated data deduplication and decontamination over 30 trillion tokens prevent overfitting.

Load video
YouTube

In this 162-minute technical discussion on Machine Learning Street Talk (November 2024, YouTube link), François Chollet (creator of Keras and the Abstraction and Reasoning Corpus, ARC-AGI) contrasts continuous pattern recognition ("System 1" value-centric curve fitting on high-dimensional manifolds) against true general intelligence ("System 2" discrete program synthesis and adaptation to novelty). Chollet explains his "Kaleidoscope Hypothesis" of abstraction mining, analyzes why static exam benchmarks measure memorized skill rather than fluid intelligence, and evaluates neuro-symbolic test-time training and program-search architectures.

Load video
YouTube

In this 64-minute presentation and interview on Vanishing Gradients (May 2025, YouTube link), Greg Kamradt (President of the ARC Prize Foundation) details the empirical methodology behind benchmarking frontier reasoning models on ARC-AGI-1 and ARC-AGI-2. He walks through the December 2024 verification of OpenAI's o3-preview model (contrasting low-compute vs. 170x high-compute inference costs), explains how first-party human calibration testing with over 400 participants in San Diego shaped ARC-AGI-2, and discusses why efficiency (energy and sample complexity) must be paired with raw accuracy when measuring AI capabilities.

Load video
YouTube

In this 21-minute keynote address at the GenAI Summit (June 2023, YouTube link), Stanford Professor Percy Liang introduces the Holistic Evaluation of Language Models (HELM) framework developed at Stanford CRFM. He explains why single-metric accuracy leaderboards fail to capture real-world deployment readiness, demonstrating how systematic multi-metric evaluation across accuracy, confidence calibration, robustness to perturbations, fairness, toxicity, and citation faithfulness (where early generative search engines supported their own citations only 74% of the time) guides safer foundation model engineering.

Archival Seminars on Early Deep Learning and Predictive ML Capabilities (2015 to 2017)

Load video
YouTube

Produced by DeepLearning.TV (December 2015, 10 minutes, YouTube link), this historical overview illustrates the first wave of commercial deep learning capabilities (Software 2.0), demonstrating convolutional neural networks for image tagging and object recognition alongside early recurrent neural networks for Twitter sentiment analysis, medical radiology triage, and financial fraud detection.

Load video
YouTube

In this 30-minute HackerEarth engineering webinar (August 2017, YouTube link), enterprise analytics practitioners walk through a classical machine learning pipeline for unstructured corporate email classification and inbox triage, illustrating stop-word cleansing, principal component analysis (PCA), $k$-means clustering, and random forest classification prior to the transformer era.

Load video
YouTube

Presented by Dr. Michał Kudelski at PyData Warsaw (November 2017, 31 minutes, YouTube link), this talk examines three production industrial machine learning deployments: edge-analytics predictive maintenance across a fleet of 40,000 commercial trucks (each streaming 60+ sensor channels to reduce warranty downtime by up to 30%), real-time hybrid rule-and-neural fraud scoring in retail banking, and deep autoencoder feature compression for telecommunications churn prediction.

Load video
YouTube

Mechanisms

Executing any capability in the cross-modal input-output matrix relies on a four-stage computational pipeline that transforms raw physical or digital signals into verified outputs:

  1. Modality Tokenization and Feature Projection (Input Encoding): Because neural networks operate exclusively on tensors of floating-point numbers, raw inputs must first be converted into sequences of dense vector embeddings $\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_L \in \mathbb{R}^d$:
    • Numeric and Tabular Data: Continuous measurements and categorical attributes are normalized, imputed, and projected via linear layers or split-point decision trees to support Regression, Classification, and Anomaly Detection.
    • Text and Symbolic Code: Raw character streams are segmented into subword units via Byte-Pair Encoding (BPE) or unigram tokenizers (typically vocabularies of 32,000 to 256,000 tokens) and mapped to learned embedding vectors with rotary position embeddings (RoPE).
    • Sound, Audio, and Speech: Continuous 1D acoustic waveforms sampled at 16 kHz to 48 kHz are transformed either into 2D time-frequency log-mel spectrograms via Short-Time Fourier Transforms (STFT) or compressed directly into discrete acoustic and semantic tokens using residual vector-quantized variational autoencoders (RVQ-VAEs such as SoundStream or EnCodec). This preserves pitch, timbre, prosody, and ambient acoustic cues for End-to-End Speech and Signal Processing.
    • Images, Video, and 3D Geometry: Following the Vision Transformer (ViT) mechanism, 2D images of resolution $H \times W \times C$ are sliced into non-overlapping patches (such as $16 \times 16$ pixels) and flattened into linear tokens. Continuous video streams add a temporal dimension $T$, slicing video into 3D spatiotemporal "tubes" ($t \times h \times w$), while 3D scenes are encoded as point clouds, voxel grids, signed distance functions (SDFs), or 3D Gaussian splats to build a 3D Model.
  2. Latent Manifold Transformation and Cross-Modal Alignment: Once tokenized, inputs are processed by a deep neural backbone (most commonly a dense or Mixture-of-Experts Transformer, a state-space model such as Mamba, or a hybrid convolutional-attention network). Cross-modal capabilities emerge through two mechanisms:
    • Contrastive Dual-Encoder Alignment: Models such as CLIP, SigLIP, and CLAP train separate encoders for two modalities (e.g., Image and Text, or Audio and Text) to maximize the cosine similarity of paired examples in a shared vector space while pushing unmatched pairs apart. This powers zero-shot Image Classification, image retrieval, and Recommendation clustering.
    • Native Early-Fusion Multimodal Attention: Modern any-to-any models interleave tokens from all modalities into a single sequence. Through cross-attention and self-attention layers, a text token can attend directly to a specific pixel patch in an image frame or a specific millisecond of a vocal waveform, allowing unified reasoning across text, vision, and audio.
  3. Discriminative, Autoregressive, and Denoising Decoding (Output Generation): The latent representation is routed to an output head matched to the target capability row:
    • Discriminative and Predictive Heads: For Numeric, Classification, Cluster, and Anomaly Detection tasks, a linear projection or distance metric outputs a scalar value ($\hat{y} \in \mathbb{R}$), a categorical probability distribution via the softmax function, or an out-of-distribution Mahalanobis/reconstruction error score.
    • Autoregressive Sequence Decoding: For Text, Autocomplete, Headlines, and Natural Language Generation (NLG), the decoder predicts the probability distribution of the next token $p_\theta(y_t \mid y_{<t}, \mathbf{x})$ step-by-step.
    • Iterative Denoising Diffusion and Flow Matching: For high-dimensional continuous outputs such as Generated Image, Video Synthesis, Reconstruction, and Synthesize Speech, predicting individual pixels or audio samples autoregressively is computationally slow and prone to artifacts. Instead, Diffusion Transformers (DiT) and Rectified Flow models start with random Gaussian noise $\mathbf{y}_1 \sim \mathcal{N}(0, \mathbf{I})$ and iteratively integrate a learned velocity vector field $v_\theta(\mathbf{y}_t, t, \mathbf{x})$ backward to a clean, high-resolution image, video, or audio waveform $\mathbf{y}_0$.
  4. Value-Centric Interpolation vs. Program-Centric Reasoning (System 1 vs. System 2): As formalized by François Chollet, standard deep neural networks fit smooth, high-dimensional parametric curves via stochastic gradient descent. This provides value-centric abstraction: embedding inputs onto a continuous manifold where perceptual similarity corresponds to Euclidean or cosine distance, enabling rapid, intuitive "System 1" interpolation (such as Style Transfer or summarizing familiar text). However, continuous curve fitting struggles with exact, discrete algorithmic composition (such as multi-step mathematical deduction, sorting arbitrary out-of-distribution sequences, or solving novel logical puzzles). To achieve program-centric abstraction ("System 2" deliberate reasoning), modern architectures pair neural intuition engines with inference-time search, reinforcement-learning-trained chain-of-thought backtracking, code interpreters, and discrete program synthesis verifiers.

Why This Matters

A rigorous, scientifically grounded map of AI capabilities matters for four practical reasons:

  • Matching Problem Types to the Minimum Viable Architecture: When engineers understand the input-output capability matrix, they avoid using expensive, hallucination-prone generative models for tasks that are mathematically better served by classical predictive algorithms. Predicting equipment failure from tabular vibration sensors (Numeric $\to$ Numeric/Anomaly Detection) or grouping retail customers by purchase history (Numeric $\to$ Cluster) can be executed with sub-millisecond latency and full explainability using gradient-boosted trees, random forests, or deep autoencoders rather than a 500-billion-parameter language model.
  • Navigating the "Jagged Intelligence" Paradox: Modern AI capabilities exhibit extreme unevenness across tasks that humans assume require similar intelligence. A single frontier model in 2025/2026 can win a gold medal on proof-based problems at the International Mathematical Olympiad and score near 100% on repository-scale software bug fixing, yet fail nearly 50% of the time when asked to read the hands of an analog clock or fail to solve a 2D visual symmetry puzzle that a four-year-old child solves in seconds. Understanding why this happens (high training-set density for text/math templates versus poor continuous-curve representation of exact spatial geometry and novel rules) prevents catastrophic over-trust in high-stakes deployments.
  • Distinguishing Validated Capabilities from Pseudo-Scientific Claims: Not every capability proposed in commercial marketing or historical taxonomies is scientifically valid. Capabilities such as optical Character Recognition, Speech Recognition, and Pathology cellular segmentation work reliably because the target label is physically present in the input pixels or audio waveform ($I(X; Y)$ is high). Conversely, capabilities such as automated facial micro-expression Lie Detection or inferring a speaker's income, honesty, or character from Voice Vectors fail in real-world deployment because internal deception and socio-economic traits do not produce invariant biological signatures in facial muscles or vocal cords. Recognizing this boundary protects organizations from deploying discriminatory, legally prohibited pseudoscience.
  • Guiding the Transition from Static Chatbots to Embodied and Agentic Systems: As text and code benchmarks saturate, the capability frontier is shifting toward multimodal world modeling (video models that implicitly learn physical buoyancy, optics, and object permanence) and interactive environment navigation.
Retrieved from "https://primo.ai/w/index.php?title=Capabilities&oldid=46697"