Representation Learning

From PRIMO.ai
Jump to navigation Jump to search

YouTube search... ...Google search

Representation learning is the part of machine learning concerned with turning raw data, such as pixels, sound waves, words, molecules, or the links in a network, into numerical descriptions that make later tasks easy. Those descriptions are usually vectors called embeddings or features, and a good one places similar things near each other, separates things that matter, and discards details that do not. Every modern foundation model is, at its core, a representation learner: a large language model builds contextual representations of words, a vision model builds representations of images, and a multimodal model aligns the two in a shared space. Representation learning is so central to the field that its main research conference, the International Conference on Learning Representations (ICLR), founded by Yoshua Bengio and Yann LeCun in 2013, is named after it.

Why representations matter was laid out in an influential review by Yoshua Bengio, Aaron Courville, and Pascal Vincent, Representation Learning: A Review and New Perspectives | Y. Bengio, A. Courville, and P. Vincent. They argued that machine learning succeeds or fails largely because of how data are represented, since a poor representation tangles together and hides the underlying factors that explain the data, such as an object's identity, pose, and lighting in a photograph. Domain experts can design representations by hand, but the review argued that learning them with general-purpose assumptions, which the authors called priors, scales better and is essential to the long-term goal of AI. Those priors include the expectations that the world has several independent explanatory factors, that concepts are organized hierarchically, that useful factors are shared across tasks, and that natural data lie near low-dimensional surfaces. The review surveyed probabilistic models, autoencoders, manifold learning, and deep networks, and it closed with questions that remain open today: what objective produces good representations, how to compute a representation for a new input efficiently, and how representation learning relates to density estimation and to the geometry of data manifolds. Two closely related ideas are defined below.

  • Feature Learning or representation learning is a set of techniques that allows a system to automatically discover the representations needed for feature detection or classification from raw data. This replaces manual feature engineering and allows a machine to both learn the features and use them to perform a specific task. Feature learning is motivated by the fact that machine learning tasks such as classification often require input that is mathematically and computationally convenient to process. However, real-world data such as images, video, and sensor data has not yielded to attempts to algorithmically define specific features. An alternative is to discover such features or representations through examination, without relying on explicit algorithms. Wikipedia
  • Manifold Learning is an approach to non-linear dimensionality reduction. Algorithms for this task are based on the idea that the dimensionality of many data sets is only artificially high. Manifold learning | SciKitLearn

The field's history is a steady shift of work from people to algorithms. Before about 2010, computer vision relied on hand-built descriptors of edges and textures, and speech recognition on hand-designed frequency features, while statistics offered linear methods such as principal component analysis. In 2006, Geoffrey Hinton and colleagues showed that stacks of restricted Boltzmann machines could be trained one layer at a time into a deep belief network, and stacked and denoising autoencoders followed; the 2012 ImageNet breakthrough then showed that features learned by a convolutional network outclassed hand-crafted ones. In language, Word2Vec (2013) and GloVe (2014) learned word vectors from co-occurrence, ELMo and BERT (2018) made those vectors depend on context, and the GPT series showed that predicting the next word builds representations rich enough for almost any language task. From 2020 onward, self-supervised methods such as SimCLR, DINO, and masked autoencoders, together with image-text models such as CLIP, made it possible to learn strong visual representations without human labels, and by 2025 Meta's DINOv3 matched weakly supervised models on standard image benchmarks.

Representation learning differs from conventional machine learning in where the knowledge lives. In a classical pipeline, an engineer decides which measurements to extract and a relatively simple model learns from them, so performance is capped by the engineer's choices. In representation learning, the model learns the measurements themselves, usually from large amounts of unlabeled data, and the resulting features can be reused across many tasks through transfer learning, often with only a small labeled dataset or a simple linear classifier on top. The price is that learned features can be opaque, can encode unwanted shortcuts or biases from the data, and can fail when the world shifts, which is why interpretability and robustness are now part of the field's core agenda.

The illustration above, from Lilian Weng's survey of self-supervised learning, shows Grasp2Vec, a Google robotics method in which a robot learns object representations from its own grasping attempts, without labels. Because the embedding of a scene before a grasp minus the embedding after the grasp should match the embedding of the object removed, the robot's own actions provide the training signal. It is a compact example of the central idea of this page: the right objective lets structure in the world teach a model how to represent that world.



A good representation makes the important things easy to see and the unimportant things invisible.
The bet of representation learning is that machines can discover such representations from data alone; the open question is whether the representations they find reflect how the world is actually organized or only what is useful for the tasks we measure.



The Mechanism

Almost every representation learner has the same skeleton: an encoder fθ maps an input x to a vector z = fθ(x), and an objective decides what makes z good. What distinguishes the methods is that objective, and most fall into a handful of families. The list below runs roughly from the oldest to the newest, though modern systems often combine several.

  1. Supervised pretraining. Train a network to classify labeled data, then reuse its inner layers as general-purpose features. ImageNet-trained networks made this the standard approach of the 2010s, and image transfer learning still relies on it.
  2. Reconstruction. An autoencoder compresses the input into a small code and rebuilds it; denoising autoencoders rebuild a clean input from a corrupted one, and a variational autoencoder makes the code a probability distribution. Masked modeling, used by BERT for text and by masked autoencoders for images, hides part of the input and asks the model to fill it in.
  3. Predicting what comes next or nearby. Word2Vec learns word vectors by predicting a word's neighbors (Skip-Gram) or a word from its neighbors (Continuous Bag-of-Words (CBoW)), and GPT-style models learn by predicting the next token, which turns out to require building rich internal representations of meaning.
  4. Contrastive learning. Two augmented views of the same example, such as two crops of one photograph, should map to nearby vectors, while views of different examples should map far apart. A common loss, InfoNCE, is L = -log [ exp(sim(zi, zj)/τ) / Σk exp(sim(zi, zk)/τ) ], where sim is cosine similarity and τ is a temperature, so the model must pick out the matching view from many distractors.
  5. Non-contrastive joint embedding and self-distillation. Methods such as BYOL, Barlow Twins, VICReg, and Meta's DINO family drop the negative examples and instead match a student network to a slowly updated teacher, or regularize the statistics of the embeddings. Their main danger is collapse, in which the encoder maps everything to the same point or to a thin subspace.
  6. Joint-embedding predictive architectures (JEPAs). Yann LeCun's proposal predicts the representation of a hidden part of the input, such as a masked image region or a future video frame, from the representation of the visible part, rather than predicting raw pixels. Because unpredictable details like the motion of individual leaves are never modeled, JEPAs aim to capture what is predictable and therefore meaningful, and the 2025 LeJEPA method showed that a single statistical regularizer can prevent collapse without the usual engineering tricks.
  7. Multimodal alignment. CLIP and its successors train image and text encoders so that matching pairs land close together, producing a shared space that supports search, zero-shot classification, and the vision side of multimodal assistants; ImageBind extends the idea to six modalities and BLIP-2 connects frozen image encoders to language models.
  8. Graph embeddings. Node2Vec and related methods turn random walks over a network into "sentences" and apply word2vec-style learning, while graph neural networks compute a node's representation by repeatedly aggregating information from its neighbors; see Large-Scale Graph.

How representations are judged. Because a representation is meant to serve many tasks, it is evaluated indirectly: researchers freeze the encoder and train a simple linear classifier on top (a "linear probe"), check nearest-neighbor retrieval, or measure how well the features transfer after fine-tuning on new datasets. Dense tasks such as segmentation and depth estimation test whether local features stay sharp, which became a key concern for large self-supervised vision models. These evaluations have their own blind spots, since a representation can score well on a benchmark by exploiting shortcuts in the data, so robustness under distribution shift is increasingly reported alongside accuracy.

The Geometry of Representations

A learned representation is a point in a high-dimensional space, and much of what makes it useful is geometric. The manifold hypothesis holds that real data, such as natural images, occupy a thin, curved surface inside that space, which is why methods like Isomap, locally linear embedding, and t-SNE can unfold or visualize them in two or three dimensions. Similarity is usually measured with the dot product or cosine similarity, so finding related items becomes a nearest-neighbor search, which approximate nearest neighbor indexes such as HNSW make fast enough for semantic search and retrieval-augmented generation. Closely related ideas appear in metric learning, which trains a representation so that distance reflects a chosen notion of similarity.

A striking regularity is that many concepts behave like directions. The word2vec observation that vector("king") - vector("man") + vector("woman") lands near vector("queen") was an early sign, and research on language models has formalized this as the linear representation hypothesis: high-level concepts such as sentiment, truthfulness, or a language correspond to directions in activation space that can be read with a linear probe and sometimes changed by adding a "steering vector." Networks appear to store far more concepts than they have dimensions by packing them into nearly perpendicular directions, a phenomenon Anthropic researchers called superposition, which explains why individual neurons often respond to several unrelated ideas. Sparse autoencoders try to undo superposition by rewriting each activation as a sparse combination of many learned feature directions, and in 2024 Anthropic used them to extract millions of interpretable features from a production model, including one for the Golden Gate Bridge that could be amplified to change the model's behavior.

Researchers have long hoped for disentangled representations in which each coordinate captures one independent factor, such as size or color. In 2019, Francesco Locatello and colleagues proved that this cannot be achieved without assumptions built into the model or data, because many different representations explain the same observations equally well, which pushed the field toward weak supervision and toward causal representation learning, whose goal is to recover the variables that cause the data rather than merely correlate with it (see Causation vs. Correlation). Practical geometry matters too: Matryoshka representation learning trains embeddings so that their first few dimensions already form a usable smaller embedding, letting systems trade accuracy for storage, and the idea has been adopted in commercial embedding services. Finally, high-dimensional geometry can mislead, because distances between random points concentrate around similar values, a fact that became central to the 2026 debate over whether different models converge on the same representation.

Why This Matters

  • Scientific importance. Representation learning explains much of deep learning's success: depth helps because each layer builds more abstract features from the layer below, and pretraining helps because it learns those features once for many tasks. Understanding what representations contain is also the foundation of interpretability research and of efforts to make models safe.
  • Engineering implications. Nearly every modern AI product is built on pretrained representations, from image search and recommendation to retrieval-augmented chat systems, and choosing or training the right embedding model is now a routine design decision. Strong frozen backbones, such as self-supervised vision models, let small teams build accurate systems with few labels.
  • Practical consequences. Embeddings from different models are not interchangeable, even when the models score similarly, so mixing them in one vector index or swapping models without re-embedding data causes silent failures. Learned features also absorb biases and shortcuts present in the training data, which makes data curation as important as model design.
  • Applications beyond text and images. Self-supervised representations of speech (such as wav2vec 2.0), proteins (protein language models used in structure prediction), medical images, satellite data, and physical simulations now let scientists apply AI where labels are scarce.
  • Conceptual implications. If very different models converge on similar representations, that would suggest learned representations reflect structure in the world rather than quirks of architecture; if they do not, representation choices remain a matter of design and data. The answer bears on questions as different as model interoperability, neuroscience comparisons between brains and networks, and how far scaling alone can go.
  • Limitations and uncertainty. There is still no agreed theory of what makes a representation good in general, collapse and shortcut learning remain practical hazards, and interpretability tools such as sparse autoencoders have shown real limits on downstream tasks. Many claims about representations rest on similarity measures that recent work shows can be inflated by model size and dimensionality.

Recent Developments

Representation learning has been one of the most active areas of AI research in 2025 and 2026, driven by large self-supervised vision and video models, a revival of interest in world models, a lively debate about whether models converge, and hard questions about interpretability tools. The table summarizes notable developments and their evidence status, and the subsections below explain them.

Date Development What was shown Evidence status
May 2024 Platonic Representation Hypothesis (MIT) Proposed that increasingly capable models converge toward a shared statistical model of reality Position paper (ICML 2024); widely debated
May 2024 Scaling Monosemanticity (Anthropic) Sparse autoencoders extracted millions of interpretable features from a production language model Published research report
Jun 2025 V-JEPA 2 (Meta) A video model trained by predicting in representation space supported zero-shot robot planning with little robot data Preprint and open release
Aug 2025 DINOv3 (Meta) Self-supervised vision scaled to a 7-billion-parameter teacher and about 1.7 billion images, with "Gram anchoring" to keep dense features sharp Technical report; open weights
Nov 2025 LeJEPA (Balestriero and LeCun) A single regularizer, SIGReg, prevents collapse; 79 percent ImageNet linear-probe accuracy with a ViT-H/14 Preprint with code
Dec 2025 Converging representations of matter Scientific foundation models align on familiar data but collapse to low-information representations far outside it Preprint
Mar 2026 AMI Labs launches Yann LeCun's new company raised $1.03 billion to build world models that predict in representation space Funding announcement
Mar 2026 LeWorldModel A stable end-to-end JEPA world model trained directly from pixels Preprint
Mar 2026 Limits of probes and SAEs Linear probes and sparse autoencoders shown to fail at compositional generalization in controlled settings Preprint
Jun 2026 V-JEPA 2.1 (Meta) Dense, spatially precise video representations alongside strong global understanding Preprint
Jul 2026 "Aristotelian" view of convergence (EPFL) After correcting similarity measures, global convergence largely disappears while local neighborhood agreement survives Peer reviewed (ICML 2026)
Sep 2026 "Structure over Geometry" Relational structure converges across vision-language models, but exact distances do not Preprint

Self-Supervised Vision and the Return of World Models

Self-supervised vision reached a milestone in August 2025 with Meta's DINOv3, which trained a 7-billion-parameter teacher network on about 1.7 billion curated images and then distilled it into a family of smaller Vision Transformer and ConvNeXt models. Its main technical novelty, Gram anchoring, addressed a long-standing problem: as self-supervised training runs grow longer, the local patch features needed for segmentation and depth estimation tend to blur, and Gram anchoring keeps the similarity structure between patches stable. The result was a single frozen backbone competitive with weakly supervised models such as CLIP on classification while being much stronger on dense tasks, with variants trained on satellite imagery as well. In June 2026, V-JEPA 2.1 pushed similar gains into video, improving dense features while keeping strong scene-level understanding, although DINOv3 still led on some tasks such as video object tracking.

The bigger story is the revival of world models built from learned representations. Yann LeCun has argued for years that generative models waste effort predicting every pixel and that intelligent systems should instead predict the future in an abstract representation space, which is the idea behind JEPAs. Meta's V-JEPA 2 (2025) learned from more than a million hours of internet video and then, with a small amount of robot data, let a robot arm plan simple manipulation tasks in new environments, and in November 2025 LeCun and Randall Balestriero's LeJEPA replaced a pile of heuristics with one theoretically motivated regularizer that pushes embeddings toward an isotropic Gaussian distribution. LeCun then left Meta and in March 2026 launched AMI Labs in Paris with a $1.03 billion seed round to build such world models for robotics, embodied AI, and healthcare. Academic work in 2026, including LeWorldModel and a stream of JEPA variants for driving and robotics, shows the approach is spreading, but no JEPA-based system has yet demonstrated general-purpose planning in the open world, and the comparison with large generative video models remains unsettled.

Do Different Models Learn the Same Representation?

In 2024, MIT's Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola proposed the Platonic Representation Hypothesis: as models grow more capable, their internal representations become more alike, even across different architectures, training data, and modalities such as vision and language, because they are all approximating the same underlying statistical structure of the world. They supported the claim with evidence that larger vision and language models measure similarity between data points in increasingly similar ways, and they named the idea after Plato's allegory of the cave, in which different shadows reflect one reality. The hypothesis has practical appeal, since convergent representations would make models easier to connect, translate between, and compare, and it has inspired a wave of follow-up work across science and engineering.

The evidence has since been tested hard. A December 2025 study of scientific foundation models for molecules and materials found that stronger models did align on data resembling their training sets, but that most models collapsed to low-information representations on data far outside it, so convergence was real but bounded. In 2026, Fabian Gröger, Shuo Wen, and Maria Brbić of EPFL showed at ICML that common similarity scores rise with model size and dimensionality even for unrelated random models, because distances in very high-dimensional spaces concentrate; after correcting for this, much of the reported global convergence disappeared, while agreement about which items are each other's nearest neighbors survived. They proposed a narrower "Aristotelian" hypothesis of convergence in local neighborhood relationships, and a September 2026 follow-up reached a similar conclusion that relational structure converges while exact geometry does not. For practitioners, the lesson is that different embedding models may agree on which things are similar while still placing them at incompatible coordinates.

Reading Representations: Interpretability and Its Limits

Interpretability research treats a model's internal representations as an object of scientific study. After Anthropic's 2024 demonstration that sparse autoencoders could extract millions of human-interpretable features from a production model, the method spread quickly across industry and academia, and in 2025 Anthropic extended the approach with attribution graphs that trace how features interact across layers to produce a specific answer. The enthusiasm met hard evidence in 2025 and 2026. Google DeepMind's interpretability team reported that sparse autoencoder features underperformed simple linear probes on a practical downstream task and deprioritized the method, and a March 2026 study showed that both linear probes and sparse autoencoders fail at compositional generalization in controlled settings, arguing that the fault lies in their fixed, one-pass encoders rather than in the superposition hypothesis itself.

The emerging picture is more careful than either the early excitement or the backlash. Concepts often are represented as directions, and those directions can be used to monitor and steer models, but the features a sparse autoencoder finds depend on its training choices, many recovered features may have little causal effect on the model's behavior, and theory now quantifies how interference between packed features limits what any linear readout can recover. Interpretability research is therefore shifting toward causal tests, which check whether changing a representation changes behavior as predicted, and toward combining several tools rather than relying on one. These questions connect directly to explainable AI and to safety work on detecting deception or hidden objectives inside models.

Open Questions

  • What objective produces the best general-purpose representation: reconstruction, contrastive learning, next-token prediction, or prediction in representation space?
  • Do sufficiently capable models converge on shared representations, and if so, at what level: local neighborhoods, relational structure, or full geometry?
  • Can world models built on learned representations support reliable long-horizon planning in the physical world?
  • How can interpretability tools distinguish features a model actually uses from patterns that merely correlate with its activations?
  • Can representations be made robust to distribution shift and free of harmful shortcuts without large amounts of labeled data?

Additional Viewing

Yann LeCun | Self-Supervised Learning, JEPA, World Models, and the future of AI
In this lecture at Harvard's Center of Mathematical Sciences and Applications, delivered as part of a conference on the geometry of machine learning while he was still Meta's chief AI scientist, Yann LeCun argues that AI should learn by predicting in representation space rather than generating pixels. He explains why joint-embedding architectures need safeguards against collapse, discusses regularized methods such as VICReg and DINO, and shows how learned world models let robots plan, which makes the talk a direct companion to the JEPA item in The Mechanism and to the world-model discussion under Recent Developments.
Watch on YouTube

2024 Nobel Prize lectures in physics | John Hopfield and Geoffrey Hinton
The official Nobel lectures from Stockholm University in December 2024, in which Geoffrey Hinton explains Boltzmann machines and how stacking restricted Boltzmann machines to learn features layer by layer helped launch deep learning, after John Hopfield's lecture on energy-based associative memory. It is a first-hand account of the history summarized in the Introduction and in Representation Learning and Deep Learning.
Watch on YouTube

Attention in transformers, step-by-step | Deep Learning Chapter 6 | 3Blue1Brown
Grant Sanderson's roughly 26-minute visual explanation of how a Transformer turns each token's embedding into a context-aware representation through queries, keys, values, and attention, with concrete examples of directions in embedding space carrying meaning. It helps readers picture the prediction-based representations described in The Mechanism and the idea of concepts as directions in The Geometry of Representations.
Watch on YouTube

Miles Cranmer: The Next Great Scientific Theory is Hiding Inside a Neural Network | Simons Foundation
In this April 2024 Simons Foundation Presidential Lecture, Cambridge researcher Miles Cranmer argues that scientists should interpret the representations a neural network learns from data and distill them into equations with symbolic regression, and he introduces the Polymathic AI effort to build scientific foundation models. It shows representation learning used as a discovery tool and complements the interpretability discussion under Recent Developments.
Watch on YouTube

Representation Learning and Deep Learning

Deep learning can be understood as representation learning carried out in layers. Each layer of a neural network transforms the representation produced by the layer below, so early layers of a vision network detect edges and textures (local features), middle layers detect parts, and later layers respond to whole objects (deep features); language models show a similar progression from surface patterns to syntax to meaning. Bengio and others argued that this depth is efficient because it reuses simpler features to build complex ones, and that such representations are distributed, meaning each concept is spread across many units and each unit takes part in many concepts, which allows an enormous number of distinctions to be encoded compactly. Historically, the key enabling trick was unsupervised, layer-by-layer pretraining with restricted Boltzmann machines and autoencoders, which made deep networks trainable before better initialization, activation functions, and hardware made end-to-end training standard. The lectures below, given by Yoshua Bengio at UCLA's Institute for Pure and Applied Mathematics, cover this foundational period and the principles behind it.

Yoshua Bengio | Institute for Pure & Applied Mathematics (IPAM)

Self-Supervised

Self-supervised learning creates its own training signal from the structure of unlabeled data, which makes it the dominant way representations are learned today. The model solves a "pretext" task whose answer is already contained in the data, such as predicting a hidden word, filling in masked image patches, telling whether two crops come from the same photo, or predicting the representation of a future video frame, and the features it builds along the way transfer to tasks people care about. Pretraining a large language model on next-token prediction is the largest-scale example, while contrastive, self-distillation, and joint-embedding predictive methods dominate vision, and speech and protein models use similar recipes. The main design questions are which pretext task captures meaningful structure rather than superficial cues, and how to avoid collapse when the objective could be satisfied trivially.

Semi-Supervised

Semi-supervised learning combines a small labeled dataset with a large unlabeled one, and representation learning is what makes the unlabeled part useful. Classic techniques assign confident predictions as "pseudo-labels" on unlabeled data, or enforce consistency so that a model gives the same answer for an input and a perturbed version of it; methods such as FixMatch (2020) combined both ideas and reached strong accuracy on image benchmarks with only a handful of labels per class. Today the most common recipe is simpler: pretrain a representation with self-supervision, then fine-tune or train a small classifier with whatever labels exist, which is also the logic behind few-shot learning. The approach works best when the unlabeled data resemble the task data, and it can quietly amplify errors when early pseudo-labels are wrong.

Unsupervised

Unsupervised learning looks for structure in data without any labels or pretext tasks defined by people, and it is where representation learning began. Clustering methods such as K-Means group similar items, dimensionality reduction methods such as PCA, singular value decomposition, and self-organizing maps compress data while keeping its main variation, and topic models such as latent Dirichlet allocation discover themes in documents. Generative models, including variational autoencoders, generative adversarial networks, and diffusion models, also learn internal representations as a by-product of modeling the data distribution, and 2024 work on representation alignment showed that diffusion models train much faster when their internal features are aligned with those of a strong self-supervised encoder. The limit of purely unsupervised learning is that, as the Locatello result shows, data alone do not determine which representation is "right," so useful results always depend on assumptions built into the method.

Supervised Learning of Rules for Unsupervised

Most unsupervised methods optimize a stand-in objective, such as reconstruction error, and simply hope the resulting representation will help with real tasks. Luke Metz and colleagues at Google Brain turned the problem around by using meta-learning: an outer training loop adjusts the unsupervised learning rule itself, rewarding rules whose representations perform well on semi-supervised classification after training. They constrained the learned rule to be local to each neuron, loosely inspired by biology, and found that it produced useful features, sometimes beating hand-designed methods, and generalized to networks of different widths and depths, to inputs with shuffled dimensions, and even from images to text. The work showed that the objective of representation learning can itself be learned, an idea that continues in research on learned optimizers and on automatically discovered training objectives.

Large-Scale Graph

Many datasets are networks rather than grids or sequences: social networks, citation graphs, molecules, supply chains, and the knowledge graphs behind search engines. Graph representation learning maps each node, edge, or entire graph to a vector that captures its position and role. Early methods such as DeepWalk (2014) and Node2Vec (2016) ran random walks and applied word2vec-style training, knowledge graph embeddings such as TransE represented relations as translations between entity vectors, and graph neural networks learned representations by passing messages between neighbors, with methods such as GraphSAGE making the approach inductive so it could embed nodes never seen in training. Scale is the defining challenge: Pinterest's PinSage, developed with Jure Leskovec's Stanford group, applied graph convolutions to a graph with billions of nodes to power recommendations, which required sampling neighborhoods and distributing computation. Current research combines graph neural networks with Transformers, adds geometric symmetries for molecules and physical systems, and studies why message passing struggles to distinguish certain graph structures and to propagate information across long distances.