Transformer

From PRIMO.ai
Jump to navigation Jump to search

YouTube ... Quora ...Google search ...Google News ...Bing News


Transformers are Neural Network architectures that use self-attention to model context and long-range dependencies in sequences. They started in Natural Language Processing (NLP), where they power chatbots, translation, summarization, and Sentiment Analysis tools, and they now sit at the core of nearly every Large Language Model (LLM) and many foundation models for images, audio, video, code, proteins, and graphs. In full self-attention, every output position can draw on every input position (in GPT-style decoders, every earlier position), and the weights connecting them are computed on the fly from the content itself rather than fixed in advance. Google achieves state-of-the-art NLP performance with an enormous language model and data set | Kyle Wiggers - VentureBeat

Origin and architects. The Transformer was introduced in the 2017 paper Attention Is All You Need by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin, working at Google Brain and Google Research. As the authors framed it, the dominant sequence transduction models of the day were complex recurrent or convolutional networks in an Autoencoder (AE) / Encoder-Decoder configuration, and the best of them also connected encoder and decoder through an attention mechanism. They proposed a simpler architecture based solely on attention, dispensing with recurrence and convolutions entirely, and showed on two machine translation tasks that it delivered better quality while being more parallelizable and much faster to train. The large model reached 28.4 BLEU on WMT 2014 English-to-German and 41.8 BLEU on English-to-French after training for 3.5 days on eight GPUs. Attention Is All You Need | A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, L. Kaiser, and I. Polosukhin

The problem it solved. Before 2017, Recurrent Neural Network (RNN)s and Long Short-Term Memory (LSTM) networks read a sentence one token at a time, passing a hidden state forward. That design has two weaknesses. Training cannot be parallelized across the positions of a sequence, and information from early tokens has to survive many sequential steps to influence later ones. Attention had already been added to Sequence to Sequence (Seq2Seq) translation models (Bahdanau, Cho, and Bengio, 2014) so the decoder could look back at the whole source sentence. The Transformer's bold move was to make attention the only mechanism for mixing information between positions.

The central mechanism. Each token is represented by a vector. In every layer, each token produces a query (what it is looking for), a key (what it offers), and a value (what it will hand over). Comparing every query against every key produces attention weights, and each token receives a weighted mix of the values. A small per-token neural network (the feed-forward or MLP block) then processes the result. Stack dozens of these layers, connect them with residual connections that keep a running "stream" of information for each token, and you have a transformer.

The paradigm shift. Because all positions are processed at once during training, transformers map neatly onto GPU and TPU matrix hardware. That made it practical to pretrain on vastly more text, which led directly to Bidirectional Encoder Representations from Transformers (BERT), the Generative Pre-trained Transformer (GPT) series, T5 (Text-To-Text Transfer Transformer) models, and eventually ChatGPT and today's Generative AI systems. The deeper shift was methodological: instead of hand-designing task-specific architectures with built-in assumptions about language, the field started scaling one general architecture with data and compute, then adapting it.

How it differs from earlier approaches. The original paper compared per-layer cost and information flow for different layer types. Here n is the sequence length, d is the representation width, k is a convolution kernel width, and r is the size of a restricted attention window.

Layer type Compute per layer Sequential operations Maximum path length between two positions
Self-attention O(n² · d) O(1) O(1)
Recurrent (RNN, LSTM) O(n · d²) O(n) O(n)
Convolutional (CNN) O(k · n · d²) O(1) O(log_k(n))
Self-attention restricted to a window of size r O(r · n · d) O(1) O(n / r)

Self-attention connects any two positions in a single step, which makes long-range dependencies easier to learn. The price is a cost that grows with the square of the sequence length. That one trade-off drives most of the research described in Recent Developments below.

Three family branches.

  • Encoder-only models (BERT, RoBERTa, DistilBERT) use bidirectional attention over the whole input. They are strong at classification, retrieval, and producing embeddings.
  • Encoder-decoder models (the original Transformer, T5, Whisper) use an encoder to read the input and a decoder that generates output while attending to the encoder through cross-attention. They fit translation, summarization, and speech recognition naturally.
  • Decoder-only models (the GPT series, Llama, Qwen, DeepSeek, Gemma, and most other open-weight LLMs) use causal, left-to-right attention and are trained to predict the next token. This is the dominant design for LLMs as of 2026.

Status in 2026. Nine years on, the transformer is still the backbone of state-of-the-art language models. What has changed is the pile of refinements layered on top: mixture-of-experts feed-forward layers, attention variants that shrink memory use, sparse and compressed long-context attention, hybrids that mix in linear-attention layers, and even widened residual streams. As Sebastian Raschka put it in a May 2026 survey, transformers remain the status quo, while much of the qualitative improvement in models appears to come from data and training recipes rather than the block itself.


A transformer can be pictured as a team of identical mini neural networks, one assigned to each token (a word or word fragment) in the input, all sharing the same weights. Each token's information flows along a main pathway, the "residual stream" or "main artery." At every layer, two kinds of branches split off that artery, perform extra computation, and merge their results back in: an attention branch, where tokens exchange information, and a feed-forward branch, where each token refines what it now knows on its own. Small models stack about a dozen of these branch-and-merge layers; the largest current models stack around a hundred or more. During attention, each token broadcasts what it holds (its keys and values) and what it is looking for (its query), fetches matching information from other tokens (in GPT-style models, only from earlier ones), and adds it to its own stream. Cycle after cycle, each token's representation grows richer and more context-aware. When the information reaches the top of the stack, a final projection and a Softmax turn it into a probability distribution over possible next tokens. In short, a transformer is an iterative information-processing system: it refines every token in parallel while letting each token consult the others, which is why it captures context and nuance so well.


The Mechanism

The thesis above describes a transformer as a set of per-token streams that repeatedly branch, compute, and merge. This section walks through the same process with the real components, using a decoder-only (GPT-style) language model as the running example.

1. Tokens and embeddings

A tokenizer splits text into tokens, usually subword pieces produced by byte pair encoding (BPE) or a similar algorithm. Vocabularies typically range from about 32,000 to more than 200,000 entries. Each token ID selects a row of an embedding matrix, producing a vector of width d_model: 512 in the original Transformer, 12,288 in GPT-3, and 7,168 in DeepSeek-V3. At this stage the vector knows which token it is, but nothing about its neighbors. The word "mole" gets the same starting vector whether it appears next to "carbon dioxide" or "biopsy." Fixing that is attention's job.

2. The residual stream

Each token's vector then flows up through the layer stack along its residual stream. Every sublayer reads from the stream, computes something, and adds its output back:

x ← x + Attention(Norm(x))
x ← x + MLP(Norm(x))

Nothing is overwritten; contributions accumulate. Anthropic researchers popularized this view in A Mathematical Framework for Transformer Circuits (2021), describing the stream as a shared communication channel that every component reads from and writes to through linear projections. It is the "main artery" of the thesis callout. Two engineering details matter here. Modern models normalize before each sublayer (pre-normalization, usually RMSNorm) because it trains more stably at depth, and the additive structure gives gradients a direct path through stacks of 100 or more layers.

3. Attention: tokens exchange information

Attention is the only place where information moves between positions.

  • Projections. Three learned matrices turn each token's stream vector into a query q, a key k, and a value v, typically 64 to 128 numbers per head.
  • Scores. Each token's query is compared with every key using a dot product, scaled by 1/√d_k, and passed through a softmax so the weights for that token sum to 1.
  • Causal mask. In decoders, scores for future positions are set to negative infinity before the softmax, so a token can only look backward. This is what lets training process every position of a sequence at once without any position "peeking" at the answer.
  • Mixing. Each token receives the attention-weighted sum of the value vectors, passes it through an output projection, and adds it to its residual stream.
  • Multiple heads. The block runs h of these "heads" in parallel with separate projections: 8 in the original model, 96 in GPT-3. Heads specialize. Some track the previous token, some copy repeated patterns (so-called induction heads, closely tied to in-context learning), and some follow syntactic relationships.
  • Cross-attention. In encoder-decoder models, an additional attention layer takes queries from the decoder and keys and values from the encoder's output.

4. The feed-forward (MLP) block

After attention, each token passes independently through a small multilayer perceptron. It expands the vector to a wider hidden size (traditionally four times d_model), applies a nonlinearity (ReLU in 2017, usually a gated SwiGLU variant today), and projects back down. No information crosses positions here; this is where each token "thinks about" what it just gathered. In dense models, MLPs hold roughly two-thirds of the parameters. 3Blue1Brown's tally for GPT-3 puts the attention matrices at about 58 billion of its 175 billion parameters. Interpretability research suggests MLP layers store much of a model's factual knowledge.

Mixture of experts (MoE). Many current LLMs replace the single MLP with a large set of expert MLPs plus a lightweight router that sends each token to only a few of them. DeepSeek-V3 has 671 billion total parameters but activates about 37 billion per token; DeepSeek V4-Pro (2026) has 1.6 trillion total with 49 billion active. MoE decouples how much a model can store from how much computation each token costs.

5. Repeat, then predict

The attention-plus-MLP layer is repeated many times: about a dozen in small models, 96 in GPT-3, 61 in DeepSeek-V3, 126 in Llama 3.1 405B, and 108 in NVIDIA's hybrid Nemotron 3 Ultra. After the last layer, a final normalization and an unembedding matrix turn the top of the last token's stream into one score (logit) per vocabulary entry. A Softmax converts those scores into probabilities, and a sampling rule (temperature, top-p, and so on) picks the next token. That token is appended to the input and the loop runs again.

Generation has two phases with very different hardware profiles:

  • Prefill processes the whole prompt in parallel and is limited mainly by arithmetic throughput.
  • Decode produces one token per step and is limited mainly by memory bandwidth.

To avoid recomputing the past at every step, models keep a KV cache: the keys and values of every previous token at every layer. That cache is why long contexts consume so much memory, and shrinking it is one of the hottest areas of current architecture work.

6. Training

  • Pretraining. Decoder-only models learn to predict the next token across trillions of tokens of text and code. Thanks to the causal mask, every position in a training sequence makes its prediction simultaneously, which is the parallelism advantage over RNNs. BERT-style encoders instead learn to fill in masked tokens.
  • Post-training. Supervised fine-tuning on instructions and dialogues comes next, followed by preference optimization (RLHF, DPO) and, since 2025, large-scale reinforcement learning with verifiable rewards (RLVR) on math and code. DeepSeek-R1 showed that rewarding correct final answers alone can teach a model to produce long chains of thought, check its own work, and backtrack.
  • Inference-time compute. Reasoning models spend extra tokens "thinking" before answering, and quality often improves as that budget grows. This added a third scaling axis alongside model size and training data.

Where uncertainty remains

Every arithmetic operation inside a transformer is known exactly. What is not well understood is which algorithms the trained weights implement. Models store more concepts than they have dimensions by packing them into overlapping directions (superposition), so individual neurons rarely mean one clean thing. Interpretability tools can now trace some circuits in production models, but they explain only part of the computation and still struggle with attention. It is also unsettled how faithfully a model's written chain of thought reflects the computation that actually produced its answer; experiments have shown stated reasoning can omit factors that demonstrably influenced the output.


Mathematical and Physical Constraints

The mathematical and physical constraints that shape it: the geometry of attention, how information is encoded in high-dimensional space, how compute and memory scale, and the hardware realities (bandwidth and energy) that now drive architecture decisions as much as accuracy does.

The attention equation

Attention(Q, K, V) = softmax(Q · Kᵀ / √d_k) · V

Q, K, and V stack the queries, keys, and values for all n tokens. The product Q · Kᵀ is an n × n grid of similarity scores, one for every pair of positions.

  • Why divide by √d_k? If the components of a query and a key are independent with mean 0 and variance 1, their dot product has variance d_k. Without scaling, scores grow with head size, the softmax saturates toward a one-hot choice, and gradients vanish. Dividing by √d_k keeps scores near unit variance.
  • A Boltzmann distribution in disguise. Softmax is mathematically the Boltzmann (Gibbs) distribution from statistical physics: scores play the role of negative energies, and the scaling factor acts like a temperature. Low temperature means sharp, nearly hard lookups; high temperature means diffuse averaging.
  • Associative memory. Attention behaves like a soft, differentiable dictionary lookup. Ramsauer and colleagues (2020) showed that one attention step is equivalent to an update of a modern continuous Hopfield network, linking transformers to a long line of physics-inspired memory models.

Multi-head attention splits the width into h heads of size d_k = d_model / h, runs attention in each, then concatenates and projects:

MultiHead(X) = Concat(head_1, ..., head_h) · W_O

Total cost stays close to one full-width head, but the model gains h independent ways to decide what is relevant.

Encoding order

Attention by itself is permutation-equivariant: shuffle the input and you shuffle the output the same way. Without position information, "dog bites man" and "man bites dog" would look identical.

  • Sinusoidal encodings (2017) add fixed waves of different frequencies to the embeddings: PE(pos, 2i) = sin(pos / 10000^(2i / d_model)), with cosine for odd dimensions.
  • Rotary position embeddings (RoPE), now standard, rotate each pair of query and key dimensions by an angle proportional to the token's position. Because rotations compose, the dot product between a query at position m and a key at position n depends only on the offset m − n. Context windows are commonly extended by rescaling RoPE frequencies (for example, the YaRN method).
  • Some recent models leave positional encoding out of selected layers entirely (often called NoPE) and rely on the causal mask and other layers to infer order.

The geometry of the residual stream

The residual stream lives in a space of thousands of dimensions, and features appear to be stored as directions in that space. High-dimensional spaces can hold vastly more nearly orthogonal directions than they have axes, which is how models pack many more concepts than dimensions (the superposition hypothesis, studied in Anthropic's Toy Models of Superposition). Normalization layers keep vector magnitudes under control, but trained models still show odd physical-looking phenomena:

  • Massive activations: a few dimensions take on values far larger than the rest.
  • Attention sinks: many heads dump a large share of attention onto the first token, using it as a harmless "no-op" target. StreamingLLM (2023) exploited this to stream very long inputs; the 2025 gated-attention work described below largely removes it.

Scaling relationships

Quantity Approximate relationship What it means
Parameters per dense layer ≈ 12 · d_model² (4 · d² for attention plus 8 · d² for a 4× MLP) GPT-3 check: 96 × 12 × 12,288² ≈ 174 billion, matching its 175B size
Training compute C ≈ 6 · N · D FLOPs (N parameters, D training tokens) Forward pass costs about 2ND, backward pass about 4ND
Inference compute ≈ 2 · N FLOPs per generated token, plus an attention term Mixture of experts cuts this by using only the active parameters
Attention vs. dense compute per token per layer ≈ 4 · n · d (attention) vs. ≈ 24 · d² (projections and MLP) Attention dominates once context length n exceeds roughly 6 · d; for d = 4,096 that is a few tens of thousands of tokens (back-of-envelope, ignoring GQA and MoE)
Compute-optimal data ≈ 20 training tokens per parameter (Chinchilla, 2022) Modern models train far past this point (Llama 3 8B saw more than 15 trillion tokens) because smaller, longer-trained models are cheaper to serve
Loss vs. scale Power laws in N, D, and C (Kaplan et al., 2020) Predictable improvement from scale; the basis for planning large training runs

Memory: the KV cache

The KV cache stores keys and values for every layer and every previous token:

KV bytes ≈ 2 × layers × KV heads × head dimension × tokens × bytes per value

Worked example. A Llama-3-70B-class configuration (80 layers, 8 KV heads of dimension 128, 16-bit values) needs about 0.33 MB per token. At a 128K-token context, that is roughly 43 GB for a single sequence. With classic multi-head attention (64 KV heads instead of 8), it would be around 344 GB. That gap explains why nearly every attention variant of the last three years targets the KV cache:

  • Multi-query and grouped-query attention (MQA, GQA) share each key-value head across several query heads.
  • Multi-head latent attention (MLA), introduced in DeepSeek-V2, caches a compressed latent vector per token and re-expands it when needed.
  • Sliding-window attention limits some layers to a local window (for example, 512 tokens), so their cache stops growing.
  • Sequence compression, as in DeepSeek V4, summarizes groups of tokens into fewer cache entries.
  • Cross-layer KV sharing, as in Gemma 4's small models, lets later layers reuse earlier layers' keys and values.

Bandwidth, not arithmetic, often sets the speed limit

During decoding, each new token requires reading every active weight from memory at least once. A dense 70-billion-parameter model stored in 16-bit precision is about 140 GB, so even with an aggregate memory bandwidth of 10 TB/s across several accelerators, single-stream decoding cannot exceed roughly 70 tokens per second. That physical ceiling motivates MoE (fewer active weights), quantization to 8-bit and 4-bit formats, batching many users together, and speculative decoding. Moving data also costs far more energy than doing arithmetic on it, so data movement dominates both latency and power.

FlashAttention (2022) applies the same insight to attention itself. Instead of writing the full n × n score matrix to slow GPU memory, it processes attention in tiles held in fast on-chip memory and uses an "online" softmax that updates running maxima and sums as it goes. The result is exact, uses memory linear in sequence length, and is much faster. Later versions co-designed the algorithm with each GPU generation, culminating in FlashAttention-4 for NVIDIA Blackwell in 2026 (see below).

Theoretical limits

  • Expressivity. Theoretical work by Merrill and Sabharwal and others places fixed-depth, fixed-precision transformers that answer in one step inside a limited circuit class (uniform TC0). Under standard complexity assumptions, that means they cannot reliably solve some inherently sequential problems, such as tracking the state of a long sequence of operations. Allowing a chain of thought changes the picture: a number of intermediate steps linear in input length lets them track finite-state machines, and a polynomial number reaches the class P under the paper's assumptions. This gives a formal rationale for reasoning tokens and for "looped" architectures that add depth.
  • Length generalization. Models often fail on inputs longer than those seen in training, which is partly a positional-encoding problem and partly a learned-algorithm problem.
  • Long-context use. Advertised context windows overstate usable context. Lost in the Middle (Liu et al., 2023) found that retrieval accuracy follows a U-shape, best for information at the start or end of the input. Chroma's 2025 "context rot" study found performance degrading with input length across 18 models, even on simple tasks.

Why This Matters

Scientific and conceptual significance

  • One architecture, many domains. The same block, with minimal built-in assumptions, learns language, images, audio, code, protein structure, and weather patterns. That suggests much of what earlier architectures hard-coded can instead be learned from data at scale.
  • Scale as a research method. Predictable scaling laws turned model building into something closer to engineering, where you can forecast the payoff of a larger run before paying for it. Stanford's CS336 course frames the lesson as "algorithms that scale are what matter."
  • Emergent abilities. In-context learning, the ability to pick up a task from examples in the prompt without changing any weights, emerged from next-token training rather than explicit design, and interpretability work has tied part of it to specific attention circuits.
  • A new object of study. Transformers are now complex enough that researchers study them like organisms, as in Anthropic's 2025 work on "the biology of a large language model."

Engineering implications

Transformers turned AI into a matrix-multiplication problem, which aligned perfectly with GPU and TPU hardware. The entire stack has since been co-designed around them: fused attention kernels, inference servers, low-precision number formats, and even GPU features such as Blackwell's on-chip tensor memory. That creates a strong incumbency effect. A competing architecture has to beat not just the transformer's quality but nearly a decade of optimization. It also explains why so many 2025 to 2026 "new architectures" are really hybrids that keep most transformer machinery intact.

Applications

Competitive programming, conversational question answering, combinatorial optimization, and graph learning all incorporate transformers as key components.

  • Competitive programming. Transformer models produce solutions from textual problem descriptions. DeepMind's AlphaCode (Science, 2022) reached roughly median-human level in Codeforces contests. By September 2025, reasoning models had gone much further: at the ICPC World Finals, an OpenAI system solved all 12 problems under contest rules, and Google DeepMind's Gemini 2.5 Deep Think solved 10, including one that no human team solved.
  • Conversational AI. The well-known chatbot ChatGPT, a GPT-based model, remains the best-known example of a transformer in daily use, alongside Claude, Gemini, and open-weight models such as those from DeepSeek, Qwen (Alibaba), and Meta.
  • Combinatorial optimization. Attention-based models have learned heuristics for routing problems such as the Travelling Salesman Problem (Kool, van Hoof, and Welling, 2019).
  • Graph learning and molecules. Graph transformers such as Graphormer performed strongly on molecular property prediction, and attention is now standard in molecular and materials models.
  • Vision and generation. The Vision Transformer (2020) treats image patches as tokens. Diffusion transformers (DiT) became the backbone of many image and video generators.
  • Biology. AlphaFold 2 and AlphaFold 3 rely on attention-based modules to reason over residue pairs when predicting the structure of proteins and their complexes.
  • Speech and audio. Encoder-decoder transformers such as Whisper handle speech recognition, and transformer decoders generate speech and music.
  • Agents. Transformer LLMs now drive agents that call tools, write and run code, and operate computer interfaces.

Transformer models have shown great versatility across modalities, such as images, audio, video, and undirected Graphs. Click here for directed graphs.

Limitations

  • Quadratic attention cost and growing KV memory make long contexts expensive.
  • Effective context is shorter than advertised because of positional bias and context rot.
  • Hallucination. Models generate fluent but false statements; interpretability work has linked some cases to internal "known entity" circuits misfiring.
  • Data and compute hunger. Frontier training runs consume tens of trillions of tokens and large amounts of energy.
  • Opacity. Interpretability is improving but explains only part of the computation.
  • Faithfulness of reasoning. Visible chains of thought do not always reflect the internal process, which complicates oversight.
  • Sequential generation. Training is parallel, but autoregressive decoding still produces one token at a time.

What would change if current research pans out

  • If hybrid or sparse attention fully matches full attention, million-token contexts become routine and cheap, and capable models run on phones and laptops.
  • If interpretability matures, auditors could check models for deception or hidden objectives directly instead of relying only on behavioral tests.
  • If looped or latent reasoning architectures spread, models could think more without writing more tokens, but that could also make their reasoning harder to monitor.
  • If diffusion language models close the quality gap, text generation could become massively parallel, cutting latency by an order of magnitude.


Evolution of the Architecture

The Transformer is a deep machine learning model introduced in 2017 and used first in natural language processing (NLP). Like Recurrent Neural Network (RNN)s, Transformers handle ordered sequences of data, such as natural language, for tasks like machine translation and text summarization. Unlike RNNs, Transformers do not need to process a sequence one element at a time during training: the model does not have to finish the beginning of a sentence before it works on the end. That allows far more parallelization, which made it practical to train on much more data than before. One important qualification: decoder-only models still generate text one token at a time, which is why inference speed has become a central engineering problem. Since their introduction, Transformers have replaced gated recurrent models such as the Long Short-Term Memory (LSTM) across essentially all frontier NLP. This led to pretrained systems such as Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformer (GPT)-2, which were trained on huge amounts of general language data and then fine-tuned for specific tasks, and later to the instruction-tuned and reasoning-trained LLMs in use today. Wikipedia

Year Milestone
2017 Attention Is All You Need introduces the Transformer for machine translation
2018 GPT (decoder-only pretraining) and BERT (encoder-only pretraining)
2019 GPT-2, Transformer-XL, XLNet, and T5 (text-to-text encoder-decoder)
2020 GPT-3 (175B parameters), scaling laws, and the Vision Transformer
2021 Switch Transformer passes a trillion parameters with mixture of experts; AlphaFold 2 published
2022 Chinchilla compute-optimal scaling, FlashAttention, and ChatGPT
2023 Llama opens the weights era; grouped-query attention; Mamba revives recurrent-style alternatives
2024 Multi-head latent attention (DeepSeek-V2), FlashAttention-3, AlphaFold 3
2025 DeepSeek-R1 and RL-trained reasoning; Native Sparse Attention; Qwen3-Next and Kimi Linear hybrids; gated attention wins a NeurIPS best paper; mHC paper
2026 FlashAttention-4, Gemma 4, DeepSeek V4 with compressed sparse attention and mHC, looped transformers in released models, Hugging Face Transformers v5
Component Original Transformer (2017) Typical frontier or open-weight LLM (2025 to 2026)
Overall layout Encoder-decoder, 6 encoder and 6 decoder layers Decoder-only, roughly 30 to more than 100 layers
Width and heads d_model 512, 8 heads (large model: 1,024 and 16) d_model in the thousands, dozens of query heads
Normalization LayerNorm after each sublayer RMSNorm before each sublayer, often with QK-Norm
Position information Fixed sinusoidal encodings added to embeddings Rotary embeddings (RoPE) on queries and keys; some layers use none
Attention Full multi-head attention GQA or MLA, mixed with sliding-window, sparse, compressed, or linear-attention layers; sometimes output-gated
Feed-forward Dense ReLU MLP Gated (SwiGLU-style) MLP, usually a sparse mixture of experts
Residual path One residual stream One stream in most models; widened multi-stream residuals (mHC) in DeepSeek V4 and some later models
Context length Sentences, a few hundred tokens 128K to 1M tokens advertised
Training signal Supervised translation pairs Next-token pretraining on trillions of tokens, then SFT, preference tuning, and RL with verifiable rewards

Recent Developments

Most of the important work from 2025 and 2026 falls into a few threads: making long contexts affordable, stabilizing and widening the core block, adding depth without adding parameters, co-designing kernels with new hardware, and looking inside the models. Unless noted, the results below are reported by the developers themselves and have not been independently replicated at full scale.

Trainable sparse attention: NSA and DeepSeek Sparse Attention

  • What was tested. DeepSeek, Peking University, and the University of Washington proposed Native Sparse Attention (NSA) in February 2025. It won a Best Paper award at ACL 2025.
  • Method. Each query attends through three paths: a compressed summary of distant blocks, a learned selection of the most relevant blocks at full resolution, and a local sliding window. The selection is trained end to end, and the kernels were designed around GPU memory access patterns.
  • Findings. On a 27-billion-parameter backbone, NSA matched or beat full attention on general, long-context, and reasoning benchmarks while running up to about 11.6× faster at decoding and about 9× faster in the forward pass at 64K tokens.
  • Follow-on. DeepSeek V3.2 (late 2025) introduced DeepSeek Sparse Attention (DSA), which uses a lightweight "indexer" to pick the top-k most relevant earlier tokens and runs full attention only on those. Notably, it can be added during long-context continued training instead of from scratch. Zhipu's GLM-5 adopted DSA, and GLM-5.2 added "IndexShare," reusing token selections across layers to cut indexer cost.
  • What it establishes, and what it doesn't. Sparse attention can match dense attention at tens of billions of parameters. The indexer itself is still quadratic (just cheap), and quality depends heavily on the selection being right for the task.

DeepSeek V4: compressed attention at a million tokens

  • What was released. On April 24, 2026, DeepSeek released V4 previews under the MIT license: V4-Pro (1.6 trillion total, 49 billion active parameters) and V4-Flash (284 billion total, 13 billion active), both with 1M-token context by default.
  • Method. Instead of compressing each token's cache entry (MLA's approach), V4 compresses along the sequence. Compressed Sparse Attention (CSA) summarizes small groups of tokens and selects among them DSA-style; Heavily Compressed Attention (HCA) collapses every 128 tokens into one entry and attends to all of them densely. Layers alternate between the two, and both keep a 128-token sliding window of uncompressed recent tokens.
  • Findings. DeepSeek reports that at 1M tokens, V4-Pro needs 27% of the single-token inference FLOPs and 10% of the KV cache of DeepSeek V3.2; V4-Flash needs 10% and 7%.
  • Caveats. Raschka notes the paper has no ablation isolating the attention change, and the overall gains are entangled with new data, the Muon optimizer, mHC residuals, and precision tricks. Treat CSA/HCA as an aggressive efficiency design that appears to preserve quality in DeepSeek's flagship, not as proven to be universally better than MLA.

Hybrid linear-attention models

Linear attention replaces the softmax with a form whose matrix products can be reordered, so the model keeps a fixed-size state that updates token by token, much like an RNN. Cost becomes linear in sequence length and memory stops growing. The catch is that a fixed-size state forgets, so pure linear models have trailed full attention on precise recall.

  • Gated DeltaNet (ICLR 2025) improved the state update with input-dependent gates and a "delta rule" that overwrites old associations instead of just accumulating them.
  • Qwen3-Next (Alibaba, September 2025) combined Gated DeltaNet layers with gated full-attention layers; Stanford's CS336 course notes that Qwen 3.5 uses a 3:1 Gated DeltaNet-to-attention hybrid.
  • Kimi Linear (Moonshot AI, October 2025) extended Gated DeltaNet with finer-grained gating (Kimi Delta Attention, KDA) and interleaved three KDA layers per MLA layer. Trained on 1.4 trillion tokens with an identical recipe, the 48B-total, 3B-active model outperformed a full-MLA baseline across short-context, long-context, and RL tasks, while cutting KV cache use by up to 75% and delivering up to 6× decoding throughput at 1M tokens. External reproductions were still underway at release.
  • NVIDIA's Nemotron 3 family mixes Mamba-2 state space layers with attention layers for throughput.
  • A counterpoint. MiniMax's earlier MiniMax-01 used seven linear "lightning attention" layers per softmax layer, yet its later MiniMax M2 returned to full attention. The trade-off between efficiency and precise long-range recall is not settled.

Gated attention and the end of the attention sink

  • What was tested. Alibaba's Qwen team systematically compared more than 30 variants of gating inside softmax attention, using 15B MoE and 1.7B dense models trained on up to 3.5 trillion tokens.
  • Findings. The best variant applies a head-specific, element-wise sigmoid gate right after the scaled dot-product attention output. It consistently improved quality, allowed higher learning rates with fewer loss spikes, and improved context-length extension. It also nearly eliminated the attention sink: in their analysis, the share of attention landing on the first token fell from about 47% to about 5%.
  • Recognition and adoption. The paper received a NeurIPS 2025 Best Paper Award, and the gate ships in Qwen3-Next. Per-head output gating has since appeared in other models, including Poolside's Laguna.
  • Why it matters. It suggests attention sinks are a workaround the model invents because standard attention has no way to "attend to nothing." Give it a gate and the workaround disappears.

Rethinking the residual stream: hyper-connections and mHC

The thesis callout describes one "main artery" per token. That picture is starting to change.

  • Hyper-connections (2024) widen the residual stream into several parallel streams with learned mixing between them, while attention and MoE layers still operate at normal width.
  • Manifold-Constrained Hyper-Connections (mHC), published by DeepSeek on December 31, 2025, constrain that mixing to doubly stochastic matrices (non-negative entries, every row and column summing to 1). The mixing then acts like a stable redistribution of information rather than something that can amplify or shrink signals unpredictably across depth. On a 27B test model with four streams, the optimized implementation added about 6.7% training-time overhead.
  • Adoption. mHC moved from paper to flagship in DeepSeek V4, and Raschka's architecture notes describe later models (such as GLM-5.3-Flash) with four-stream mHC residual paths.
  • Open question. Gains so far are modest and reported mostly by developers. The practical cost is memory traffic and implementation complexity more than arithmetic.

Shrinking the KV cache further

  • Cross-layer KV sharing (Gemma 4, April 2026). In Google's small Gemma 4 E2B model, only the first 15 of 35 layers compute their own keys and values; later layers reuse those of an earlier layer of the same attention type. Raschka estimates this saves about 2.7 GB at a 128K context in 16-bit precision. The trade-off is reduced capacity, which earlier research found small for small models.
  • Compressed convolutional attention (Zyphra ZAYA1-8B). Queries, keys, and values are compressed and attention runs directly in the compressed space, with cheap convolutions restoring some local context. This cuts attention FLOPs as well as cache size. The model was also notable for being trained on AMD GPUs.
  • Per-layer attention budgets (Poolside Laguna XS.2). 30 sliding-window layers (512-token window) and 10 global layers get different numbers of query heads, spending attention capacity where it helps most.

Looped transformers and recurrent depth

A looped transformer runs the same block or stack of blocks several times, increasing effective depth without adding parameters. The idea dates to the Universal Transformer (2018) and has returned in force.

  • Research results. A 2025 recurrent-depth study trained a 3.5B model on 800 billion tokens and showed it could use more loops at inference time to improve reasoning tasks. Mixture-of-Recursions (2025) routed each token through a different number of loops. A June 2025 study found looping barely increases memorization capacity but does improve multi-step reasoning, which fits the theory that depth, not storage, limits sequential reasoning.
  • Released models. Nanbeige4.2-3B (July 2026) runs its 22-block stack twice. Its authors found that sharing the KV cache between passes halved memory but hurt quality, so each pass keeps its own cache. ByteDance's Ouro applies 48 blocks four times.
  • Compute-matched evidence. The SMELT paper (September 2026) compared looped and conventional MoE transformers at matched compute, parameters, and KV cache, up to 54B non-embedding parameters. Fitted scaling curves suggest the looped version reaches the same loss with roughly 6.8% to 18% less training compute within the studied range. This is a single, very recent study.
  • GPT-6 Astra. Shortly before OpenAI released GPT-6 Astra in early September 2026, The Information reported that it uses recurrent depth and raised concerns that this hides reasoning. OpenAI has not confirmed the architecture. Its chief scientist, Jakub Pachocki, stated that the depth of the computation graph for its current frontier models, including Astra, is within a factor of two of GPT-4, and that pressures on chain-of-thought monitorability are not caused by architecture changes. Raschka judges some looping likely but argues Astra's gains come mainly from data and training.

Hardware co-design: FlashAttention-4

  • What was built. Tri Dao and collaborators from Princeton, Together AI, Meta, NVIDIA, and Colfax released FlashAttention-4 on March 5, 2026 (published at MLSys 2026).
  • The problem. On NVIDIA Blackwell GPUs, tensor-core throughput roughly doubled over Hopper, but shared-memory bandwidth and the units that compute exponentials did not keep pace. Attention's bottleneck moved from matrix math to the softmax and data movement.
  • Method. FA4 computes some exponentials in software using polynomial approximations on general-purpose units, skips rescaling the running output unless the softmax maximum shifts enough to threaten numerical stability, stores intermediates in Blackwell's new on-chip tensor memory, and uses paired thread blocks to cut shared-memory traffic. It is written entirely in a Python-embedded DSL (CuTe-DSL).
  • Findings. Up to about 1,605 TFLOPs/s on a B200 (around 71% utilization), roughly 1.3× faster than cuDNN 9.13 and 2.7× faster than Triton implementations. PyTorch's FlexAttention added FA4 as a backend, and NVIDIA has folded several FA4 techniques into newer cuDNN releases.
  • Why it matters. Attention speed now depends as much on matching the algorithm to each chip's balance of resources as on the math itself.

Reasoning through reinforcement learning

  • DeepSeek-R1's methods were published in Nature on September 17, 2025, the first major LLM to go through formal peer review. The paper shows that reinforcement learning on verifiable rewards alone can induce self-verification, reflection, and longer reasoning in a standard transformer, and that those patterns can be distilled into smaller models.
  • The architecture did not change; the training did. The ICPC 2025 results and gold-medal-level performance at the 2025 International Mathematical Olympiad came from transformer-based reasoning systems trained this way.
  • RL is now spreading backward into pretraining. A CS25 (Spring 2026) talk reported gains from adding reasoning data early and from treating "think before predicting" as a pretraining objective, results that remain early and developer-reported.

Interpretability: looking inside the residual stream

  • In March 2025, Anthropic published Circuit Tracing and On the Biology of a Large Language Model. The team replaced Claude 3.5 Haiku's MLPs with a cross-layer transcoder of roughly 30 million interpretable features and built "attribution graphs" that trace how features pass information toward an output, then tested those hypotheses by intervening on the features.
  • Findings. Asked for the capital of the state containing Dallas, the model internally represents "Texas" before producing "Austin," which is evidence of genuine two-step reasoning. It plans rhymes before writing a line of poetry, uses a mix of language-specific and language-independent circuits, and some hallucinations trace to a "known entity" circuit misfiring.
  • Limits. The authors stress the graphs are simplifications. The replacement model does not replace attention layers, and the method only partially explains each prompt.
  • Recognition. MIT Technology Review named mechanistic interpretability one of its 10 Breakthrough Technologies of 2026, citing work from Anthropic, Google DeepMind, OpenAI, and Neuronpedia.

Diffusion language models: same transformer, different generation

Diffusion language models start from masked or noisy text and refine many positions in parallel over several steps, rather than writing left to right. Importantly, the network doing the refining is still a (bidirectional) transformer; what changes is the generation procedure.

  • LLaDA (2025) scaled masked diffusion to 8 billion parameters and reported results competitive with similarly sized autoregressive models such as Llama 3 8B, with an advantage on "reversal" tasks that trip up left-to-right models.
  • Inception's Mercury became the first commercial-scale diffusion LLM; its Mercury Coder models exceeded 1,000 tokens per second on H100 GPUs. Mercury 2 (February 2026) added reasoning and claims roughly 1,000 tokens per second.
  • Google's Gemini Diffusion remains an experimental preview; Google lists a sampling speed of 1,479 tokens per second, excluding a fixed overhead of about 0.84 seconds.
  • Status. Speed advantages are real, but quality comparisons rest mostly on vendor benchmarks, and frontier capability still belongs to autoregressive transformers.

Long-context reality checks

  • Chroma's mid-2025 Context Rot report held task difficulty constant while varying only input length across 18 models, including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3. Performance degraded with length, often non-uniformly, even on simple retrieval and copying tasks, and degraded further with plausible distractors.
  • Caveat: Chroma builds retrieval infrastructure and has a commercial interest in this conclusion. Even so, the findings agree with earlier independent work such as Lost in the Middle, and they explain why long-context efficiency work (compression, sparsity, hybrids) is paired with better retrieval and context engineering rather than replacing them.

Open Questions and Active Debates

  • Will hybrids replace full attention? Kimi Linear reports a hybrid beating full attention under a fair comparison; MiniMax moved back to full attention; DeepSeek bets on compression plus sparsity. Independent, compute-matched comparisons at frontier scale are still scarce.
  • Architecture or recipe? Several analysts, including Raschka, argue that architecture changes now mostly reduce cost, while quality gains come from data and training. If that holds, the transformer's longevity has less to do with its optimality and more to do with how good it is as a substrate for scaling.
  • Is long context actually used? Million-token windows exist, but context-rot and positional-bias studies show models use them unevenly. Better long-context benchmarks remain an open need.
  • Can we keep reasoning visible? Looped and latent-reasoning designs could move more computation out of readable text. OpenAI's leadership calls chain-of-thought monitorability fragile and trending in a negative direction for reasons unrelated to architecture; how to preserve it is an active research goal.
  • Will interpretability scale? Circuit tracing works on production models but explains only part of the computation and largely sidesteps attention.
  • Autoregressive versus diffusion. Diffusion LMs promise parallel generation; whether they can match frontier reasoning quality is unknown.
  • Theory versus practice. Formal expressivity limits explain why extra depth or reasoning steps help, but they do not yet predict which tasks a given model will fail on.


Tensor2Tensor (T2T) | Google Brain

Tensor2Tensor, or T2T for short, is a library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research. T2T was developed and maintained by researchers and engineers within the Google Brain team and a community of users. Its Colab notebook shows the datasets available in T2T, how to download and use them, some of the models, how to download and use pretrained models, and how to create and train your own models. | Tensor2Tensor Colab, Google Brain

Historical role and current status. T2T holds a special place in transformer history: the original Attention Is All You Need code was released through it. The library is now deprecated. Google recommends its successor, Trax, and the T2T GitHub repository was archived and made read-only on July 7, 2023. T2T is still useful for reading the original implementation, but new projects should use actively maintained frameworks such as the Hugging Face Transformers library on PyTorch, or JAX-based tools.

Multi-head scaled dot-product attention mechanism. (Image source: Fig 2 in Vaswani, et al., 2017)


Transformers Course | HuggingFace

This course teaches natural language processing (NLP) and large language models using libraries from the Hugging Face ecosystem: 🤗 Transformers, 🤗 Datasets, 🤗 Tokenizers, and 🤗 Accelerate, as well as the 🤗 Hugging Face Hub. In April 2025, Hugging Face renamed the NLP Course to the LLM Course after adding chapters on fine-tuning large language models and building reasoning models in the style of DeepSeek-R1. The classic NLP chapters remain, because a solid grasp of NLP fundamentals still matters when working with LLMs.


'https://elastic-ai.com/wp-content/uploads/2021/06/hfc_summary.png'
  • Chapters 1 to 4 provide an introduction to the main concepts of the 🤗 Transformers library. By the end of this part of the course, you will be familiar with how Transformer models work and will know how to use a model from the Hugging Face Hub, fine-tune it on a dataset, and share your results on the Hub!
  • Chapters 5 to 8 teach the basics of 🤗 Datasets and 🤗 Tokenizers before diving into classic NLP tasks and LLM techniques. By the end of this part, you will be able to tackle the most common NLP problems by yourself.
  • Chapter 9 covers building and sharing interactive demos of your models.
  • Chapters 10 to 12 dive into advanced LLM topics: fine-tuning large language models, curating high-quality datasets, and building reasoning models. (Earlier versions of the course used the final chapters to explore speech processing and computer vision; that material has been reorganized as the course shifted toward LLMs.)

This course:

  • Requires a good knowledge of Python
  • Is better taken after an introductory deep learning course, such as fast.ai’s Practical Deep Learning for Coders or one of the programs developed by DeepLearning.AI
  • Does not expect prior PyTorch or TensorFlow knowledge, though some familiarity with PyTorch will help. Note that the Transformers library itself is now PyTorch-first: version 5 sunsets TensorFlow and Flax support, so older TensorFlow examples in tutorials may no longer run.

After you’ve completed this course, we recommend checking out DeepLearning.AI’s Natural Language Processing Specialization, which covers a wide range of traditional NLP models like naive Bayes and Long Short-Term Memory (LSTM)s that are well worth knowing about! For a deeper, build-it-yourself path through transformer internals, systems, and scaling, Stanford's CS336: Language Modeling from Scratch publishes its lectures online.


Videos

Classic Explainers

These explainers were collected on earlier versions of this page and remain good introductions to the original architecture.

Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube

Additional Viewing

Load video
YouTube

Stanford CS336 Language Modeling from Scratch, Spring 2026, Lecture 4: Attention Alternatives (about 87 minutes). Stanford's from-scratch LLM course explains why attention's quadratic cost dominates at long context, why FlashAttention helps only by a constant factor, and how linear attention, Mamba-2, Gated DeltaNet, and DeepSeek Sparse Attention change the math, before spending its second half on mixture-of-experts routing and load balancing. It is the best single companion to the Physics and Recent Developments sections of this page. Watch on YouTube

Load video
YouTube

Stanford CS336 Language Modeling from Scratch, Spring 2026, Lecture 1: Overview, Tokenization (about 80 minutes). The course opener, from Stanford's CS336 taught by Percy Liang and Tatsunori Hashimoto, covers why building language models yourself still matters, the history from n-grams to transformers, the efficiency mindset behind scaling, and a deep dive into byte pair encoding. It also flags a useful caution: behavior at small scale does not always transfer, since the share of compute spent in MLP layers grows sharply as models get larger. Watch on YouTube

Load video
YouTube

Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy (about 1 hour 56 minutes, 2023). Karpathy codes a decoder-only transformer in PyTorch, starting from a simple bigram model and adding self-attention, multiple heads, residual connections, and layer normalization one piece at a time, then trains it on Shakespeare. Watching it alongside The Mechanism section turns every equation on this page into running code. Watch on YouTube

Load video
YouTube

Deep Dive into LLMs like ChatGPT by Andrej Karpathy (about 3 hours 31 minutes, February 2025). A general-audience tour of the full pipeline: collecting and filtering web-scale training data, tokenization, transformer internals, pretraining, supervised fine-tuning, and reinforcement learning, including discussion of DeepSeek-R1-style reasoning and why models hallucinate. Useful for readers who want the context around the architecture rather than the math. Watch on YouTube

Load video
YouTube

State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI, Lex Fridman Podcast #490 with Nathan Lambert (Ai2) and Sebastian Raschka (about 4.5 hours, January 2026). The architecture discussion argues that the transformer core has stayed remarkably stable since GPT-2 while mixture of experts, grouped-query and latent attention, sliding windows, and Gated DeltaNet-style linear layers push efficiency. The guests also cover the three scaling axes (pretraining, RL, and inference) and the open-weight model race. Watch on YouTube

Load video
YouTube

Stanford CS25: Transformers United V6, From Next-Token Prediction to Next-Generation Intelligence (about 58 minutes, May 2026). A researcher formerly on NVIDIA's Nemotron pretraining team presents results on two-phase data curricula, adding reasoning data early in pretraining, and "reinforcement learning as a pretraining objective," where the model learns to think before predicting the next token. The talk shows where transformer training, rather than architecture, is heading; the reported gains are the speaker's own and not yet independently replicated. Watch on YouTube


References and Further Reading

Foundational Papers

Architecture and Efficiency

Looped Transformers and Recurrent Depth

Reasoning and Training

Interpretability

Long Context and Evaluation

Diffusion Language Models

Applications Beyond Text

Overviews, Courses, and Tools

Retrieved from "https://primo.ai/w/index.php?title=Transformer&oldid=53445"