...understand new architectures
Jump to navigation
Jump to search
| AI Solver map |
|---|
|
Your current page is in bold
0 · Start
1 · Predict & analyze
tables, numbers, events
Predict values
Predict categories Forecast time series Cluster / segment Find outliers Make recommendations Find key drivers 2 · Understand unstructured data
text, documents, images, audio
3 · Generate & converse
4 · Decide & act
5 · Build it right
applies to every path
|
YouTube ... Quora ...Google search ...Google News ...Bing News
- AI Solver ... Algorithms ... Neural Network ... Discriminative vs. Generative ... Generative AI ... ...choose a model
Use this page to understand why the options on other pages exist. You do not need to build these architectures to benefit from them, but they change what is cheap, fast, private or possible, and therefore which model you should pick.
The architectures and what they mean for you
| Architecture | In one line | What it changes for you | Examples |
|---|---|---|---|
| Transformer (2017) | Attention lets every token look at every other token | The base of today's language, vision and speech models; cost grows with context length | GPT, Claude, Gemini, Llama, Qwen |
| Mixture of experts (MoE) | Many "expert" sub-networks; a router activates only a few for each token | Large-model quality for less compute per token, but all experts must fit in memory: great in the cloud, harder on devices | Mixtral, DeepSeek, Qwen MoE, Llama 4, gpt-oss, Kimi |
| Reasoning models (test-time compute) | The model works through a problem step by step before answering; trained with reinforcement learning on checkable tasks | Much better at math, code and planning; slower and uses more tokens, so set the effort per task | OpenAI reasoning models, DeepSeek-R1, Claude and Gemini "thinking" modes, Qwen thinking models |
| State-space and hybrid models (Mamba, linear attention) | Recurrent-style layers that use constant memory per token, mixed with a few attention layers | Faster, cheaper long contexts and high-throughput serving; attractive for agents and edge devices | Jamba (AI21), NVIDIA Nemotron-H and Nemotron 3, IBM Granite 4.0, Qwen3-Next, Kimi Linear |
| Diffusion (images, video, audio, and now text) | Start from noise and refine the whole output in parallel | The standard for images and video; diffusion LLMs generate 1,000+ tokens per second for latency-critical apps, at mid-tier quality so far | Stable Diffusion, FLUX, Veo, Sora; Inception Mercury, Gemini Diffusion (experimental) |
| Multimodal and "omni" models | One model reads, and often writes, text, images, audio and video | Document understanding, visual Q&A, natural voice agents without separate pipelines | GPT, Gemini, Claude (vision), Qwen-VL, Gemma |
| Small language models (SLMs) | Roughly 1–15 billion parameters, often distilled from larger models | Run on laptops and phones; cheap and private; ideal for narrow tasks and agent sub-steps | Gemma, Phi, small Qwen and Llama models, Granite, SmolLM, Apple's on-device model |
| Embedding and retrieval models | Turn content into vectors so similar things are close together | The backbone of search, RAG, deduplication and clustering | Commercial and open embedding and reranking models (compare on MTEB) |
| Tabular and time-series foundation models | Pretrained on many real or synthetic datasets; predict new data "in context" | Strong results with little or no training; usually need a GPU | TabPFN, TabICL; Chronos-2, TimesFM, Moirai, TimeGPT, Toto |
| Graph neural networks (GNNs) | Learn from nodes and the links between them | Fraud rings, molecules, traffic and weather | GraphCast (weather), traffic-aware arrival-time models |
| World models | Learn how an environment changes, so a system can imagine the results of actions | Simulation for robots, games and planning; early but moving fast | Google DeepMind Genie 3, Meta V-JEPA 2, NVIDIA Cosmos |
| Vision-language-action (VLA) models | Camera images + an instruction → robot motor actions | General-purpose robots that follow plain-language instructions | Google DeepMind Gemini Robotics, Physical Intelligence π0, NVIDIA GR00T |
| Compound AI / agentic systems | Not one network: models + retrieval + tools + memory + control logic | Most real products; quality comes from system design and evals as much as from the model | RAG assistants, coding agents, MCP- and A2A-connected agents |
Classics still earning their keep: convolutional networks (efficient vision on small devices), RNNs/LSTMs (small streaming signals), GANs (fast single-step generation, super-resolution), autoencoders and VAEs (anomaly detection; the compressed space many diffusion models work in), and gradient-boosted trees (most tabular problems).
What this means for your decisions
- Price-performance improves every few months. Keep evals and a model-neutral design so you can switch (...choose a model).
- Inference-time compute is a dial. More "thinking" buys accuracy with latency and cost; tune it per task.
- Long context and retrieval are partners, not rivals. Hybrid and MoE models make long contexts cheaper, but retrieval still wins for large, changing or permissioned data (...answer questions from my data).
- Small models handle more of the work. Route routine steps to SLMs and keep large models for the hard parts (...build an agent).
- Try a pretrained baseline before training anything — even for tables and time series (...predict values, ...forecast time series).
Learn more
- Attention Is All You Need (the Transformer) | arXiv
- Mixtral of Experts | arXiv
- DeepSeek-R1: incentivizing reasoning with reinforcement learning | arXiv
- Mamba: linear-time sequence modeling with selective state spaces | arXiv
- Nemotron-H: hybrid Mamba-Transformer models | NVIDIA - arXiv
- Mercury diffusion language models | Inception
- Chronos-2: universal time-series forecasting | arXiv
- TabPFN: a tabular foundation model | Nature
- The shift from models to compound AI systems | Berkeley AI Research
- Algorithms ... Neural Network ... Generative AI
Where next? ◀ Build it right · ...choose a model · ...deploy AI · ...pick a product or platform · ▶ ...evaluate and trust AI