Difference between revisions of "Architectures"

From
Jump to: navigation, search
m (→‎Ad Viewing)
 
(One intermediate revision by the same user not shown)
Line 41: Line 41:
 
<hr>
 
<hr>
  
−
= Introduction / Overview =
+
 
  
 
An '''AI architecture''' describes how the parts of an artificial intelligence system are organized and how information moves between them.
 
An '''AI architecture''' describes how the parts of an artificial intelligence system are organized and how information moves between them.
Line 743: Line 743:
 
<hr>
 
<hr>
  
−
= Ad Viewing =
+
= Additional Viewing =
  
 
The following long-form videos provide useful technical introductions to architectures discussed on this page. Each is longer than 20 minutes.
 
The following long-form videos provide useful technical introductions to architectures discussed on this page. Each is longer than 20 minutes.
Line 807: Line 807:
  
 
This is the most direct long-form introduction to the architectural ideas behind JEPA and its relationship to systems intended to understand and operate in the physical world.
 
This is the most direct long-form introduction to the architectural ideas behind JEPA and its relationship to systems intended to understand and operate in the physical world.
−
 
−
<hr>
 
−
 
−
'''Related topics:''' [[Neural Architecture]] ... [[Generative AI Stack]] ... [[Mixture-of-Experts (MoE)]] ... [[Memory]] ... [[Lifelong Learning]] ... [[Neuromorphic Computing]] ... [[Decentralized: Federated & Distributed]] ... [[Processing Units - CPU, GPU, APU, TPU, VPU, FPGA, QPU]] ... [[Containers; Docker, Kubernetes & Microservices]] ... [[Serverless]]
 

Latest revision as of 12:07, 20 September 2026

YouTube ... Quora ...Google search ...Google News ...Bing News

Additional architecture topics:


An AI architecture describes how the parts of an artificial intelligence system are organized and how information moves between them.

The term can refer to several different levels of a system:

  • Model architecture — the structure of the neural network itself, such as a transformer, convolutional network, graph neural network, mixture-of-experts model, or Joint Embedding Predictive Architecture.
  • Learning architecture — how the model learns, including supervised, self-supervised, reinforcement, generative, predictive, and transfer-learning approaches.
  • System architecture — how models connect with memory, retrieval, tools, databases, applications, APIs, and other services.
  • Compute architecture — the CPUs, GPUs, TPUs, accelerators, networking, storage, and distributed infrastructure used to train and run the system.
  • Deployment architecture — how the completed system operates across cloud, edge, mobile, robotic, enterprise, or decentralized environments.

Modern AI rarely depends on a single architecture. A production system may use a transformer as its core model, a mixture-of-experts layer for scaling, vector or graph storage for memory, external tools for actions, GPUs for inference, and containers or serverless services for deployment.

The architecture determines more than raw performance. It affects:

  • What kinds of information the system can process.
  • How efficiently it can learn.
  • How much compute and memory it requires.
  • How easily it scales.
  • How quickly it responds.
  • Whether it can adapt to new tasks.
  • Whether it can reason about sequences, graphs, images, video, language, or physical environments.
  • How reliably it can be deployed and maintained.

The important question is therefore not simply "Which AI architecture is best?"

It is:

Which combination of architectures best fits the problem being solved?

Technical Architecture

A Modern AI System

A simplified AI architecture can be viewed as a series of layers: 

                    ┌───────────────────────┐
                    │        Users          │
                    │ People / Apps / Agents│
                    └───────────┬───────────┘
                                │
                                ▼
                    ┌───────────────────────┐
                    │   Interface Layer     │
                    │ Text / Voice / Vision │
                    │ API / Robot / Sensors │
                    └───────────┬───────────┘
                                │
                                ▼
                    ┌───────────────────────┐
                    │ Application / Agent   │
                    │ Orchestration Layer   │
                    │ Routing / Tools /     │
                    │ Policies / Workflows  │
                    └───────────┬───────────┘
                                │
               ┌────────────────┼────────────────┐
               │                │                │
               ▼                ▼                ▼
       ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
       │ Core Models  │ │    Memory    │ │    Tools     │
       │ Transformer  │ │ Context      │ │ Search       │
       │ MoE / JEPA   │ │ Retrieval    │ │ APIs         │
       │ GNN / Hybrid │ │ Databases    │ │ Applications │
       └──────┬───────┘ └──────────────┘ └──────────────┘
              │
              ▼
       ┌─────────────────────────┐
       │ Compute & Infrastructure│
       │ GPU / TPU / CPU / Edge │
       │ Distributed Computing  │
       │ Containers / Serverless│
       └─────────────┬───────────┘
                     │
                     ▼
              Output / Action

Each layer can use a different architectural approach.

Transformer Architectures

Transformers became a dominant architecture for language and increasingly for vision, audio, video, and multimodal AI.

Instead of processing information strictly one step at a time, transformers use attention to determine which parts of the input are most relevant to other parts.

Important transformer components include:

  • Token or input embeddings.
  • Positional information.
  • Self-attention.
  • Multi-head attention.
  • Feed-forward layers.
  • Residual connections.
  • Normalization.
  • Encoder, decoder, or encoder-decoder arrangements.

Transformer systems can be:

Encoder-only
Optimized primarily for understanding or representation tasks.
Decoder-only
Common in generative large language models, where the model predicts subsequent tokens.
Encoder-decoder
Useful when one sequence must be transformed into another, such as translation or structured generation.

Modern implementations also use techniques such as grouped-query attention, sparse attention, optimized key-value caches, and alternative positional representations to improve efficiency.

Mixture-of-Experts Architecture

Mixture-of-Experts (MoE) increases model capacity without activating every parameter for every input.

Instead of sending information through one large feed-forward network, an MoE layer contains multiple specialized experts.

A router determines which experts should process each token or input. 

                      Input
                        │
                        ▼
                     Router
               ┌────────┼────────┐
               ▼        ▼        ▼
            Expert 1 Expert 2 Expert 3 ... Expert N
               │        │
               └────┬───┘
                    ▼
                  Output

Only a subset of the experts may be active at one time.

This produces a sparse architecture: the model can contain a very large number of total parameters while using a smaller portion of them for each inference step.

Advantages can include:

  • Greater model capacity.
  • Computational efficiency relative to similarly sized dense models.
  • Specialization among experts.
  • Scalability across distributed hardware.

Challenges include:

  • Routing efficiency.
  • Load balancing among experts.
  • Communication between accelerators.
  • Training stability.
  • Memory requirements.

Joint Embedding Predictive Architecture (JEPA)

Joint Embedding Predictive Architecture (JEPA) is a family of predictive, self-supervised architectures associated with work led by Yann LeCun and researchers at Meta FAIR.

JEPA takes a different approach from systems that learn primarily by reconstructing raw inputs or predicting the next token or pixel.

The central idea is:

Predict the representation of missing or future information rather than reconstructing the information itself.

In a simplified JEPA architecture: 

              Observed Input
                    │
                    ▼
             Context Encoder
                    │
                    ▼
            Context Embedding
                    │
                    │
                    ▼
                Predictor
                    │
                    ▼
          Predicted Representation
                    │
                    │ compare
                    ▼
            Target Representation
                    ▲
                    │
             Target Encoder
                    ▲
                    │
              Target Input

The encoders convert observations into abstract numerical representations called embeddings.

The predictor then attempts to infer the representation of a hidden, missing, or future portion of the input.

The system learns by making its predicted representation close to the target representation.

This matters because many details in the physical world are difficult or unnecessary to predict precisely.

For example, imagine watching a person walk behind a parked truck.

A pixel-generating system might attempt to predict:

  • Exact colors.
  • Shadows.
  • Texture.
  • Clothing movement.
  • Background details.
  • Every possible future pixel.

A JEPA-style system can instead learn a higher-level representation capturing the useful prediction:

The person continues moving and is likely to emerge from the other side of the truck.

The architecture therefore attempts to concentrate learning on predictable structure rather than requiring exact reconstruction of every low-level detail.

I-JEPA

I-JEPA applies Joint Embedding Predictive Architecture to images.

Parts of an image are hidden from the context encoder. The system predicts representations of the hidden portions from the visible portions.

The objective isn't to recreate the missing pixels. It is to predict their abstract representations.

V-JEPA

V-JEPA extends this approach to video.

Instead of predicting exact future video frames, the model learns representations of objects, motion, interactions, and events by predicting masked portions of a video's representation.

This makes JEPA particularly relevant to learning how the physical world changes over time.

V-JEPA 2

V-JEPA 2, introduced by Meta in 2025, extends the architecture toward a world model capable of:

  • Understanding observed environments.
  • Predicting how situations may change.
  • Anticipating actions.
  • Planning toward a goal.
  • Supporting robotic control.

V-JEPA 2 is first trained through self-supervised learning using large amounts of video. Additional action-conditioned training allows the model to predict how actions affect the environment.

For robotic planning, the system can compare:

  1. The representation of the robot's current state.
  2. A representation of the desired goal.
  3. Predicted representations resulting from candidate actions.

The robot can then select actions expected to move the environment closer to the goal.

This creates an important architectural shift: 

Traditional Generative Model
Observed State
      │
      ▼
Generate Future Tokens / Pixels


JEPA World Model
Observed State
      │
      ▼
Encode Abstract State
      │
      ▼
Predict Future Abstract State
      │
      ▼
Compare with Goal
      │
      ▼
Choose Action

The approach is particularly relevant to robotics, autonomous systems, video understanding, and systems that must reason about physical cause and effect.

Meta's V-JEPA research continues to evolve, including newer V-JEPA 2.x models and work on richer representations of video and physical environments.

Official resources:

Graph Architectures

Graph Convolutional Network (GCN), Graph Neural Networks (Graph Nets), Geometric Deep Learning are designed for information represented as connected entities rather than simple sequences or grids.

Examples include:

  • Social networks.
  • Molecules.
  • Transportation networks.
  • Knowledge graphs.
  • Recommendation systems.
  • Biological networks.
  • Communication networks.

A graph architecture represents entities as nodes and relationships as edges.

Information can then propagate between neighboring nodes.

This allows the system to learn not only from individual objects but also from the relationships connecting them.

Convolutional and Capsule Architectures

Convolutional neural networks remain important in image processing, signal processing, edge systems, and specialized computer-vision applications.

Capsule Networks (CapNets) were proposed as an alternative for representing hierarchical spatial relationships between features.

Although transformers have taken over many tasks once dominated by convolutional architectures, convolution remains highly useful where efficiency, spatial locality, or specialized hardware matter.

Ensemble Architectures

An ensemble combines multiple models rather than depending on a single model.

Common strategies include:

  • Averaging model outputs.
  • Voting.
  • Weighted voting.
  • Bagging.
  • Boosting.
  • Stacking.
  • Model routing.
  • Specialized-model selection.

The models may use the same architecture or entirely different architectures.

For example: 

                    Input
                      │
        ┌─────────────┼─────────────┐
        ▼             ▼             ▼
      Model A       Model B       Model C
        │             │             │
        └─────────────┼─────────────┘
                      ▼
               Aggregation Layer
                      │
                      ▼
                 Final Output

Ensembles can improve reliability and accuracy when the errors made by the individual models aren't perfectly correlated.

The tradeoff is additional compute, memory, orchestration, and latency.

Hybrid Architectures

Hybrid architectures combine different computational approaches within one system.

Examples include:

  • Transformer + convolution.
  • Transformer + state-space model.
  • Transformer + Mixture-of-Experts (MoE).
  • Neural network + symbolic reasoning.
  • Language model + graph processing.
  • Generative model + predictive world model.
  • Cloud model + edge model.
  • Neural model + external search and tools.

Hybrid architectures are increasingly common because different components are good at different tasks.

The objective isn't architectural purity. It's effective division of work.

Hierarchical Architectures

A hierarchical architecture divides processing into multiple levels.

Lower levels may process detailed or immediate information while higher levels represent more abstract goals, concepts, or plans.

For example: 

             Strategic Goal
                   │
                   ▼
              Task Planning
                   │
                   ▼
             Action Planning
                   │
                   ▼
              Motor Action
                   │
                   ▼
                Sensors

Hierarchical designs are especially relevant to robotics, planning, autonomous systems, and longer-horizon reasoning.

Neuromorphic and Bio-Inspired Architectures

Neuromorphic Computing and Bio-inspired Computing explore architectures influenced by biological nervous systems.

These approaches may use:

  • Spiking neural networks.
  • Event-driven computation.
  • Distributed processing.
  • Local memory and computation.
  • Low-power hardware.
  • Adaptive connectivity.

The objective isn't necessarily to reproduce the brain literally. Biological systems provide architectural ideas for building machines that process information efficiently, adapt continuously, and operate under strict energy constraints.

See also Neuroscience, Symbiotic Intelligence, and Evolutionary/Genetic.

Decentralized and Distributed Architectures

Decentralized: Federated & Distributed architectures divide AI workloads across multiple systems, devices, data centers, or organizations.

Distributed architecture is now fundamental to large-scale AI because training and serving modern models may require many processing units working together.

Common approaches include:

  • Data parallelism.
  • Model parallelism.
  • Pipeline parallelism.
  • Tensor parallelism.
  • Expert parallelism.
  • Federated learning.
  • Distributed inference.
  • Edge/cloud cooperation.

Golem is one example of a distributed-computing network.

See also Service Capabilities and Processing Units - CPU, GPU, APU, TPU, VPU, FPGA, QPU.

Technologies Implemented

Modern AI systems frequently combine several architectural technologies.

Architecture / Technology Primary Function Typical Use
Transformer Uses attention to model relationships across input elements Language, multimodal AI, vision, audio, video, code
Encoder Architecture Converts input into useful representations Classification, retrieval, embeddings, perception
Decoder Architecture Generates sequences from previous context Large language models, text generation, code generation
Encoder-Decoder Maps one representation or sequence into another Translation, summarization, multimodal transformation
Mixture-of-Experts (MoE) Routes inputs to selected specialized subnetworks Large-scale sparse models
Joint Embedding Predictive Architecture Predicts hidden or future information in representation space Self-supervised learning, vision, video, world models, robotics
Convolutional Networks Learns spatially local patterns Vision, signals, edge AI
Graph Neural Networks Learns from entities and relationships Molecules, networks, recommendations, knowledge graphs
Capsule Networks Represents hierarchical spatial relationships Experimental and specialized computer vision
Ensembles Combines predictions from multiple models Reliability, classification, forecasting, model aggregation
Hybrid Architectures Combines different model types Multimodal systems, agents, reasoning, specialized AI
Hierarchical Architectures Organizes computation at different levels of abstraction Planning, robotics, autonomous systems
Neuromorphic Architecture Uses event-driven and brain-inspired computation Low-power AI, robotics, sensor processing
Distributed Architecture Divides computation or learning across machines Large-model training, inference, federated AI
Memory Maintains information beyond the immediate model calculation Agents, personalization, retrieval, persistent systems
Accelerated Compute Executes training and inference workloads GPUs, TPUs, specialized AI accelerators
Containers & Microservices Packages and orchestrates system components Production deployment, scaling, model services
Serverless Runs application functions without dedicated persistent servers APIs, event processing, AI workflows

Ecosystem Fit

AI architecture sits underneath nearly every other part of the AI ecosystem.

Model Layer

At the model layer, architecture determines how information is represented and processed.

Examples include:

  • Transformers.
  • Mixture-of-Experts (MoE).
  • Graph neural networks.
  • Convolutional networks.
  • Joint Embedding Predictive Architectures.
  • Ensembles.
  • Neuromorphic systems.
  • Hybrid systems.

These approaches aren't necessarily competitors.

A single future system could combine several of them.

Generative AI

Generative AI Stack systems usually depend on transformer-based architectures, but the surrounding stack also includes:

  • Data pipelines.
  • Embeddings.
  • Retrieval.
  • Memory.
  • Model serving.
  • Tools.
  • APIs.
  • Safety systems.
  • User interfaces.
  • Compute infrastructure.

Model architecture is therefore one layer within a much larger production architecture.

Agents and World Models

Architectures such as JEPA become particularly interesting when AI moves from generating answers toward understanding environments and selecting actions.

A world model attempts to represent:

If the world is currently in this state, what is likely to happen next — especially if I take a particular action?

That capability connects prediction to planning.

This is especially relevant to:

  • Robotics.
  • Autonomous systems.
  • Physical AI.
  • Simulation.
  • Interactive environments.
  • Long-horizon planning.

Memory and Lifelong Learning may also become important when these systems must accumulate experience over time rather than restart from a fixed pretrained state.

Enterprise Architecture

Enterprise Architecture (EA) operates above the individual AI model.

Enterprise systems must decide:

  • Which models to use.
  • Where models run.
  • How data reaches them.
  • How models access business systems.
  • How permissions are enforced.
  • How services communicate.
  • How models are monitored.
  • How systems recover from failure.
  • How AI fits existing business processes.

This connects AI model architecture with Service Capabilities, Messaging & Routing, Enterprise Portfolio Management (EPM), and Zero Trust.

Compute Infrastructure

Modern AI architectures are closely linked to their hardware.

Processing Units - CPU, GPU, APU, TPU, VPU, FPGA, QPU support different workload characteristics.

Large models may distribute computation across hundreds or thousands of accelerators.

This requires architecture for:

  • High-speed networking.
  • Memory management.
  • Distributed training.
  • Distributed inference.
  • Model partitioning.
  • Storage.
  • Fault tolerance.
  • Scheduling.

Model architecture and compute architecture increasingly have to be designed together.

Cloud, Containers, and Edge

Production AI may operate across several environments:

  • Centralized cloud infrastructure.
  • Private data centers.
  • Edge servers.
  • Personal computers.
  • Smartphones.
  • Vehicles.
  • Robots.
  • Embedded systems.

Containers; Docker, Kubernetes & Microservices allow individual AI services to be packaged and scaled independently.

Serverless architectures can handle event-driven AI functions without requiring dedicated application servers.

Edge architectures can reduce latency and keep sensitive data closer to the user, but they must operate within tighter compute, memory, and energy limits.

Sociotechnical Systems

AI architecture doesn't stop at the technical boundary.

Real systems contain:

  • People.
  • Organizations.
  • Policies.
  • Models.
  • Data.
  • Interfaces.
  • Business processes.
  • Incentives.
  • Regulations.
  • Security controls.

These are sociotechnical systems.

An AI system can therefore be technically sophisticated and still fail if its architecture doesn't account for how people actually use it.

Architecture must address both:

How does the technology work?

and

How does the technology fit into the larger human system?

Will AGI Require New Architectures?

Whether artificial general intelligence will require fundamentally new architectures remains an open research question.

One possibility is that increasingly capable systems will emerge by extending today's architectures with more compute, better training data, larger context, improved memory, stronger tool use, multimodal perception, and better learning methods.

Another possibility is that important capabilities will require architectural changes.

Current research explores several directions:

  • Mixture-of-experts — increasing capacity while activating only part of the model.
  • Hybrid architectures — combining transformers with other computational approaches.
  • Hierarchical architectures — reasoning and planning across multiple levels of abstraction and time.
  • World models — learning how environments change so an AI can predict and plan.
  • Joint Embedding Predictive Architectures — learning abstract representations through prediction rather than reconstructing raw input.
  • Neuromorphic architectures — exploring event-driven and biologically inspired computation.
  • Collective and ensemble systems — allowing several models or agents to work together.
  • Persistent memory — allowing systems to accumulate and reuse information over long periods.
  • Embodied architectures — connecting AI directly with sensors, actions, robotics, and physical environments.

It is too early to know whether one of these directions will become the dominant architecture for more general intelligence.

A more likely near-term pattern is architectural convergence: multiple specialized approaches working together inside increasingly complex AI systems.

Additional Viewing

The following long-form videos provide useful technical introductions to architectures discussed on this page. Each is longer than 20 minutes.

Transformer Architecture

Stanford CME295 Transformers & LLMs — Lecture 1: Transformer — Stanford Online, 2025.

This graduate-level lecture develops the transformer architecture from the foundations. It begins with tokenization and word representations, moves through recurrent neural networks and self-attention, and then builds the transformer architecture in detail.

It's particularly useful for understanding why attention became such an important architectural shift and how modern language models evolved from earlier sequence-processing systems.

Topics include:

  • Tokenization.
  • Embeddings.
  • Recurrent neural networks.
  • Self-attention.
  • Transformer architecture.
  • Detailed transformer examples.

Large Language Models and Mixture-of-Experts

Stanford CME295 Transformers & LLMs — Lecture 3: Transformers & Large Language Models — Stanford Online, 2025.

This lecture moves from basic transformer models into contemporary LLM architecture.

Of particular relevance to this page is its treatment of Mixture-of-Experts (MoE), including dense versus sparse expert architectures and how MoE is incorporated into large language models.

It also explains several architectural and inference concepts that matter when moving from a research model to a production system:

  • Mixture-of-experts.
  • Context length.
  • Sampling.
  • In-context learning.
  • Chain of Thought.
  • Key-value caching.
  • Inference optimization.
  • PagedAttention.
  • Multi-head latent attention.

JEPA, Self-Supervised Learning, and World Models

Yann LeCun — Self-Supervised Learning, JEPA, World Models, and the Future of AI — Harvard Center of Mathematical Sciences and Applications, 2025.

Yann LeCun presents the motivation behind Joint Embedding Predictive Architecture and explains why predictive world models may provide a different path from purely generative AI.

The lecture connects:

  • Self-supervised learning.
  • Joint embedding architectures.
  • Abstract representations.
  • World models.
  • Prediction.
  • Planning.
  • Autonomous intelligent systems.

This is the most direct long-form introduction to the architectural ideas behind JEPA and its relationship to systems intended to understand and operate in the physical world.

Retrieved from "https://primo.ai/index.php?title=Architectures&oldid=44658"