LLaMA

From PRIMO.ai
Jump to navigation Jump to search

YouTube search... ...Google search


Introduction/Overview

LLaMA (originally an acronym for "Large Language Model Meta AI," later styled Llama) is a family of large language models developed by Meta Platforms Inc. (formerly Facebook Inc.). The first release, in February 2023, came from a team at Meta's Fundamental AI Research (FAIR) lab whose paper was led by Hugo Touvron, Thibaut Lavril, Gautier Izacard, and Guillaume Lample. Later generations were built by Meta's generative AI organization, and from mid-2025 Meta's model work moved into Meta Superintelligence Labs (MSL), led by chief AI officer Alexandr Wang.

The problem it addressed. In early 2023 the strongest language models, such as the GPT-3.5 and GPT-4 systems behind ChatGPT and Google's PaLM, were reachable only through company-controlled APIs. Researchers could not inspect their internals, run them privately, or adapt them freely. LLaMA attacked that problem from two directions. Technically, it showed that a comparatively small model trained on far more text than was then customary could rival much larger ones: the 13-billion-parameter LLaMA beat the 175-billion-parameter GPT-3 on most of the paper's benchmarks. Strategically, Meta distributed the trained weights themselves rather than only an API.

The central mechanism. Every Llama model is a decoder-only Transformer that predicts the next token, refined with pre-normalization (RMSNorm), SwiGLU feed-forward layers, rotary positional embeddings (RoPE), and, from Llama 2 onward, grouped-query attention. Llama 4 (April 2025) switched to a mixture-of-experts design that activates only a fraction of its parameters for each token and processes images natively.

The paradigm shift. Llama turned the language model from a remote service into a downloadable artifact. Within about a week of the first release the weights leaked publicly, and projects such as Stanford's Alpaca and the llama.cpp runtime showed that capable models could be fine-tuned for a few hundred dollars and run on laptops, phones, and even a Raspberry Pi. Llama-style architectures, file formats, and chat templates became the default template for much of the open model ecosystem, and Meta reported more than one billion cumulative downloads by April 2025.

How it differs from the conventional approach. Llama was never "open source" in the sense defined by the Open Source Initiative. Meta releases weights and inference code under a custom community license with usage restrictions, but not the training data or the full training pipeline. That combination made Llama the archetype of the open-weight model: you get the finished model, not the recipe.

Status as of October 2026. Llama 4 Scout and Maverick (April 2025) remain the most recent Llama-branded models. After Llama 4 disappointed, Meta reorganized its AI efforts, launched a proprietary model (Muse Spark, April 2026), and then returned to open weights under the Muse name with Muse Glimmer (August 2026, Apache 2.0) and Muse Spark Open (October 2026, custom license). Meta says existing Llama models remain available; it has not announced any further Llama-branded release.


Llama turned capable language models from gated cloud services into downloadable infrastructure that anyone could run, study, and adapt. Its later history shows that releasing weights is a strategic choice a company can revise at any time, while the weights it has already released stay in the world for good.


Load video
YouTube
Load video
YouTube

Above: Meta announces Llama 3 at the Weights & Biases conference (CoreWeave channel), followed by a second Llama 3 era presentation.

Mechanisms

The Mechanism

Running a Llama model means answering one question over and over: given everything so far, what token comes next? The steps below follow a single request through a dense Llama 3 model, then describe what changed in Llama 4 and in Meta's newer Muse models.

1. Tokenization. Text is split into subword tokens. LLaMA 1 and Llama 2 used a SentencePiece byte-pair-encoding vocabulary of 32,000 tokens. Llama 3 moved to a 128,256-token vocabulary that combines roughly 100,000 tokens from OpenAI's tiktoken library with about 28,000 extra tokens for non-English text. Meta said it encodes the same text with up to 15 percent fewer tokens, which means less compute per request and more text inside a fixed context window.

2. Embedding. Each token ID selects a learned vector: 4,096 dimensions in Llama 3 8B, 8,192 in the 70B model, and 16,384 in Llama 3.1 405B.

3. The decoder stack. The vectors pass through a stack of identical blocks (32 in the 8B model, 80 in 70B, 126 in 405B). Each block does the same thing:

  • RMSNorm normalizes the input.
  • Causal self-attention, with RoPE applied to queries and keys, lets each token gather information from earlier tokens.
  • The result is added back to the input (a residual connection).
  • A second RMSNorm feeds a SwiGLU feed-forward network, whose output is again added back.

Normalizing the input to each sub-layer ("pre-normalization") rather than its output keeps deep stacks stable during training. LLaMA borrowed pre-normalization from GPT-3, SwiGLU from Google's PaLM, and RoPE from the RoFormer and GPT-Neo line of work. LLaMA also removed bias terms from its linear layers.

4. Grouped-query attention (GQA). In standard multi-head attention, every attention head keeps its own keys and values. GQA lets groups of query heads share one key/value head. In Llama 3 8B, 32 query heads share 8 key/value heads, so the model stores one quarter as much key/value data with little quality loss. Llama 2 used GQA only in its 70B model; Llama 3 applied it to every size.

5. Prediction and sampling. After the final block, one more RMSNorm and a linear "unembedding" layer produce a score (logit) for every vocabulary entry. A softmax turns the scores into probabilities, and a decoding rule (greedy, temperature sampling, or top-p "nucleus" sampling) picks the next token. That token is appended to the sequence and the loop repeats.

6. Prefill, decode, and the KV cache. Processing the prompt ("prefill") computes keys and values for every prompt token in parallel. During generation ("decode"), the model stores those keys and values in a cache and computes them only for the newest token, so each step costs roughly the same instead of growing with a full recomputation. In practice the size of this cache, not raw compute, usually limits long contexts and the number of simultaneous users a server can handle.

7. How the weights were learned. Llama 3 was pretrained on more than 15 trillion tokens of public web text, code, and multilingual data. Meta used classifiers built with Llama 2 to filter for quality, then extended the context window (to 128K tokens for Llama 3.1) and finished with an "annealing" phase on high-quality data. Post-training turns the raw next-token predictor into an assistant. Llama 2 used supervised fine-tuning followed by reinforcement learning from human feedback (RLHF) with rejection sampling and PPO. Llama 3 used several rounds of supervised fine-tuning, rejection sampling, and direct preference optimization (DPO), with large amounts of synthetic data and the 405B model acting as a teacher for smaller models.

8. Llama 4: mixture of experts and early fusion. Llama 4 replaced the single feed-forward network in some blocks with a mixture-of-experts (MoE) layer. A small router network scores each token and sends it to a few "expert" networks. In Maverick, MoE layers alternate with dense layers; each MoE layer holds 128 routed experts plus one shared expert, and every token visits the shared expert and one routed expert. The model therefore stores about 400 billion parameters but uses about 17 billion for any given token. Scout uses 16 experts and about 109 billion total parameters.

Llama 4 is also natively multimodal through "early fusion": image patches, encoded by a vision encoder derived from Meta's MetaCLIP, enter the same token stream as text from the start of pretraining. Earlier Llama 3.2 Vision models instead attached an image adapter to a finished text model. For long context, Scout uses an architecture Meta calls iRoPE, which interleaves attention layers that use no positional encoding among RoPE layers and rescales attention temperature at inference time; this is the basis of its advertised 10-million-token window. Meta trained in FP8 precision and said Maverick was "codistilled" from the much larger Behemoth teacher model.

9. The Muse era: distillation as the main pipeline. Meta's 2026 models lean heavily on distillation. Meta describes Muse Glimmer's training in three phases:

  • Pretraining by logit distillation from Muse Spark: the student learns to match the teacher's full probability distribution over the next token, not just the single most likely token.
  • Mid-training on longer-context, agent-heavy data with richer reasoning traces, mixed with organic data.
  • Post-training that combines supervised fine-tuning with on-policy distillation and reinforcement learning across reasoning, coding, and agentic tasks.

Glimmer ships with a small "drafter" network based on DFlash that proposes blocks of tokens for the main model to verify in parallel (speculative decoding). Muse Spark Open, the larger October 2026 release, is a mixture-of-experts model with 816 billion total and 42 billion active parameters and a 1-million-token context window.

Where uncertainty remains. Meta has never released Llama's training datasets, so the exact data mix, filtering rules, and share of copyrighted material are known only through Meta's descriptions and court filings. Behemoth was never released, so claims about it cannot be checked. Muse Spark's architecture is largely undisclosed, and Meta's efficiency claims for the Muse family are self-reported.

The Physics

For a language model, "physics" means the arithmetic of matrix multiplication, memory, and bandwidth. A handful of relationships explain most of what you experience when you train or run Llama.

Attention. Each attention head computes Attention(Q, K, V) = softmax(QKᵀ / √dₖ + M) · V, where the mask M blocks attention to future tokens and dividing by √dₖ keeps dot products from saturating the softmax. During prefill the cost grows with the square of the prompt length, which is why very long prompts are slow even on small models.

RMSNorm. RMSNorm(x) = g ⊙ x / √(mean(x²) + ε). Unlike LayerNorm, it skips subtracting the mean and adding a bias, which saves computation while performing comparably in practice.

SwiGLU. FFN(x) = W₂ · (SiLU(W₁x) ⊙ W₃x), where SiLU(z) = z · σ(z). The gated path lets each hidden unit scale the other path's signal. Because SwiGLU uses three weight matrices instead of two, Llama shrinks the hidden width (to roughly two thirds of the usual 4× expansion) to keep the parameter count constant.

RoPE. Rotary embeddings rotate each pair of query and key dimensions (2i, 2i+1) at position m by the angle m·θᵢ, with θᵢ = b^(−2i/d). Because rotations compose, the dot product between a query at position m and a key at position n depends only on the offset m − n: relative position is built into the geometry. Llama 2 used a base b of 10,000. Llama 3 raised it to 500,000 so that slowly rotating dimensions can still tell distant positions apart, and Llama 3.1 rescaled the frequencies to reach 128K tokens.

Training compute. Training a dense transformer costs roughly C ≈ 6 × N × D floating-point operations, where N is the parameter count and D the number of training tokens (about 2ND for the forward pass and 4ND for the backward pass). For Llama 3.1 405B, 6 × 4.05×10¹¹ × about 1.56×10¹³ ≈ 3.8×10²⁵ FLOPs, which matches Meta's reported figure and exceeds the 10²⁵ FLOP threshold at which the EU AI Act presumes a general-purpose model carries systemic risk. Meta's model card reported a cumulative 39.3 million H100 GPU-hours for the Llama 3.1 family. For a mixture-of-experts model, N is the active parameter count: Llama 4 Maverick (17 billion active parameters, about 22 trillion tokens) works out to roughly 2×10²⁴ FLOPs, less than a tenth of the 405B run.

Deliberate over-training. DeepMind's 2022 "Chinchilla" analysis found that, for a fixed training budget, about 20 training tokens per parameter minimizes loss. Llama ignored that rule on purpose. LLaMA 7B saw 1 trillion tokens (about 140 tokens per parameter) and Llama 3 8B saw about 15 trillion (nearly 1,900 per parameter); Meta reported that both its 8B and 70B models were still improving log-linearly at that point. The trade is simple: you pay extra once during training to get a smaller model that is cheaper every time it runs. That choice is a large part of why Llama models run well on consumer hardware.

Memory for weights. Stored weights take about N × bits per weight ÷ 8 bytes. The table counts weights only; the KV cache, activations, and runtime overhead come on top.

Model Parameters (total / active) 16-bit (BF16) 8-bit About 4-bit
Llama 3.2 3B 3B ~6 GB ~3 GB ~2 GB
Llama 3.1 8B 8B ~16 GB ~8 GB ~5 GB
Llama 3.3 70B 70B ~140 GB ~70 GB ~40 GB
Llama 4 Scout 109B / 17B ~218 GB ~109 GB ~60 GB
Llama 4 Maverick 400B / 17B ~800 GB ~400 GB ~220 GB
Llama 3.1 405B 405B ~810 GB ~405 GB ~230 GB
Muse Glimmer 30B over 55 GB (Meta) ~30 GB under 20 GB (Meta)
Muse Spark Open 816B / 42B ~1.6 TB ~816 GB ~460 GB

KV cache. The cache needs about 2 × layers × KV heads × head dimension × bytes per value per token. For Llama 3.1 8B (32 layers, 8 KV heads, head dimension 128, 16-bit values) that is 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or about 128 KiB per token. A full 128K-token context therefore needs about 16 GiB of cache, twice the size of the 8-bit weights. Without grouped-query attention (32 KV heads instead of 8) it would need about 64 GiB. Llama 3.1 70B, with 80 layers, needs about 320 KiB per token, or roughly 40 GiB at 128K tokens.

Bandwidth sets generation speed. When generating for a single user, each new token requires reading every active weight from memory once. That gives an upper bound of tokens per second ≤ memory bandwidth ÷ bytes of active weights. An 8B model quantized to about 4.5 GB on a GPU with about 1 TB/s of bandwidth tops out near 200 tokens per second; a 70B model at about 40 GB on a laptop-class unified-memory chip with 400 to 550 GB/s tops out near 10 to 14 tokens per second. Mixture of experts changes the bandwidth term but not the capacity term: Maverick needs memory for 400 billion parameters but reads only about 17 billion per token. Meta used exactly this reasoning for Muse Spark Open, which runs on two linked Apple M5 Ultra Mac Studios: together they provide about a terabyte of unified memory to hold the model, while about 1.2 TB/s of bandwidth per machine is enough to read the small active slice each token needs.

Quantization error. Uniform quantization with step size Δ adds error with variance of about Δ²/12; each extra bit halves Δ and cuts that variance by four. LLM weights contain outliers, so modern formats quantize small blocks with their own scale factors. GGUF's Q4_K_M format averages roughly 4.8 bits per weight once scales are included, and the OCP MXFP4 format stores 32 four-bit values with one shared 8-bit exponent (4.25 bits per weight). In practice 8-bit is close to lossless; 4-bit usually costs a little quality on large models and more on small models, long reasoning chains, and code; below about 3 bits, quality falls quickly. Meta reported "minimal to no degradation" on agentic tasks for Muse Glimmer's 4-bit build.

Experimental constraints. Advertised context length is not usable context. Scout advertises 10 million tokens, but independent long-context comprehension tests published after launch found Llama 4 models degrading sharply at lengths far below that limit. Benchmark scores are also sensitive to contamination, prompt formatting, and chat tuning, as the Llama 4 leaderboard episode (see Recent Developments) showed.

Why This Matters

Scientific importance. Llama gave researchers a strong model whose internals they could actually probe. Interpretability studies, quantization research, parameter-efficient fine-tuning methods such as LoRA and QLoRA, and many safety studies used Llama weights as standard test subjects, which is impossible with API-only models. "The Llama 3 Herd of Models" (2024) remains one of the most detailed public accounts of frontier-scale training.

Engineering implications. Llama created the local-LLM software stack: llama.cpp and its GGUF file format, Ollama, LM Studio, and the Llama-specific optimizations in serving engines such as vLLM and SGLang. Those tools now serve nearly every open model, not just Llama. OpenAI-compatible local endpoints let developers swap between cloud and local models with a one-line change.

Practical applications. Llama models power private on-premises assistants, retrieval-augmented generation over internal documents (for example the TIB llama_index_mediawiki-service for wiki users listed above), on-device assistants built on the 1B and 3B Llama 3.2 models, and domain fine-tunes in medicine, law, and software engineering. In November 2024 Meta opened Llama to U.S. government agencies and defense contractors.

Market and policy consequences. Llama's license pattern (free use, a cap for very large companies, attribution, and an acceptable use policy) became a template that others adapted, including Moonshot AI's Kimi K3 license and Meta's own Muse Model License, both of which gate large commercial model-serving businesses. Llama also anchored the policy debate over openly released weights. Reuters reported in 2024 that researchers linked to China's military had built a military-oriented tool on a Llama 2 model, a reminder that license terms cannot control use once weights are public.

Limitations. Llama is not open source by the OSI definition; its training data is undisclosed; its license excludes the largest companies and, for multimodal models, entities based in the EU; safety training can be removed by fine-tuning; and Llama 4 fell short of Meta's own expectations and of rival open models from China.

What would change if the open approach proves out. If frontier-class capability keeps arriving in downloadable form a few months behind the best closed systems, as Epoch AI's measurements suggest, AI capability becomes infrastructure in the way Linux did. Prices fall toward the cost of hardware, and governance has to focus on uses and deployments rather than on a few model providers. If Meta's Muse experience instead shows that the best models must stay closed to pay for their development, open weights settle into a permanent role one tier below the frontier.

What remains uncertain. Whether Meta will release the weights of its strongest Muse models; whether the Llama brand will return; how courts will ultimately treat training on copyrighted books; how U.S. distillation and export policy will shape open releases; and whether open models can ever be made resistant to having their safeguards fine-tuned away.

Model Generations

Generation Released Sizes Context Key changes License
LLaMA (1) Feb 24, 2023 7B, 13B, 33B, 65B 2K Trained only on public data (1.0 to 1.4T tokens); RMSNorm, SwiGLU, RoPE Noncommercial research license; weights leaked within about a week
Llama 2 Jul 18, 2023 7B, 13B, 70B (base and chat) 4K 2T tokens; GQA on 70B; RLHF-tuned chat models Llama 2 Community License (commercial use, 700M monthly user cap)
Code Llama Aug 24, 2023 7B, 13B, 34B; 70B in Jan 2024 up to 100K Llama 2 further trained on about 500B code tokens Llama 2 license
Llama 3 Apr 18, 2024 8B, 70B 8K 15T+ tokens; 128K-token vocabulary; GQA at every size Llama 3 Community License
Llama 3.1 Jul 23, 2024 8B, 70B, 405B 128K First openly downloadable model Meta called frontier-level; eight languages; tool calling Llama 3.1 license; outputs may be used to improve other models
Llama 3.2 Sep 25, 2024 1B, 3B (text); 11B, 90B (vision) 128K On-device models; first multimodal Llama; Llama Stack Llama 3.2 license; multimodal rights excluded for EU-based entities
Llama 3.3 Dec 6, 2024 70B 128K Post-training gains approaching 405B quality on many tasks Llama 3.3 license
Llama 4 Apr 5, 2025 Scout (17B active, 109B total, 16 experts); Maverick (17B active, 400B total, 128 experts); Behemoth previewed (288B active, about 2T total), never released 10M (Scout), 1M (Maverick) First mixture-of-experts and natively multimodal Llama Llama 4 Community License

LLaMA 1 (February 2023). Four sizes trained only on publicly available sources: CommonCrawl, C4, GitHub, Wikipedia, ArXiv, StackExchange, and books (including the Books3 collection later central to copyright litigation). The 7B and 13B models saw 1.0 trillion tokens; the 33B and 65B models saw 1.4 trillion. Meta offered weights case by case to researchers under a noncommercial license, but a torrent of the weights appeared on 4chan about a week later. The community built on the leak rather than ignoring it: Stanford's Alpaca fine-tuned the 7B model on 52,000 instruction-following examples for under $600, and Vicuna, Koala, and many others followed.

Llama 2 (July 2023). Released with Microsoft as a launch partner, Llama 2 allowed free commercial use for any company with fewer than 700 million monthly active users. It doubled the training data to 2 trillion tokens, doubled the context window to 4K, and added chat models aligned with RLHF. Code Llama followed in August 2023.

Llama 3 (April 2024). See the next section for the launch details as originally documented on this page.

Llama 3.1 (July 2024). The 405B model was the first openly downloadable model that Meta compared directly with GPT-4o and Claude 3.5 Sonnet. All three sizes gained a 128K context window, support for eight languages, and tool calling. The license changed to allow developers to use Llama outputs to improve other models, provided derivative model names begin with "Llama." Mark Zuckerberg's accompanying essay, "Open Source AI Is the Path Forward," argued that open models would follow the path of Linux and predicted that future Llama models would lead the industry.

Llama 3.2 (September 2024). Added 1B and 3B text models for phones and edge devices and 11B and 90B vision models, Llama's first multimodal release, along with the Llama Stack developer toolkit. The multimodal license excluded individuals and companies based in the EU, which Meta attributed to an unpredictable European regulatory environment.

Llama 3.3 (December 2024). A 70B text model that Meta said approached 405B performance on many tasks at a fraction of the serving cost.

Llama 4 (April 2025). Scout and Maverick shipped and Behemoth was previewed; see Recent Developments.

Llama 3 at Launch (April 2024)

When Llama 3 launched in April 2024, it represented a significant advancement in open-weight large language models from Meta, establishing the company as a leader in this space with highly capable and scalable models that were widely accessible. The key points at the time were:

  • LLaMA 3 models were released in 8 billion and 70 billion parameter sizes, with an 8K context window, representing a significant increase in capability compared to the previous LLaMA 2 models. (Zuckerberg said the 8B model was nearly as powerful as the largest Llama 2 model.)
  • The LLaMA 3 models were trained on over 15 trillion tokens of data, 7 times more than the LLaMA 2 models, including 4 times more code data. This resulted in major improvements in performance on benchmarks like MMLU, GSM8K, and HumanEval.
  • Key new capabilities of LLaMA 3 included enhanced reasoning, code generation, and instruction following, as well as improved safety features like reduced false refusal rates and increased response diversity.
  • Meta was also training even larger LLaMA 3 models over 400 billion parameters, with multimodal and multilingual capabilities planned. Outcome: this became Llama 3.1 405B (July 2024), a text-only model supporting eight languages. Image understanding arrived separately in the Llama 3.2 Vision models (September 2024).
  • The LLaMA 3 models were made openly available by Meta to the developer community, with support from major hardware providers like Intel, Qualcomm, and AMD. Note: "openly available" means open weights under the Llama 3 Community License, not open source in the OSI sense; see Open Weights.

Recent Developments

Llama 4 and the benchmark controversy (April 2025)

Meta released Llama 4 Scout and Maverick on Saturday, April 5, 2025, and previewed Behemoth (288 billion active parameters, 16 experts, nearly 2 trillion total), which it said was still training. Meta's claims were ambitious: Scout fit on a single H100 GPU with 4-bit quantization and beat Gemma 3, Gemini 2.0 Flash-Lite, and Mistral 3.1 on widely reported benchmarks; Maverick beat GPT-4o and Gemini 2.0 Flash; and Behemoth beat GPT-4.5, Claude Sonnet 3.7, and Gemini 2.0 Pro on several STEM benchmarks. Meta said the overall pretraining mixture exceeded 30 trillion tokens across 200 languages; the model cards list about 40 trillion tokens for Scout and 22 trillion for Maverick.

What was tested and what went wrong. Meta's announcement highlighted an Elo score of 1417 for Maverick on LMArena, a crowdsourced leaderboard where people vote between anonymous model outputs, placing it second overall. Researchers then noticed fine print saying the tested model was "Llama-4-Maverick-03-26-Experimental," a chat-optimized variant that Meta had not released. LMArena said Meta should have made clearer that the entry was customized to optimize for human preference, published data from the matchups, and changed its policies. When LMArena scored the actual release, "Llama-4-Maverick-17B-128E-Instruct," it ranked around 32nd, below months-old models including GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro. Meta's head of generative AI, Ahmad Al-Dahle, denied rumors that the models had been trained on benchmark test sets and attributed uneven early quality to implementations that were still being stabilized.

Long-context results. Independent long-context comprehension tests published shortly after launch found that Maverick did no better than Llama 3.3 70B and that Scout performed poorly, despite their 1-million and 10-million-token advertised windows.

What this does and does not establish. The episode does not show that Meta's standard benchmark numbers were fabricated. It does show that preference-based leaderboards reward style and formatting that can be tuned for, that labs can submit variants different from what they release, and that a reported score means little unless it is tied to the exact downloadable checkpoint. A 2026 arXiv survey, "The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notes," consolidates the publicly reported technical details for anyone trying to separate verified facts from launch claims. Meanwhile, adoption momentum shifted: Alibaba's Qwen3 arrived weeks later under Apache 2.0, and by September 2025 Qwen had overtaken Llama in cumulative Hugging Face downloads, according to the ATOM Project's measurements.

LlamaCon. At its first developer conference on April 29, 2025, Meta reported more than one billion Llama downloads and announced a preview of the Llama API (with fast inference from Cerebras and Groq and compatibility with the OpenAI SDK), new Llama Stack integrations, and new protection tools: Llama Guard 4, LlamaFirewall, Llama Prompt Guard 2, and CyberSecEval 4.

Behemoth stalls and Meta reorganizes (May to December 2025)

The Wall Street Journal reported in May 2025 that Behemoth's release had slipped from April to June and then to the fall or later over concerns about its capabilities; it was never released. Reports at the time noted that most of the researchers on the original 2023 LLaMA paper had left Meta.

In June 2025 Meta invested about $14.3 billion for a 49 percent stake in Scale AI and brought in its co-founder Alexandr Wang as chief AI officer of a new unit, Meta Superintelligence Labs, formed on June 30, 2025. In a July 2025 letter, Zuckerberg wrote that Meta would need "to be rigorous about mitigating these risks and careful about what we choose to open source," although a spokesperson said Meta's position on open source was unchanged. That same month Meta declined to sign the EU's voluntary Code of Practice for general-purpose AI. In October 2025 Meta cut about 600 jobs in its AI organization, including at FAIR, and in November 2025 Yann LeCun, Meta's chief AI scientist and FAIR's founding director, announced he would leave at year's end to found a startup focused on what he calls Advanced Machine Intelligence, with Meta as a partner. In an October 2026 interview, Wang described Llama 4 as a clearly disappointing result inside Meta.

From Avocado to Muse Spark: Meta goes closed (December 2025 to July 2026)

In December 2025 CNBC reported that Meta's next frontier model, code-named "Avocado," might launch as a proprietary model, a reversal of the strategy Zuckerberg had championed. Bloomberg reported that its development drew on models from competitors, including Alibaba's Qwen and Google's Gemma.

On April 8, 2026, Meta launched Muse Spark, the first model from Meta Superintelligence Labs, built on a training stack rebuilt over nine months. It is a natively multimodal reasoning model with a "Contemplating" mode in which several sub-agents reason in parallel and then merge their answers, and Meta said it worked with more than 1,000 physicians on health reasoning. Meta claimed Muse Spark matched Llama 4 Maverick's capability with more than ten times less compute; that claim is self-reported. Muse Spark was not released as open weights. It powered the Meta AI app and website first, rolled out to WhatsApp, Instagram, Facebook, Messenger, and Ray-Ban Meta glasses, and was offered to select partners through a private API preview. Meta said that "current Llama models will continue to be available," which commentators noted said nothing about future Llama development. In May 2026 Wang said Muse Spark had triggered safety checks that made it not suitable for open sourcing.

The Muse line then iterated quickly: Muse Spark 1.1 (July 2026) launched alongside the Meta Model API, Muse Spark 1.2 (August 2026) launched with Muse Code, a terminal-based coding agent, and Muse Spark 1.3 (September 2026) scored 61 on Artificial Analysis's Intelligence Index, near the top of the field.

The return to open weights: Muse Glimmer and Muse Spark Open (August to October 2026)

On August 10, 2026, Meta released Muse Glimmer and Zuckerberg published a 6,500-word essay, "The Future is for Everyone." The essay argued that concentrating advanced AI in a few companies or governments is the larger risk, urged Washington to rethink its policies on distillation and training-data use so that American open models can compete with Chinese ones, and opposed restricting access to foreign open models. Meta also promised open weights for Muse Spark 1.2.

  • Muse Glimmer is a 30-billion-parameter dense model (a 28B text decoder plus a roughly 2B perception encoder for images) released under the permissive Apache 2.0 license, a notable break from the custom Llama licenses. It is built for "always-on" local agents: tool calling, multi-step coding, recovery from failed tool calls, and adjustable reasoning effort, across more than 100 languages. Quantized to about 4 bits, the language model fits under 20 GB, leaving room for the KV cache, image encoder, and speculative-decoding drafter within a 24 GB or 32 GB GPU. Meta reported decoding speedups from the drafter of 3.1x on an RTX 5090, 1.8x on an M5 Max, and 1.5x on an M4 Max. Llama.cpp, MLX, ExecuTorch, vLLM, SGLang, Ollama, LM Studio, and Unsloth added support, and hosted access came through Together AI, Fireworks AI, and OpenRouter.
  • What was tested. Meta compared Glimmer with Gemma4-31B and Qwen3.6-27B using its own evaluation harness and reported leading results on agentic benchmarks such as MCP-Atlas and SWE-Bench Pro. An independent community benchmark run on a single Apple M1 Max through Ollama found a more mixed picture: on 12 core agentic tasks all models were close (Glimmer passed 12 of 12), but on a larger 30-task suite including data-science coding, Qwen 3.8 27B passed all 30 tasks to Glimmer's 24 and ran about 2.1 times faster, with several Glimmer failures caused by tool loops that exceeded the turn budget. That test is small, single-machine, and author-designed, so it is a useful check rather than a verdict: Glimmer looks competitive for its size, not dominant.

On October 15, 2026, Meta released Muse Spark Open, which it called "the most capable US open-weight model to date."

  • Specifications. A natively multimodal mixture-of-experts model with 816 billion total and 42 billion active parameters and a 1-million-token context window, built for coding, agents, and long-horizon tasks. Weights are available from Hugging Face (as a gated repository), Amazon SageMaker AI, Google Cloud Model Garden, and Microsoft Foundry, in BF16 and NVFP4 builds; hosted access comes from Baseten, Fireworks AI, Groq, Modal, Nebius, OpenRouter, and Together AI. Meta demonstrated it running on two Apple M5 Ultra Mac Studios with MLX.
  • Safety. Meta said the open version received additional safety post-training and met the release thresholds of its Meta Superintelligence Scaling Framework, with a full evaluation report promised later.
  • License. The Muse Model License permits free use, modification, and redistribution but requires every product built on the model or a derivative to display "Built with Muse Spark Open." Companies whose revenue, with affiliates, exceeds $20 million over any 12 months must notify Meta and sign a separate agreement before offering the model as a hosted service. Derivatives are defined broadly enough to include models trained on its outputs, and the usage policy bars military and critical-infrastructure uses.
  • Standing. Meta's own comparison placed it behind China's GLM-5.3, Kimi K3, and Qwen3.8 on Artificial Analysis's Intelligence Index but ahead of other U.S. open models; independent confirmation was pending at release. Meta's strongest models, Muse Spark 1.3 and its newer closed flagship, remain without public weights.

Legal and policy developments

  • Copyright. In Kadrey v. Meta, 13 authors including Richard Kadrey and Sarah Silverman alleged that Meta trained Llama on pirated copies of their books downloaded from shadow libraries. On June 25, 2025, Judge Vince Chhabria granted Meta summary judgment on fair use because the plaintiffs failed to show market harm, while stressing that the ruling did not mean Meta's use was lawful and suggesting that a "market dilution" theory could succeed in other cases. Claims about distributing pirated files through torrenting were left unresolved. Two days earlier, in Bartz v. Anthropic, Judge William Alsup held that training was fair use but that building a library from pirated copies was not. The broader legal question remains open.
  • Europe. Llama 3.2, 3.3, and 4 withhold multimodal model rights from EU-based individuals and companies, although EU end users may use products built on them. Under the EU AI Act, a model trained with more than 10²⁵ FLOPs, such as Llama 3.1 405B, is presumed to carry systemic risk and cannot claim the Act's open-source exemptions.
  • United States. After Reuters reported that researchers linked to China's military had built on a Llama 2 model, Meta opened Llama to U.S. agencies and defense contractors in November 2024. In 2026 the policy debate shifted to distillation and to Chinese open models, the context for Zuckerberg's August essay; see Open Weights for details.

Open questions

  • Will Meta release weights for Muse Spark 1.3 or later frontier models, or keep its open releases a step behind its closed ones?
  • Will the Llama name return, or is Llama 4 the final Llama generation?
  • Will Meta's claimed efficiency gains for the Muse family hold up under independent evaluation?
  • Will Llama 4's advertised long-context capabilities ever be matched by effective long-context performance in open models?
  • How will U.S. distillation policy and pending legislation affect models trained partly on other models' outputs?

Licensing and Openness

The Llama Community License (Llama 2 onward, with version-specific changes) grants broad rights with several conditions:

  • Free use, modification, and redistribution, including commercial use.
  • A scale cap. A company whose products had more than 700 million monthly active users in the month before a version's release must request a separate license, which Meta may grant at its sole discretion.
  • Attribution. Products must prominently display "Built with Llama," and from Llama 3.1 onward, derivative models must have names beginning with "Llama."
  • Acceptable use policy. Prohibits uses such as weapons development, illegal activity, and deception.
  • Distillation. From Llama 3.1, outputs may be used to train or improve other models, subject to the naming rule.
  • EU multimodal restriction. For Llama 3.2, 3.3, and 4 multimodal models, rights are not granted to individuals domiciled in the EU or companies with their principal place of business there. Non-EU companies may still distribute products in the EU, and EU-based employees of non-EU companies may work with the models.

The Open Source Initiative does not consider these terms open source, because they discriminate against particular users and restrict fields of use, and because Meta does not release training data or full training code. "Open weights" is the accurate description.

Release License Commercial use Main conditions
Llama 2 to Llama 4 Llama Community License Yes, below 700M monthly active users Attribution, naming rule, acceptable use policy, EU multimodal exclusion
Muse Glimmer (Aug 2026) Apache 2.0 Yes Standard notices; no usage caps
Muse Spark Open (Oct 2026) Muse Model License Yes, with a $20M gate for hosted model services Attribution, broad derivative definition, usage policy barring military and critical-infrastructure uses

Implement a Chat with LLaMA

When Meta shares the "weights" of the LLaMA model, they are providing the parameters that have been learned during the training process, which include embedding, self-attention, feed-forward, and normalization weights (Llama's linear layers omit bias terms). These weights are essential for the model to function correctly and are what enable the model to process natural language and generate coherent and contextually relevant text. Together with the model's configuration file, tokenizer, and chat template, they make up a complete, working model you can run on your own hardware.

To implement a chat locally on your machine using the weights from Meta for LLaMA, you will need to follow these steps. By following these steps, you can set up and run a LLaMA-based chat locally on your machine, allowing you to interact with the model and develop applications that leverage its natural language processing capabilities.

Step 1: Choose a model and download the pretrained weights. Obtain the pretrained model weights from official sources such as Meta's webpage (llama.com), GitHub, Hugging Face, Kaggle, or Ollama. On Hugging Face, the meta-llama repositories are gated: sign in, accept the license on the model page, and authenticate with an access token before downloading. Pick a size your hardware can hold (see the memory table in The Physics):

  • Laptop or phone, under 8 GB of memory: Llama 3.2 1B or 3B.
  • Single consumer GPU with 8 to 16 GB: Llama 3.1 8B Instruct, quantized to 4 or 8 bits.
  • Workstation with 48 GB or more, or a high-memory Mac: Llama 3.3 70B Instruct at 4 bits.
  • Multi-GPU server: Llama 4 Scout or Maverick, or Llama 3.1 405B.
  • Newer Meta open weights: Muse Glimmer (30B, Apache 2.0) fits a single 24 GB GPU at 4 bits; Muse Spark Open needs data-center hardware or two top-end Mac Studios.

Step 2: Set up your local environment. Ensure that your local machine has the necessary hardware: a recent CPU, and as much GPU memory (or Apple unified memory) as possible. If you have enough GPU memory, you can run larger models with full 16-bit precision. CPU-only inference works with quantized GGUF files through llama.cpp or Ollama, but is much slower. Remember to budget memory for the KV cache as well as the weights, especially for long conversations.

Step 3: Install required libraries and dependencies. Use Python to write the script for setting up and running the model. Install the transformers and accelerate libraries from Hugging Face, plus PyTorch, and log in so gated downloads work: 

pip install -U transformers accelerate torch
huggingface-cli login        # newer huggingface_hub versions also accept: hf auth login

Step 4: Write your Python script. Earlier versions of this page imported LlamaForCausalLM and LlamaTokenizer. Those classes still work for Llama 2 and 3, but the Auto classes are now recommended because they pick the right architecture automatically (Llama 4 uses a different, multimodal class) and apply the model's chat template, which inserts the special header and end-of-turn tokens Llama expects. Save the following as chat_llama.py: 

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "meta-llama/Llama-3.1-8B-Instruct"   # gated: accept the license on Hugging Face first

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,   # full 16-bit precision, about 16 GB of GPU memory
    device_map="auto",            # spreads layers across available GPUs and CPU
)

messages = [
    {"role": "system", "content": "You are a concise technical assistant."},
    {"role": "user", "content": "Explain grouped-query attention in two sentences."},
]

# The chat template adds Llama's special header and end-of-turn tokens.
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

output = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.6, top_p=0.9)
prompt_length = inputs["input_ids"].shape[-1]
print(tokenizer.decode(output[0][prompt_length:], skip_special_tokens=True))

Step 5: Run the model locally. Save your Python script and execute it using the command python chat_llama.py. Provide different prompts as input to generate responses and test the model's performance. To hold a multi-turn conversation, append each assistant reply and the next user message to the messages list and generate again.

Step 6: Use open-source tools for local execution and serving. Hugging Face's Transformers library pulls models directly from the Hugging Face Hub, as above. For an application that several programs or users share, run an OpenAI-compatible server instead: 

# vLLM: high-throughput GPU serving with an OpenAI-compatible API on port 8000
pip install vllm
vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 32768

# llama.cpp (https://github.com/ggml-org/llama.cpp): run a quantized GGUF file on CPU, GPU, or Apple silicon
llama-cli -m Llama-3.1-8B-Instruct-Q4_K_M.gguf -cnv
llama-server -m Llama-3.1-8B-Instruct-Q4_K_M.gguf --port 8080

Any client written for the OpenAI API can then talk to your local model by changing its base URL.

Step 7: Build an interactive chat interface. For an interactive chat interface, you can wrap the model inside Gradio. Install it with pip install gradio and run: 

import gradio as gr
from transformers import pipeline

pipe = pipeline("text-generation", model="meta-llama/Llama-3.2-3B-Instruct", device_map="auto")

def chat(message, history):
    messages = []
    for turn in history:                      # works with both Gradio history formats
        if isinstance(turn, dict):
            messages.append({"role": turn["role"], "content": turn["content"]})
        else:
            user, bot = turn
            messages += [{"role": "user", "content": user}, {"role": "assistant", "content": bot}]
    messages.append({"role": "user", "content": message})
    result = pipe(messages, max_new_tokens=512)
    return result[0]["generated_text"][-1]["content"]

gr.ChatInterface(chat).launch()

Step 8: Run LLaMA through Ollama. Ollama is one of the easiest ways to run Llama models locally, and it now supports Windows as well as Linux and macOS. It downloads quantized models and exposes a local REST API: 

ollama run llama3             # the original command still works (Llama 3 8B)
ollama run llama3.1:8b        # 128K-context Llama 3.1
ollama run llama3.2:3b        # small model for laptops and edge devices

# Original single-prompt API example:
curl -X POST http://localhost:11434/api/generate -d '{ "model": "llama3", "prompt":"Why is the sky blue?" }'

# Chat API with message history:
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [{"role": "user", "content": "Why is the sky blue?"}],
  "stream": false
}'

Check the Ollama model library for current Llama 4 and Muse Glimmer tags rather than guessing names.

Step 9: Use quantization to reduce model size. If necessary, you can reduce the size of the models while largely maintaining performance by quantizing their parameters. A 4-bit model needs about a quarter of the memory of its 16-bit version (see The Physics for the error trade-offs). Ollama and GGUF files from llama.cpp come pre-quantized; Q4_K_M is a common default and Q8_0 is nearly lossless. In Transformers you can quantize on load with bitsandbytes: 

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.3-70B-Instruct", quantization_config=bnb, device_map="auto"
)   # roughly 40 GB of GPU memory instead of about 140 GB

Step 10: Customize with fine-tuning. To specialize a model for your domain, train a small LoRA adapter rather than the full model. QLoRA keeps the base model frozen in 4-bit form and trains only the adapter, which makes fine-tuning an 8B model practical on a single consumer GPU. Common tools include Hugging Face PEFT and TRL, Unsloth, and Meta's PyTorch-based TorchTune and TorchTitan. Check the license before distributing a fine-tune: Llama derivatives must carry "Llama" at the start of their names.

AI Agents with LLaMA

An AI agent wraps a language model in a loop: the model plans, calls tools (search, code execution, databases, APIs), reads the results, and decides what to do next until the task is done. Llama became a common base for agents once it gained reliable tool calling.

Tool calling in Llama. Llama 3.1 introduced built-in tool support. Its chat format defines an ipython role for tool results, built-in tools for web search (Brave Search), Wolfram Alpha, and a Python code interpreter (triggered by the <|python_tag|> token), and JSON-formatted calls to custom functions that you describe in the system or user prompt. The small Llama 3.2 models brought tool calling to on-device agents, and Llama 4 Maverick added image understanding for agents that must read screenshots or documents.

A minimal tool call through Ollama. The model decides whether to call your function and returns structured arguments; your code runs the function, appends the result as a message with the role tool, and asks the model again: 

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "stream": false,
  "messages": [{"role": "user", "content": "What is the weather in Paris right now?"}],
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_current_weather",
      "description": "Get the current weather for a city",
      "parameters": {
        "type": "object",
        "properties": {"city": {"type": "string", "description": "City name"}},
        "required": ["city"]
      }
    }
  }]
}'

Frameworks. Llama models work with LangChain, LlamaIndex (an independent project despite its name, used by the TIB MediaWiki service listed above), and any framework that speaks the OpenAI API, through local servers such as Ollama, vLLM, or llama.cpp. Meta's own Llama Stack provides standard APIs for inference, safety, memory, and agents.

Safety for agents. Agents that read web pages or documents can be hijacked by instructions hidden in that content (prompt injection). Meta's PurpleLlama project provides open safeguards: Llama Guard 4, a 12B multimodal classifier that screens inputs and outputs; Llama Prompt Guard 2, small classifiers (86M and 22M parameters) that detect jailbreak and injection attempts; and LlamaFirewall, a framework that combines Prompt Guard with AlignmentCheck, which audits an agent's reasoning for goal hijacking, and CodeShield, which scans generated code for insecure patterns. CyberSecEval 4 benchmarks the security behavior of models.

The 2026 picture. Meta's newest agent-oriented open model is Muse Glimmer, trained specifically for long-running local agents, precise tool calls, and recovery from failed tool calls, and tested with scaffolds such as OpenClaw and OpenCode. Llama 3.x models remain popular for lightweight local agents because of their mature tooling.

Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube

Additional Viewing and References

Additional Viewing

Load video
YouTube

LLaMA explained: KV-Cache, Rotary Positional Embedding, RMS Norm, Grouped Query Attention, SwiGLU (Umar Jamil, August 2023, about 70 minutes). A whiteboard-style walkthrough of the LLaMA 1 and Llama 2 architecture that starts from the original Transformer and explains each Llama change: RMSNorm, rotary embeddings, the KV cache, grouped-query attention, and SwiGLU. It is the best visual companion to The Mechanism and The Physics sections of this page. Watch on YouTube

Load video
YouTube

Coding LLaMA 2 from scratch in PyTorch (Umar Jamil, 2023, more than two hours). The companion coding session implements Llama 2 line by line in PyTorch, including the KV cache, grouped-query attention, rotary embeddings, and RMSNorm, and loads Meta's weights into the result. Pair it with the hkproj/pytorch-llama repository if you want to understand exactly what a weights file contains and how inference uses it. Watch on YouTube

Load video
YouTube

Mark Zuckerberg: Llama 3, $10B models, Caesar Augustus, & 1 GW datacenters (Dwarkesh Patel podcast, April 2024, about 78 minutes). Recorded on Llama 3's launch day, Zuckerberg explains Meta's reasons for open-sourcing models, the training choices behind Llama 3, energy limits on scaling, and when Meta might stop releasing weights. It is a useful baseline for judging how far Meta's strategy moved by 2026. Watch on YouTube

Load video
YouTube

Mark Zuckerberg: AI will write most Meta code in 18 months (Dwarkesh Patel podcast, April 2025, about 75 minutes). Recorded just after Llama 4's launch, it covers how Llama 4 compares with rivals, benchmark gaming, DeepSeek and China, and Meta's view of open-source AI. Watch it alongside the Recent Developments section to see Meta's public position right before its reorganization. Watch on YouTube

References

Retrieved from "https://primo.ai/w/index.php?title=LLaMA&oldid=53394"