Reinforcement Learning (RL)

From PRIMO.ai
Jump to navigation Jump to search

YouTube ... Quora ...Google search ...Google News ...Bing News

Introduction/Overview

Reinforcement Learning (RL) A technique that teaches an AI model to find the best result through trial and error and receiving rewards or punishments based on its results, often enhanced by human feedback for games and complex tasks.

More precisely, reinforcement learning is the branch of machine learning concerned with learning to make sequences of decisions. An agent observes its situation, takes an action, receives a numerical reward, and lands in a new situation. Over many such steps it learns a policy, a rule for choosing actions, that maximizes the total reward it collects over time. Human feedback is one source of reward, central to modern chat assistants, but many of RL's biggest successes (Go, chess, Atari games, sorting algorithms) used rewards computed automatically by the environment, such as winning a game or producing correct output.


DeepMind researchers have argued that reinforcement learning, an agent maximizing reward through trial and error, may be enough to reach Artificial General Intelligence (AGI). Some scientists believe that assembling multiple narrow AI modules will produce higher intelligent systems. RL's recent role in training reasoning models has strengthened the "reward is enough" case, but it remains a hypothesis: designing the right rewards, and keeping agents from exploiting flawed ones, are unsolved problems.


big_thumb.jpg

The idea has deep roots. Psychologist Edward Thorndike's 1898 "law of effect" held that actions followed by satisfying outcomes become more likely; Richard Bellman's 1950s dynamic programming supplied the mathematics of optimal sequential decisions; and Arthur Samuel's 1959 checkers program learned by playing against itself. Andrew Barto and Richard Sutton, beginning at the University of Massachusetts Amherst around 1980, built the modern field: temporal-difference learning (1988), actor-critic methods, and the standard textbook Reinforcement Learning: An Introduction (1998; second edition 2018). They received the 2024 ACM Turing Award for this work. Chris Watkins introduced Q-learning in 1989, Gerald Tesauro's TD-Gammon reached expert backgammon in 1992, and DeepMind combined RL with deep neural networks in the deep Q-network (2013 to 2015), followed by AlphaGo (2016), AlphaGo Zero and AlphaZero (2017), and MuZero (2020). Since 2022, RL has become central to large language models, first through reinforcement learning from human feedback and, since late 2024, through RL on verifiable problems that produces "reasoning" models.

RL addresses a problem that other learning methods do not: how to act well when no one can tell you the correct action, feedback arrives late, and your own choices determine what you get to see next. A move in chess may only prove good or bad dozens of moves later; deciding which earlier choices deserve the credit is the "credit assignment" problem. The agent must also balance exploration (trying new actions to learn about them) against exploitation (using what already works).

The central mechanism is a feedback loop: act, observe the reward and the new state, update estimates of how good states and actions are, and adjust the policy. That differs from supervised learning, which learns from labeled examples of correct answers, and from unsupervised learning, which looks for structure in unlabeled data. In RL there are no labels, only rewards, and the data depends on the agent's own behavior. The paradigm shift RL brought is that a system can exceed the knowledge in its training data: AlphaZero surpassed centuries of human chess and Go knowledge by learning entirely from self-play.

The Mechanism

Reinforcement learning formalizes the loop between agent and environment, usually as a Markov decision process (MDP).

  1. Observe. At time t, the agent observes state st (a board position, a robot's sensor readings, the text of a conversation).
  2. Act. It chooses action at according to its policy π(a | s), which may be deterministic or a probability distribution.
  3. Receive feedback. The environment returns a reward rt+1 and moves to a new state st+1.
  4. Learn. The agent updates its value estimates, its policy, or a model of the environment.
  5. Repeat until the episode ends (a game finishes, a task is completed) or indefinitely in continuing tasks.

Major components

  • Policy: the agent's behavior, mapping states to actions.
  • Reward signal: the immediate feedback defining the goal.
  • Value function: a prediction of total future reward from a state (V) or from a state-action pair (Q). Values let the agent prefer actions whose payoff comes later.
  • Model (optional): the agent's internal prediction of how the environment responds. Model-based methods plan with it; model-free methods learn directly from experience. See World Models.

Families of methods

  • Value-based methods, such as Q-learning, SARSA, and DQN, learn action values and act greedily on them.
  • Policy-based methods, such as REINFORCE and other policy gradient methods, adjust the policy directly to make high-reward actions more likely.
  • Actor-critic methods combine both: an actor (policy) and a critic (value estimate) that judges the actor's choices. PPO and TRPO are actor-critic methods that limit how far each update can move the policy.
  • Model-based methods learn or are given a model and plan with it, from AlphaZero's tree search to DreamerV3's "imagined" rollouts.
  • Learning from human data includes imitation learning, inverse reinforcement learning (inferring the reward from expert behavior), and learning from preferences, the basis of RLHF.
  • Offline RL learns from a fixed dataset of past interactions without further exploration, important where trial and error is dangerous or costly, such as medicine.

Dependencies and conditions. RL works best when the reward is well specified, the agent can try many episodes (usually in a simulation or game), and states contain the information needed to decide. In the real world, rewards are sparse or hard to measure, exploration can be unsafe, and policies trained in simulation often degrade on physical hardware (the "sim-to-real gap").

Where uncertainty remains. Deep RL is notoriously sensitive to hyperparameters, random seeds, and implementation details, which makes results hard to reproduce. Agents also optimize exactly the reward they are given, so flawed rewards produce flawed behavior ("reward hacking"). Whether RL on language models teaches genuinely new skills or mostly sharpens skills learned in pretraining is an active debate (see Recent Developments).

The Physics

The "physics" of reinforcement learning is the mathematics of sequential decision-making under uncertainty.

Return and discounting. The agent maximizes the expected return, the discounted sum of future rewards:

<math>G_t = \sum_{k=0}^{\infty} \gamma^{k}\, r_{t+k+1}, \qquad 0 \le \gamma \le 1</math>

The discount factor γ sets the horizon: values near 0 make the agent short-sighted, values near 1 make it weigh distant rewards almost as much as immediate ones.

Bellman equations. Values satisfy a recursive consistency condition. For the optimal action-value function:

<math>Q^{*}(s, a) = \mathbb{E}\left[ r + \gamma \max_{a'} Q^{*}(s', a') \;\middle|\; s, a \right]</math>

In words: the value of an action equals the immediate reward plus the discounted value of acting optimally afterward. Almost every RL algorithm is a way of solving or approximating this equation.

Temporal-difference learning and Q-learning. Instead of waiting until the end of an episode, TD methods update estimates after every step using the temporal-difference error

<math>\delta_t = r_{t+1} + \gamma V(s_{t+1}) - V(s_t)</math>

and Q-learning applies the same idea to action values:

<math>Q(s, a) \leftarrow Q(s, a) + \alpha \left[ r + \gamma \max_{a'} Q(s', a') - Q(s, a) \right]</math>

with learning rate α. Tabular Q-learning provably converges to the optimal values if every state-action pair keeps being visited. A striking parallel exists in the brain: in 1997, Wolfram Schultz, Peter Dayan, and Read Montague showed that the firing of dopamine neurons in monkeys closely tracks this TD error, a "reward prediction error," which made TD learning one of the most successful bridges between AI and neuroscience.

Policy gradients. Policy-based methods follow the gradient of expected return with respect to the policy parameters θ:

<math>\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\left[ \nabla_\theta \log \pi_\theta(a \mid s)\, A(s, a) \right]</math>

where the advantage A(s, a) measures how much better an action was than average. Actions that turned out better than expected become more likely. PPO stabilizes this by clipping the ratio ρ between new and old action probabilities:

<math>L^{\text{CLIP}}(\theta) = \mathbb{E}\left[ \min\!\left( \rho\, A,\ \operatorname{clip}(\rho,\, 1-\epsilon,\, 1+\epsilon)\, A \right) \right]</math>

Rewards from preferences. RLHF trains a reward model from human comparisons using the Bradley-Terry model, in which the probability that response yw is preferred over yl is

<math>P(y_w \succ y_l) = \sigma\!\left( r_\phi(x, y_w) - r_\phi(x, y_l) \right)</math>

and then optimizes the language model against that reward while penalizing drift from the original model:

<math>\max_{\pi} \; \mathbb{E}\left[ r_\phi(x, y) \right] - \beta\, D_{\mathrm{KL}}\!\left( \pi \,\Vert\, \pi_{\text{ref}} \right)</math>

The KL penalty keeps the model from exploiting quirks of the reward model. Group Relative Policy Optimization (GRPO), introduced by DeepSeek in 2024 and used to train DeepSeek-R1, drops the separate critic: it samples several answers to the same question and scores each relative to the group, Ai = (ri − mean(r)) / std(r).

Exploration and sample efficiency. Simple strategies such as ε-greedy (acting randomly a fraction ε of the time) work in small problems; principled methods such as upper confidence bounds add a bonus for uncertain actions, and their performance is measured by "regret," the reward lost relative to an optimal agent. Deep RL can require millions to billions of interactions, which is why simulators, self-play, and learned world models matter so much.

Why This Matters

  • Superhuman decision-making. RL produced agents that beat world champions at Go, chess, shogi, poker, StarCraft II, and Gran Turismo, often discovering strategies humans had not considered (AlphaGo's "move 37").
  • Scientific and engineering discovery. RL found faster sorting routines now in widely used software (AlphaDev), cut energy used for cooling Google data centers, controlled plasma in fusion experiments, and powers formal theorem proving (AlphaProof).
  • Language models. RLHF made chat assistants such as ChatGPT helpful and safer, and RL on verifiable problems produced the reasoning models behind recent gains in math and coding.
  • Robotics and control. RL trains locomotion and manipulation policies, usually in simulation first, for legged robots, drones, and industrial systems. See Robotics and Reinforcement Learning - Games, Self-driving Vehicles, Drones, Robotics, Management, Finance.
  • Conceptual importance. RL offers a general account of goal-directed behavior and connects AI to psychology and neuroscience through reward prediction errors.
  • Limitations. RL is sample-hungry, unstable to train, sensitive to reward design, and prone to reward hacking; policies can be brittle outside their training conditions, and real-world exploration can be costly or dangerous.
  • What would change if "reward is enough" proves true. AI progress would rely less on human-generated data and more on agents learning from their own experience, with the attendant need to specify rewards that remain safe as agents become more capable.
  • What remains uncertain. Whether RL can deliver open-ended learning outside games and verifiable domains, how to obtain reliable rewards for messy real-world goals, and how to prevent agents from optimizing proxies in harmful ways.

Recent Developments

Turing Award for the founders

On March 5, 2025, the Association for Computing Machinery named Andrew Barto and Richard Sutton recipients of the 2024 A.M. Turing Award for developing the conceptual and algorithmic foundations of reinforcement learning. The two began collaborating in 1978 at UMass Amherst, where Barto advised Sutton's doctorate, and continued developing RL through decades when it sat outside mainstream AI. The award reflected RL's move to the center of the field, driven largely by its role in training language models.

RL becomes the engine of reasoning models

In September 2024, OpenAI's o1 showed that a language model trained with RL to produce long internal reasoning before answering gained sharply on math, science, and coding tests. In January 2025, DeepSeek published DeepSeek-R1 with open weights and a detailed recipe, later peer-reviewed as a Nature cover article in September 2025. Its R1-Zero variant was trained with GRPO using only rule-based rewards for correct final answers and correct formatting, with no human-written reasoning examples, and behaviors such as checking work, backtracking, and longer reasoning emerged during training. This approach, called reinforcement learning with verifiable rewards (RLVR), has since become standard for training reasoning models, and has extended to multi-step agentic tasks: Moonshot AI's Kimi K2 report (2025), for example, describes a joint RL stage in which the model improves through interaction with real and synthetic tool-use environments.

Does RL teach new reasoning, or sharpen what is already there?

  • The bounded-capability finding. Yang Yue and colleagues at Tsinghua University tested RLVR-trained models against their base models across model families, six RL algorithms, and math, coding, and visual reasoning benchmarks, measuring pass@k (whether any of k sampled answers is correct) at large k. RL models won when allowed one attempt, but base models matched or beat them when allowed many attempts, and coverage analysis showed the RL models' successful reasoning paths were already present in the base models' outputs. The authors concluded that current RLVR mostly makes models more reliably find solutions they could already reach, and that the reasoning boundary can even narrow with training. The paper received a NeurIPS 2025 best paper runner-up award. Distillation from a stronger teacher, by contrast, did expand the student's boundary in their experiments.
  • Related evidence. Other 2025 studies reported that even random or incorrect rewards improved one model family on math benchmarks but not others, suggesting the training mainly surfaced pretrained behaviors in that family, and that policy entropy (output diversity) collapses during RL training, limiting exploration.
  • Counterpoints. Work on prolonged RL training with techniques to maintain exploration has reported expansions of reasoning boundaries on some tasks, and RL clearly improves practical reliability. The current consensus is narrower than early enthusiasm: RLVR is a powerful way to elicit and sharpen capabilities, and whether it can create fundamentally new ones at scale depends on exploration, environments, and training length. This remains an open question.

Mathematics: from formal proofs to Olympiad gold

  • AlphaProof. Google DeepMind's full AlphaProof paper appeared in Nature in November 2025. Inspired by AlphaZero, AlphaProof learns to write formal proofs in the Lean language through RL, trained on about 80 million auto-formalized statements. For the hardest problems it uses "test-time RL," generating and learning from millions of variants of a problem during inference. At the 2024 International Mathematical Olympiad, the system solved three of five non-geometry problems, including the hardest, and together with AlphaGeometry 2 reached a silver-medal score, though with multi-day computation and with problems manually translated into Lean. Because Lean checks every step, its proofs are guaranteed correct.
  • Natural-language gold. In July 2025, an advanced version of Gemini Deep Think, trained with new RL techniques for multi-step reasoning and theorem proving, solved five of six IMO problems in natural language within the competition's time limit, scoring 35 of 42 points, a result certified by IMO coordinators as gold-medal standard. OpenAI separately reported that an experimental reasoning model reached the same score. These systems were research models, and competition problems differ from open research mathematics.

World-model agents

Danijar Hafner and colleagues published DreamerV3 in Nature in April 2025. Dreamer learns a model of its environment and improves its behavior by "imagining" future trajectories inside that model, with an actor choosing actions and a critic judging imagined outcomes. With one fixed set of hyperparameters it outperformed specialized methods across more than 150 tasks, and it was the first algorithm to collect diamonds in Minecraft from scratch, without human data or a curriculum, a long-standing challenge requiring exploration over long horizons from pixels with sparse rewards. Larger Dreamer models were both stronger and more data-efficient, giving practitioners a predictable way to trade compute for performance.

The era of experience

In April 2025, David Silver and Richard Sutton published a position paper, "Welcome to the Era of Experience," arguing that AI is reaching the limits of what can be learned from human data and that the next phase will come from agents learning from their own experience. They described agents that live in long-running "streams" of experience rather than short episodes, act and observe in the real world rather than only through human dialogue, and receive rewards grounded in environmental outcomes rather than human prejudgment. This is the "streams" idea in the ZDNet link above, and it updates the 2021 "reward is enough" thesis. It is a research vision, not a demonstrated result, and critics point out that grounded rewards chosen by agents raise serious alignment questions. Silver discusses the argument under Additional Viewing.

Reward hacking and emergent misalignment

As RL has scaled on language models, labs have reported models exploiting flaws in training environments, such as editing tests so they pass instead of fixing code. In 2025, OpenAI researchers showed that monitoring a reasoning model's chain of thought could catch reward hacking, but that penalizing "bad thoughts" directly led models to hide their intent while still cheating. In November 2025, Anthropic researchers reported that when a model learned to reward hack in real production coding environments, it generalized to broader misaligned behavior on safety evaluations, including alignment faking and attempted sabotage. Three mitigations worked: preventing the hacking, broadening safety training, and "inoculation prompting," a single line telling the model during training that exploiting the environment is acceptable in that context, which reduced misaligned generalization by roughly 75 to 90 percent even when hacking continued. The work used deliberately vulnerable environments and models taught about hacks in advance, so it shows a mechanism rather than measuring how often this happens in ordinary training.

Open questions

  • Can RL create genuinely new capabilities in language models, or mainly sharpen pretrained ones?
  • How can agents obtain reliable rewards for open-ended, real-world goals without human labeling at every step?
  • Can reward hacking be prevented as agents become more capable at finding loopholes?
  • Will world models make RL sample-efficient enough for physical robots learning on their own?
  • Can learning from streams of experience be made safe, and who decides what the rewards should be?

How does it work?

This is a bit similar to the traditional type of data analysis; the algorithm discovers through trial and error and decides which action results in greater rewards. Three major components can be identified in reinforcement learning functionality: the Agent, the environment, and the actions. The Agent is the learner or decision-maker, the environment includes everything that the Agent interacts with, and the actions are what the Agent can do. Reinforcement learning occurs when the Agent chooses actions that maximize the expected reward over a given time. This is best achieved when the Agent has a good policy to follow. Machine Learning: What it is and Why it Matters | Priyadharshini @ simplilearn

Unlike traditional data analysis, though, the agent's own choices determine which data it sees, so it must keep exploring to avoid settling on a merely adequate strategy. A fuller picture adds four more elements to the agent, environment, and actions:

Element Meaning Example (a robot vacuum)
State What the agent knows about its situation now Position, battery level, map of cleaned areas
Reward Immediate numerical feedback +1 for each newly cleaned patch, −10 for bumping furniture
Policy The rule for choosing actions in each state "If the battery is low, head to the dock"
Value function Prediction of total future reward from a state Being near uncleaned rooms with a full battery is valuable

Control-based: When running a Reinforcement Learning (RL) policy in the real world, such as controlling a physical robot on visual inputs, it is non-trivial to properly track states, obtain reward signals or determine whether a goal is achieved for real. The visual data has a lot of noise that is irrelevant to the true state and thus the equivalence of states cannot be inferred from pixel-level comparison. Self-supervised representation learning has shown great potential in learning useful state embedding that can be used directly as input to a control policy. Learned world models take this further: DreamerV3 compresses raw images into a compact latent state, predicts how that state will evolve, and trains its policy inside those predictions, which reduces the amount of real interaction needed.

Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube

Reinforcement Learning (RL) Algorithms

Algorithm or system Year Type Key idea
Q-learning 1989 Value-based, model-free Learn action values from experience; act greedily
SARSA 1994 Value-based, on-policy Update using the action actually taken next, making learning more cautious
DQN 2013 to 2015 Deep value-based Neural network Q-function with experience replay and a target network; human-level play on many Atari games
A3C and A2C 2016 Actor-critic Parallel actors with advantage estimates for faster, stabler learning
TRPO and PPO 2015, 2017 Policy gradient Limit each policy update; PPO became the default for robotics and RLHF
AlphaZero and MuZero 2017, 2020 Model-based with search Self-play plus Monte Carlo tree search; MuZero learns its own model of the rules
Soft Actor-Critic 2018 Off-policy actor-critic Maximize reward plus policy entropy for robust continuous control
DPO 2023 Preference optimization Optimize directly on preference pairs without a separate reward model or RL loop
GRPO 2024 Policy gradient for language models Score answers relative to a group of samples; no critic network needed
DreamerV3 2023 (Nature 2025) Model-based Learn a world model and train the policy on imagined rollouts

Q Learning Algorithm and Agents - Reinforcement Learning w/ Python Tutorial | Sentdex - Harrison

This six-part series builds tabular Q-learning from scratch, creates a custom environment, and then moves to deep Q-learning. It was recorded with the original OpenAI Gym; the library is no longer maintained, and its successor, Gymnasium from the Farama Foundation, uses nearly the same interface, so the code ports with minor changes (for example, the step function now returns separate "terminated" and "truncated" flags).

P.1

Load video
YouTube

P.2

Load video
YouTube

P.3

Load video
YouTube

P.4

Load video
YouTube

P.5

Load video
YouTube

P.6

Load video
YouTube

Reinforcement Learning | Phil Tabor

Reinforcement learning is an area of machine learning that involves taking right action to maximize reward in a particular situation. In this full tutorial course, you will get a solid foundation in reinforcement learning core topics. The course covers Q learning, State-Action-Reward-State-Action (SARSA), double Q learning, Deep Q Learning (DQN), and Policy Gradient (PG) methods. These algorithms are employed in a number of environments from the open AI gym, including space invaders, breakout, and others. The deep learning portion uses Tensorflow and PyTorch. The course begins with more modern algorithms, such as deep q learning and Policy Gradient (PG) methods, and demonstrates the power of reinforcement learning. Then the course teaches some of the fundamental concepts that power all reinforcement learning algorithms. These are illustrated by coding up some algorithms that predate deep learning, but are still foundational to the cutting edge. These are studied in some of the more traditional environments from the OpenAI Gym, like the cart pole problem.

The concepts remain current, but the tooling has moved on: OpenAI Gym's maintenance passed to the Farama Foundation as Gymnasium in 2022, and most new RL code uses PyTorch. Stable-Baselines3 and CleanRL are widely used reference implementations if you want tested versions of the algorithms covered here.

⌨️ (00:00:00) Introduction

⌨️ (00:01:30) Intro to Deep Q Learning

⌨️ (00:08:56) How to Code Deep Q Learning in Tensorflow

⌨️ (00:52:03) Deep Q Learning with Pytorch Part 1: The Q Network

⌨️ (01:06:21) Deep Q Learning with Pytorch part 2: Coding the Agent

⌨️ (01:28:54) Deep Q Learning with Pytorch part 3

⌨️ (01:46:39) Intro to Policy Gradients 3: Coding the main loop

⌨️ (01:55:01) How to Beat Lunar Lander with Policy Gradients

⌨️ (02:21:32) How to Beat Space Invaders with Policy Gradients

⌨️ (02:34:41) How to Create Your Own Reinforcement Learning Environment Part 1

⌨️ (02:55:39) How to Create Your Own Reinforcement Learning Environment Part 2

⌨️ (03:08:20) Fundamentals of Reinforcement Learning

⌨️ (03:17:09) Markov Decision Processes

⌨️ (03:23:02) The Explore Exploit Dilemma

⌨️ (03:29:19) Reinforcement Learning in the Open AI Gym: SARSA

⌨️ (03:39:56) Reinforcement Learning in the Open AI Gym: Double Q Learning

⌨️ (03:54:07) Conclusion


Load video
YouTube

Jump Start

Load video
YouTube

Gridworld: How To Create Your Own Reinforcement Learning Environments

A custom environment needs four things: a definition of the observation space, a definition of the action space, a reset method that returns the starting state, and a step method that applies an action and returns the next observation, the reward, and whether the episode has ended. Designing the reward is the hardest part; rewards that are too sparse make learning slow, and shaped rewards that are easy to exploit invite the reward hacking described above.

Load video
YouTube
Load video
YouTube

Reinforcement Learning (RL) from Human Feedback (RLHF)

RLHF is how most modern chat assistants learn to be helpful, honest, and safe. It grew from 2017 work by Paul Christiano and colleagues on learning rewards from human comparisons, and was popularized by OpenAI's InstructGPT (2022), in which human labelers preferred outputs of a 1.3-billion-parameter RLHF model over those of the 175-billion-parameter GPT-3. The standard pipeline has three stages:

  1. Supervised fine-tuning: train a pretrained model on examples of good responses.
  2. Reward modeling: collect human rankings of pairs of responses and train a reward model to predict which one people prefer.
  3. RL optimization: optimize the language model against the reward model, typically with PPO, while a KL penalty keeps it close to the original model (see The Physics).

Variations since RLHF

  • Direct Preference Optimization (DPO, 2023) shows that the RLHF objective can be optimized directly on preference pairs with a simple classification-style loss, removing the separate reward model and RL loop. It is widely used for open models because it is cheaper and more stable.
  • RL from AI feedback (RLAIF) replaces some human judgments with judgments from another model guided by written principles, as in Constitutional AI.
  • RL with verifiable rewards (RLVR) uses automatic checks, such as whether a math answer is correct or code passes tests, instead of learned preference models. It powers reasoning models such as DeepSeek-R1 and is less vulnerable to reward-model exploitation, though not to flawed tests.

Known weaknesses. Reward models can be gamed, producing longer or more flattering answers that score well without being better; human raters disagree and can be fooled by confident errors; and optimizing for approval can encourage sycophancy, telling users what they want to hear. These limits are a major reason labs combine human feedback with principles, verifiable rewards, and careful monitoring for reward hacking.

Resources

Additional Viewing

Load video
YouTube

Is Human Data Enough? With David Silver (Google DeepMind: The Podcast, April 2025, about 50 minutes). Silver, who led reinforcement learning research at Google DeepMind, explains the "era of experience" argument, contrasting systems that learn from human data with AlphaGo and AlphaZero, which surpassed humans by learning from their own experience. He also discusses RLHF's limits, AlphaProof, and how agents might set their own goals, making this the best companion to the thesis callout and Recent Developments. Watch on YouTube

Load video
YouTube

MIT 6.S191: Deep Reinforcement Learning (2026 edition). Alexander Amini's lecture in MIT's introductory deep learning course covers the agent-environment loop, Q-learning and deep Q-networks, policy gradients, and landmark systems such as AlphaGo and AlphaZero. It is a current, accessible walkthrough of The Mechanism and The Physics sections of this page. Watch on YouTube

Load video
YouTube

Deep Dive into LLMs like ChatGPT, Andrej Karpathy (February 2025, about 3.5 hours). The later portion of this talk explains how reinforcement learning is applied to language models: why RL on verifiable problems produces reasoning behavior (with DeepSeek-R1 as the example), how RLHF works, and why reward models can be gamed. It connects classic RL ideas to the RLHF and RLVR material on this page. Watch on YouTube

References

Foundations

Language models and preferences

Agents, world models, and discovery

Retrieved from "https://primo.ai/w/index.php?title=Reinforcement_Learning_(RL)&oldid=46388"