Actor Critic

YouTube search... ...Google search

• Reinforcement Learning (RL)
  • Monte Carlo (MC) Method - Model Free Reinforcement Learning
  • Markov Decision Process (MDP)
  • State-Action-Reward-State-Action (SARSA)
  • Q Learning
    • Deep Q Network (DQN)
  • Deep Reinforcement Learning (DRL) DeepRL
  • Distributed Deep Reinforcement Learning (DDRL)
  • Evolutionary Computation / Genetic Algorithms
  • Actor Critic
    • Asynchronous Advantage Actor Critic (A3C)
    • Advanced Actor Critic (A2C)
    • Lifelong Latent Actor-Critic (LILAC)
  • Hierarchical Reinforcement Learning (HRL)
• Beyond DQN/A3C: A Survey in Advanced Reinforcement Learning | Joyce Xu - Towards Data Science
• Policy ... Policy vs Plan ... Constitutional AI ... Trust Region Policy Optimization (TRPO) ... Policy Gradient (PG) ... Proximal Policy Optimization (PPO)

Policy gradients and Deep Q Network (DQN) can only get us so far, but what if we used two networks to help train and AI instead of one? Thats the idea behind actor critic algorithms. Actor-critic algorithms are a type of reinforcement learning algorithm that combines the strengths of policy-based and value-based methods. They are composed of two components: an actor and a critic.

The actor is responsible for selecting actions based on the current state of the environment. The critic evaluates the actions selected by the actor and provides feedback to help the actor improve its policy.

Actor-critic algorithms work by iteratively updating the actor and critic networks. The actor network is updated using policy gradients, which are calculated using the feedback from the critic network. The critic network is updated using temporal difference (TD) learning to estimate the value function of the current state.

The main idea behind actor-critic algorithms is to use the critic network to provide the actor network with a signal that indicates how well it is performing. This signal is then used by the actor network to update its policy in a direction that will lead to higher rewards.

Actor-critic algorithms have several advantages over other reinforcement learning algorithms. First, they are able to learn policies in continuous action spaces, which is not possible with many other reinforcement learning algorithms. Second, actor-critic algorithms are sample-efficient, meaning that they can learn good policies with relatively few samples from the environment. Third, actor-critic algorithms are able to learn policies in real time, which makes them suitable for applications such as robotics and video games.

Here is a simplified example of how an actor-critic algorithm might work:

1. The agent observes the current state of the environment.
2. The actor network selects an action based on the current state.
3. The agent takes the selected action and observes the next state and reward.
4. The critic network evaluates the selected action and provides feedback to the actor network.
5. The actor network updates its policy based on the feedback from the critic network.
6. The critic network updates its value function based on the new state and reward.

The agent repeats steps 1-6 until it reaches a terminal state.

Actor-critic algorithms are a powerful tool for reinforcement learning and have been used to achieve state-of-the-art results in a variety of tasks, including robotics, video games, and finance.