Actor-critic is a class of ♟️ Reinforcement Learning algorithms that optimizes the policy using a learned value function; the "actor" is the policy, and "critic" is the value function. This technique can be viewed as a extension of 🚓 Policy Gradients that replaces the trajectory reward with the learned value, which aims to reduce variance and improve generalization.

Starting from the policy gradient objective, we observe that the reward summation (with causality) is actually a single-sample estimate of the reward-to-go, ie the Q-function, and thus can be written as

We can also incorporate a baseline to reduce variance: baselining with the average Q-function, which is exactly the value function, we arrive at

where is the advantage function, intuitively defined as the difference between expected reward of chosen action and the average reward at .

Note that this formulation is extremely similar to ♻️ Policy Iteration; the main difference is that we're performing a gradient ascent step on the gradient whereas policy iteration directly redefines the policy to the optimal value.

Advantage Computation#

To compute the advantage function, we can approximate the Q-function via a single-sample estimate of the dynamics,

which gives us a simple formula,

To incorporate a discount factor , we simply multiply the next state's value, . Estimating this can be done via 💯 Policy Evaluation, either by training in batches or online updates after every move. Finally, this can be generalized and improved even further with 🪜 N-Step Bootstrapping and 👑 Generalized Advantage Estimation.

Parallel Actor-Critic#

To stabilize training the value function, we can run multiple actor-critics at the same time with the same policy but different random seeds. Each instance makes online updates to the same neural network for .

Synchronized parallel actor-critic makes these updates together in a batch whereas asynchronous parallel actor-critic updates whenever an instance finishes. One slight theoretical drawback to the asynchronous version is that some instances might train the network on samples drawn using an old policy; however, practical performance benefits usually outweigh this drawback.