The off-policy actor critic improves the sample efficiency of 🎭 Actor-Critic by accepting off-policy data (from a replay buffer), rather than just the current policy's rollouts. To work with off-policy samples, we need two changes:

1. The original value target is incorrect since this estimate would be measuring the value of based on the action taken by another policy, not the current one.
2. Our gradient step is using the action taken by another policy, not the one our current policy would've chosen.

Value Estimation#

To address the first problem, we can instead estimate the Q-function

which has no requirement that comes from our current policy. Thus, we'll train a critic for Q-function instead, with target

where we keep in mind that the next step's action is sampled from our current policy, thus keeping the bootstrapping correct.

Action Sampling#

For the second issue, we'll sample from our current policy for the gradient

Note the distinction between the action used to update our Q-value estimate and the action used to update our policy; the former can use any action, while the latter must come from the current policy.

In practice, since computing advantage requires some estimate of the state-value, we use the Q-function directly instead,

Though this would increase our variance by getting rid of the baseline, we can make up for this by simply sampling multiple times from our policy and running this update from each one—the key idea here is that can come directly from our policy and has no reliance on the actual environment.

Direct Differentiation#

One last observation is that after these changes, the policy gradient is now estimating

and since we're now learning , we can directly differentiate through the model weights instead of using the policy gradient. If we structure , then the 🪄 Reparameterization Trick gives us

which gives us the gradient