The standard 🚓 Policy Gradient method is on-policy, requiring samples collected by running the same policy we're optimizing for. Once we update our policy, we need to collect new samples.
The goal of the off-policy policy gradient is to be able to update our policy using samples not from our trajectory. Let
We start with the original objective and apply 🪆 Importance Sampling to rewrite the expectation in terms of the sample distribution
Next, we can apply the 🦄 Log Derivative Trick:
While this gradient can work on its own, there are a few adjustments we make for practicality. First, similar to 🚓 Policy Gradient, we can use causality,
However, these importance weights are problematic: products grow exponentially, which increases the variance of our estimates. To address this, we can:
- Ignore the second product, which gives us a ♻️ Policy Iteration algorithm that can be shown to still improve the policy, even if it's no longer the actual gradient.
- For the first product, only use the importance weight at
as an estimation. The reasoning is that if we look at the state-action marginal (occupancy measure) formulation of our objective, we get
by assuming
Note that while this not technically the correct policy gradient, the error is bounded if