Generalized advantage estimation (GAE) balances the tradeoff between 1-step returns (no bias, higher variance) and full -step returns (biased, lower variance) by generalizing 🪜 N-Step Bootstrapping. Rather than choosing just one as the bootstrap cutoff, we'll cut everywhere at once. Specifically, if we let denote the advantage with cutoff at ,
then GAE computes the weighted average of multiple -step return estimators as
with weights since we usually prefer cutting earlier to reduce the variance of our single-sample estimate.
Shuffling terms and reorganizing, we see that the GAE advantage has an elegant form,