Imitation learning is a class of algorithms in ♟️ Reinforcement Learning that learn from expert demonstrations.

1. 🌀 Behavioral Cloning models expert demonstrations via 🎓 Supervised Learning.
2. 🗡️ DAgger adds robustness to BC by incorporating manual feedback.

Fitting the Expert#

Aside from the training algorithm used to imitate the expert, another important question is whether the policy can fit the expert demonstrations in the first place. Some reasons why this is hard include:

1. Non-Markovian behavior: the human produces actions not just based on the current observation, but memory of the past.
2. Multimodal behavior: the human's behavior is not deterministic or consistent, and there may be multiple actions that are equally valid (eg, going left or right around a tree).

The policy architecture can get around these issues. To address non-Markovian behavior, the policy can receive a history of past observations and actions as input, along with the current observation; however, note that this increases the danger of distribution shift and causal confusion. To address multimodality, the policy can be designed to naturally capture multiple modes, such as via 🕯️ Diffusion.