Behavioral cloning is an 🐵 Imitation Learning technique that trains a policy to mimic an expert . Our policy is trained to predict actions given observation by training on expert-generated pairs, essentially converting our problem to a standard supervised learning problem,
One significant problem with this approach is that only learns states that the expert encounters. Since the expert performs "correct" actions, the dataset's distribution only consists of states that we reach after such actions. Thus, the dataset has no examples on states after the agent makes a mistake, and once an agent makes a mistake—due to random chance or incorrect prediction—it will make even larger mistakes.
If we assume that is trained well (ie, probability of error is at most , ), then the number of mistakes will make in timesteps grows quadratically with . Intuitively, this is because the difference between the policy's distribution and at time is bounded by (since it has timesteps to make mistakes), so summing up over timesteps gives us a quadratic. This result is concerning because the number of mistakes grows quadratically as we continue running the policy whereas ideally, we want this to be linear.
To improve robustness toward this problem, we can:
Introduce imperfections and augmentations to our data that teach the policy corrective actions.
Reduce modeling error using more powerful models. Since humans are generally non-Markovian and multimodal, the policy could use a history of observations (eg, via a 💬 Recurrent Neural Network) and produce a multimodal action distribution (eg, via 🕯️ Diffusion or autoregressive discretization).
Learn multiple tasks via goal-conditioned BC, which may improve data coverage.
Iteratively improve the expert dataset by annotating mistakes—this is the key idea behind 🗡️ DAgger.
Note that one danger in modeling non-Markovian behavior is causal confusion: mistaking correlations as causations. For example, if a camera sees the brake light in a car turn on during braking, it might associate the brake light for the reason behind braking rather than the obstacle behind the window.
The following outlines why the policy's error grows quadratically with . Assume our supervised learning policy has error (predicting incorrectly with probability ).
Consider state after steps. If we make no mistakes, our state will be in the training distribution. Otherwise, it'll be in some mistake distribution. We can split our policy distribution into these two cases,
