A Markov decision process (MDP) models the ♟️ Reinforcement Learning environment as , with states , actions , transitions , and reward function . At each time step, the 🧸 Agent takes an action based on the current state, resulting in a new state and a reward.
The transition operator describes (similar to ⛓️ Markov Chains), and the reward function produces a scalar given a state and action.
A partially observable MDP (POMDP) makes a distinction between the environment's states and agent's observations by introducing an additional observation space and emission probability . The emissions the probability of getting a certain observation from the current state.
Crucially, in a POMDP, the states satisfy the Markov property, whereas our observations might depend on the past. This influences the design of some reinforcement learning algorithms to avoid the Markov property with observations.
Like with MDPs, model-based reinforcement learning algorithms that have observations (images, for example) seek to learn the world transitions. In POMDPs, we need to learn both and . The former is called the dynamics model, and the latter is called the observation model.
We can use a similar log likelihood objective as MDPs, but since our true states are unknown, we need to take an expectation over them,
To compute this expectation, we need an approximate posterior encoder
much like with 🖋️ Variational Autoencoders. There are many choices for this posterior, ranging from the fully expressive to the single-step .