🕹️ Optimal Control

Optimal control is a branch of optimization that aims to find a control (a series of actions) in a dynamic system to optimize some function. For example, a common problem is

More generally, optimal control is defined with differential equations,

Note that this field is closely related to ♟️ Reinforcement Learning (where is state and is action ). The main difference is that optimal control methods generally arise from more mathematical roots whereas reinforcement learning is more applied.

Dynamic System §

A dynamic system is defined by state transitions, which, in the context of reinforcement learning, can be deterministic or stochastic.

1. Deterministic transitions follow

2. Stochastic transitions follow

Moreover, the system may be open-loop or close-loop.

1. Open-loop systems offer no feedback or new information—we’re expected to output a sequence of actions given just the initial state . Algorithms that deal with this case include 🎲 Cross Entropy Method and the Linear-Quadratic Regulator (LQR). Note that open loop control is often only feasible with deterministic transitions.
2. Closed-loop systems provide new information after we take an action . In this case, we often derive a policy , falling closer into the realm of reinforcement learning.