The linear quadratic regular (LQR) is a method for solving 🕹️ Optimal Control for a problem that satisfies certain constraints. Specifically, we assume:

1. Linear system dynamics: .
2. Quadratic stage cost (time-independent): , with PSD and PD .
3. Quadratic terminal cost: , with PSD .

Putting this together, our objective is to optimize

such that .

To solve this problem, we call upon principles from the 🔔 Bellman Equation and 🧨 Dynamic Programming. First, rewriting the goal using the bellman equation, with value function , we have

As with other dynamic programming approaches, we'll recursively solve this problem, starting from and working backwards to . In our first step, with , we can simplify

The middle matrix is PSD since the sum above is non-negative, and also is PD since is PD. Then, we can use general lemmas for matrices of this structure to compute the optimal values,

Since is also quadratic, we can perform the same step for , switching out for , and so forth. Thus, the full LQR iteration computes , where

and ending with . Finally, knowing , we can compute optimal control

Continuous Time#

We can apply the same principles to continuous time systems, where states follow

and the value function is

With the constraints detailed above, our objective is to optimize

such that and .

Hamilton-Jacobi-Bellman Equation#

Before assuming the LQR constraints, we can first derive a more general form of the solution. Assuming some and small , we can derive a continuous analogue of the Bellman equation,

where we apply 🎤 Taylor Expansion around to both terms. Rearranging and dropping since it's arbitrarily small, we have

and after applying chain rule and rearranging, we arrive at

This PDE is called the Hamilton-Jacobi-Bellman (HJB) equation.

Riccati Differential Equation#

While the HJB equation may be difficult to solve generally, it does have a closed-form solution for the LQR problem. Incorporating the constraints, our HJB equation becomes

with .

Similar to the discrete-time solution, we'll look for a such that and . With these assumptions, we have and , and plugging into the LQR HJB equation above, we arrive at

Thus, we have

which is called the Riccati differential equation (RDE). This can be solved backwards in time to find , and from there, we can find optimal inputs via

While it's common to solve the Riccati equation numerically, there is a way to solve it analytically with a auxiliary matrix differential equation,

and defining so that

we can solve for and in the combined linear equation

and compute .

Infinite Horizon#

Finally, we can apply LQR to the infinite horizon setting, where :

Note that there is no terminal cost, .

Forward Riccati Differential Equation#

We must first reframe our solution to solve forward in time, instead of backward. Generally, for the Riccati differential equation above, let

so that and . Then, following the RDE,

with . The value function is

and the optimal control is

Algebraic Riccati Equation#

Observe that the forward RDE has a term with in the value function. The question is whether this value is bounded in the infinite horizon case, with . Since our cost depends on and , we note that if a system is stabilizable, then there exists state feedback that ensures converges to zero, as well as also converging to zero. Thus, the cost is bounded for stabilizable systems.

Thus, the steady state limit exists and is PSD, and converges to . Hence,

which we call the Algebraic Riccati Equation (ARE).

The value function is thus

and the optimal control is

Observe that is actually a feedback law, which creates the closed-loop system

Lastly, this feedback law guarantees . If is PD, this implies that , making the system asymptotically stable.