🗼 Least Mean Squares

Theory §

LMS is an online learning (streaming) alternative to 🏦 Linear Regression; the closed form calculation can be extremely expensive, so LMS approximates it.

As before, we optimize the loss, but this time with stochastic ⛰️ Gradient Descent.

Info

For learning rate , LMS converges if where is the largest value of covariance matrix .

Locally Weighted Regression §

If our data is non-linear, we can approximate a prediction by fitting a line to points that are nearby. However, this is a non-parametric method that requires fitting a new line for every prediction.

Specifically, we aim to find that minimizes the weighted loss

where , controlling the bandwidth of the sudo-gaussian.

Model §

The model itself is the exact same as linear regression: we keep a set of weights that are applied to each feature in the input . We also maintain learning rate as a hyperparameter used during training.

Training §

Given training data , initialize weights randomly.

Then, go through each observation sequentially,

1. Let residual
2. Update weights

Info

To derive the update step, we observe that the derivative of L2 loss is , so a move in the gradient step with scaling gives us our update equation.

Prediction §

Similar to linear regression, given input , our prediction .