Least Squares Regression

Least squares regression is, given a set of paired observations $(x_0,y_0),(x_1,y_1),\dots ,(x_{n-1},y_{n-1})$, assuming a linear relationship defined by $y_i=a_0+a_1{x}_i+e$, where $e$ is the error (or residual) between the model and the observations; we then minimise the sum of the squares of these errors.

In practice, least squares regression means minimising $S_r={\sum }_{i=0}^{n-1}{e}_i^2={\sum }_{i=0}^{n-1}(y_i-a_0-a_1{x}_i{)}^2$ as an error criterion; this results in $a_1=\frac{n\sum x_i{y}_i-\sum x_i\sum y_i}{n\sum x_i^2-{\left(\sum x_i\right)}^2}$ and $a_0=\bar{y}-a_1\bar{x}$.

Expanding the model to a general polynomial of degree $m$ gives $y_i=a_0+a_1{x}_i+a_2{x}_i^2+\cdots +a_m{x}_i^m+e$, and we once again minimise the sum of squared residuals $S_r={\sum }_{i=0}^{n-1}(y_i-a_0-a_1{x}_i-a_2{x}_i^2-\cdots -a_m{x}_i^m{)}^2$ to obtain the best-fitting polynomial coefficients.