Least Squares Estimation

The method of least squares is about estimating parameters by minimizing the squared discrepancies between observed data on the one hand, and their expected values on the other. It is commonly used in the context of a regression model, where one observes pairs of variables (X, Y), with X a covariable and Y a response variable. The expectation of the response variable Y given the covariable X is modeled as a function fβ(X) depending on parameters β. We present an explicit expression for the least squares estimator of β in the linear regression model, and also give its covariance matrix. The calculations are illustrated with a numerical example. The F statistic for testing linear hypotheses on β is also given. We conclude with some extensions. Keywords: least squares; linear hypothesis; regression