Least Squares Estimation
Priliminaries A Simple Linear Regression the column space Another Example of Linear Regression 1 In the blog A Simple Linear Regression, squares of the difference between the output of a predictor and the target were used as a loss function in a regression problem. And it could be also written as: \[ \ell(\hat{\mathbf{y}}_i,\mathbf{y}_i)=(\hat{\mathbf{y}}_i-\mathbf{y}_i)^T(\hat{\mathbf{y}}_i-\mathbf{y}_i) \tag{1} \] The linear regression model in a matrix form is: \[ y=\mathbf{w}^T\mathbf{x}+\mathbf{b}\tag{2} \] What we do in this post is analyze the least-squares methods from two different viewpoints...