4.2-final-2006¶
Final Exam, 2006 · Section 4.2: Projections onto Lines and Subspaces Source: finalexam_1.pdf, p. 151
Problem¶
8 (4+3+4=11 pts.) Suppose a rectangular matrix \(A\) has independent columns.
(a) How do you find the best least squares solution \(\widehat{x}\) to \(Ax = b\)? By taking those steps, give me a formula (letters not numbers) for \(\widehat{x}\) and also for \(p = A\widehat{x}\).
(b) The projection \(p\) is in which fundamental subspace associated with \(A\)? The error vector \(e = b - p\) is in which fundamental subspace?
(c) Find by any method the projection matrix \(P\) onto the column space of \(A\):
\[
A = \begin{bmatrix} 1 & 0 \\ 3 & 0 \\ 0 & -1 \\ 0 & -3 \end{bmatrix}.
\]
Solution
(a) The least-squares "solution" solves the normal equations \[ A^TA\widehat{x} = A^Tb. \] Since \(A^TA\) is invertible: \[ \widehat{x} = (A^TA)^{-1}A^Tb, \] and the projection is \[ p = A\widehat{x} = A(A^TA)^{-1}A^Tb. \] (b) \(p = A\widehat{x}\) is a linear combination of columns of \(A\), so it is in the column space \(C(A)\). The error \(e = b - p\) is orthogonal to this space, so it is in the left nullspace \(N(A^T)\). (c) I used \(P = A(A^TA)^{-1}A^T\). Since \(A^TA = \begin{bmatrix} 10 & 0 \\ 0 & 10 \end{bmatrix}\), its inverse is \(\begin{bmatrix} 1/10 & 0 \\ 0 & 1/10 \end{bmatrix} = \frac{1}{10}I\), and \[ P = \frac{1}{10} \begin{bmatrix} 1 & 3 & 0 & 0 \\ 3 & 9 & 0 & 0 \\ 0 & 0 & 1 & 3 \\ 0 & 0 & 3 & 9 \end{bmatrix}. \]Keywords: least squares, projection matrix, column space, normal equations