4.3-exam2-2005¶
Quiz 2, 2005 · Section 4.3: Least Squares Approximations Source: exam2_1.pdf, pp. 109-110, 116-117
Problem¶
3 (40 pts.) Our measurements at times \(t = 1, 2, 3\) are \(b = 1, 4\), and \(b_3\). We want to fit those points by the nearest line \(C + Dt\), using least squares.
(a) Which value for \(b_3\) will put the three measurements on a straight line? Which line is it? Will least squares choose that line if the third measurement is \(b_3 = 9\)? (Yes or no.)
(b) What is the linear system \(Ax = b\) that would be solved exactly for \(x = (C, D)\) if the three points do lie on a line? Compute the projection matrix \(P\) onto the column space of \(A\). Remember the inverse
(c) What is the rank of that projection matrix \(P\)? How is the column space of \(P\) related to the column space of \(A\)? (You can answer with or without the entries of \(P\) computed in (b).)
(d) Suppose \(b_3 = 1\). Write down the equation for the best least squares solution \(\hat{x}\), and show that the best straight line is horizontal.
Solution
3a. The three data points lie on the same line when \(b_3 = 7\). This line is \(b = -2 + 3t\). If the third measurement is 9, least squares will NOT choose this line. (A quick way to see this is from the fact that the line chosen by least squares will give the average of the given \(b\)'s at the time equal to the average of the given \(t\)'s; in this case, the best fit line would take the value \((1 + 4 + 9)/3 = 14/3\) at \(t = (1 + 2 + 3)/3 = 2\), whereas our line gives 4 at \(t = 2\).) 3b. The linear system is \[ \begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix} \begin{bmatrix} C \\ D \end{bmatrix} = \begin{bmatrix} 1 \\ 4 \\ b_3 \end{bmatrix}. \] We compute the projection matrix \(P\) onto the column space of \(A\) using the projection matrix formula: \[ A^T A = \begin{bmatrix} 3 & 6 \\ 6 & 14 \end{bmatrix}, \qquad (A^T A)^{-1} = \frac{1}{6} \begin{bmatrix} 14 & -6 \\ -6 & 3 \end{bmatrix}, \] \[ P = A(A^T A)^{-1} A^T = \frac{1}{6} \begin{bmatrix} 5 & 2 & -1 \\ 2 & 2 & 2 \\ -1 & 2 & 5 \end{bmatrix}. \] 3c. The column space of \(P\) is the space consisting of all the vectors \(Pb\), i.e. all the projections of vectors in \(\mathbb{R}^3\) onto the column space of \(A\), which is precisely the column space of \(A\). Thus the rank of \(P\) is equal to the rank of \(A\), which is 2. 3d. The equation for the best least squares solution is \(A^T A \hat{x} = A^T b\), where \(b = (1, 4, 1)\). Writing out this system, we get \[ \begin{bmatrix} 3 & 6 \\ 6 & 14 \end{bmatrix} \begin{bmatrix} C \\ D \end{bmatrix} = \begin{bmatrix} 6 \\ 12 \end{bmatrix}. \] The solution to this system is \(\hat{x} = (2, 0)\), so the best fit line is the horizontal line \(b = 2\).Keywords: least squares, line fitting, projection matrix, rank