Welcome to a lesson on how to determine
the least-squares solutions to the equation
matrix A times vector X equals vector b.
Let's suppose that matrix A times vector X equals vector B
has no solution.
We can't compute an approximate solution
by determining the least-squares solution.
Looking at the theorem below,
if we let A be an m by n matrix, and let vector b
be a vector in Rm, a least-squares solution
of matrix A times vector X equals vector b
are the solutions to A transpose times A times, vector X
equals A transpose times vector b.
So notice how we simply multiply sides of the equation
by A transpose.
So to compute a least-squares solution,
step one we compute A transpose times A,
and A transpose times vector b.
Step two we form an augmented matrix for the equation
A transpose A times vector X
equals A transpose times vector b,
and write the augmented matrix in reduced row echelon form.
Step three the solution is a least-squares solution.
Let's look at an example.
Let's find the least-squares solutions
to matrix A times vector X equals vector b,
where matrix A is the given three by two matrix,
and vector b is the vector six, zero, zero.
Let's first verify there is no solution to the equation
A times vector X equals vector b,
where the corresponding matrix equation
would be the equation shown here on the right.
To try to solve this we would write the augmented matrix
where the first row is zero, six, one,
the second row is one, one, zero,
and the third row is two, one, zero, which we see here.
I've already written this in reduced row echelon form.
Remember the first column
would be the coefficients of X one,
the second column would be the coefficients of X two,
third column would be the constants.
Notice we have a contradiction in row three.
The third row indicates that zero equals one
which of course is not true, which verifies the equation
A times vector X equals vector b has no solution.
So now to find the least-squares solution
we need to find A transpose times A,
as well as A transpose times vector b.
Remember to find A transpose,
we take the first column matrix A,
and this becomes the first row of A transpose,
and the second column of matrix A
becomes the second row of A transpose.
So now we need to find A transpose times A,
which I've already done here,
the result is the two by two matrix
where the first row is five, three,
the second row is three, three.
You may want to pause and verify this product.
Next we find A transpose times vector b,
which again I've already done here.
The result there is the vector written as a column matrix
with entries zero, six.
So now that we have A transpose times A
as well as A transpose times vector b,
step two is to set up and solve the equation
A transpose times A times vector x
equals A transpose times vector b,
which gives us the following matrix equation.
Next we write the corresponding augmented matrix,
and write the augmented matrix in reduced row echelon form.
So for the augmented matrix,
the first row is five, three, zero,
the second row is three, three, six.
With the augmented matrix in reduced row echelon form,
notice X one equals negative three, X two equals five,
and therefore the least-squares solution
is the vector negative three, five.
Instead of using vector X with the least-squares solution,
we often use different notation,
in this case, I'm going to use the vector X hat.
So the least-squares solution is X hat equals
the vector negative three, five.
Remember this solution minimizes the distance from
matrix A times the vector X hat to vector b.
Going back to the original problem for a moment,
if there was a solution to the equation
matrix A times vector X equals vector b,
there would've been a vector X one X two
or matrix A times the vector X one X two
is equal to the vector six, zero, zero.
Because there was no solution,
we found the least-squares solution
which gave us the approximate solution
of vector X hat equals to vector negative three, five.
Notice matrix A times the vector X hat
is equal to the vector five, two, one,
not the vector six, zero, zero.
So next question might be,
how close is the vector five, two, one
to the vector six, zero, zero?
We could determine this by determining the magnitude
of the difference of vector b,
and the vector resulting from matrix A times vector X hat
which is the vector five, two, one,
which gives us the magnitude of the difference
of the vector six, zero, zero, and the vector five, two, one
which results in the square root six.
This indicates that the vector five, two, one
is square root of six units from the vector six, zero, zero.
Let's take a look at this graphically.
First the purple plane is the column space of matrix A,
where vector V one is the vector from the first column,
the vector zero, one, two,
and vector V two is the vector from the second column,
the vector one, one, one.
The red point is the terminal point of vector b,
the vector six, zero, zero,
and the blue point is the terminal point
of the vector resulting from matrix A
times the vector X hat, which is the vector five, two, one.
We can also say the vector five, two, one,
where the vector resulting from matrix A
times vector X hat is equal to the vector b
projected onto the column space of matrix A.
And again the distance from the vector five, two, one
and the vector b is equal to this distance here,
the square root six.
So again the least-squares solution minimizes
the distance from matrix A times vector X hat to vector b.
And the components of vector b minus
matrix A times vector X hat
is equal to vector b minus the projection of vector b
onto the column space of matrix A,
which is also equal to the projection of vector b
onto the orthogonal compliment
of the column space of matrix A.
I hope you found this helpful.