Why must least squares be used to approximate a solution to an overdetermined system?

In an overdetermined system, there are more data points than absolutely needed to solve the system. A very simple example would be trying to find the equation of a line:

#y=mx+b#

If we have two points, we have exactly enough information to find the equation of the line and the line that we find will always go through these two points as in the following graph:

graph{((x-1)^2+(y-1)^2-.01)((x-5)^2+(y-2)^2-.01)(0.25*x+0.75-y)=0 [-3.3, 10.75, -1.68, 5.34]}

if we are given a third point, thereby overdetermining the system, there is only one case where there will be only one possible solution the new point must lie exactly on the previous line. Even the smallest error in the determination of the points will cause a discrepancy in the solutions which one would find using any pair of points.

graph{((x-9)^2+(y-3.2)^2-.01)((x-1)^2+(y-1)^2-.01)((x-5)^2+(y-2)^2-.01)(0.25x+0.75-y)=0 [-3.3, 10.75, -1.68, 5.34]}

Clearly, our previous line doesn't hit the new point. We need a way to find the best fit for our solution. Least squares is the most common way to do this. It gives us an average solution that minimizes the mean-squared error between our solution and the data points we provide.

graph{((x-9)^2+(y-3.2)^2-.01)((x-1)^2+(y-1)^2-.01)((x-5)^2+(y-2)^2-.01)(0.275x+0.692-y)=0 [-3.3, 10.75, -1.68, 5.34]}