Derivation of Line of Best Fit

[tmobilerocks]

Homework Statement

Consider a set of data points: (x1, y1), (x2,y2). One seeks to find the best coefficients A and B such that the sum of squared vertical distances of the data f(x) = Ax + B is minimized. Let D = ∑[y_i - f(x_i]². By requiring the derivatives of D respect to both A and B each to vanish, find expressions for the values of A and B in terms of the data points. Why are these derivatives made to vanish?

Homework Equations

Line of best fit is a linear equation: f(x) = Ax + B
D must be a minimum

The Attempt at a Solution

I am totally lost by this question. I do not understand how to differentiate D with respect to these parameters (maybe implicit differentiation?)

 

[voko]

You have ##D(A, B) = [y_1 - f(A, B, x_1)]^2 + [y_2 - f(A, B, x_2)]^2 ##. ##x_1, x_2, y_1, y_2## are all known and constant, ##A, B## are unknown variables.

 

[tmobilerocks]

Am I supposed to differentiate D? How would I go about doing that?

 

[Ray Vickson]

Am I supposed to differentiate D? How would I go about doing that?

Yes, that is what you are supposed to do. First, though, you need to express ##D## in more explicit form; just use the equation for ##f(x)## at ##x = x_1## and ##x = x_2## to get a sum that contains ##A, B## in fairly simple form. At that point you are supposed to know what to do next, using what you learned in calculus 101.

Are you sure you copied the problem correctly? For just two points you don't need to bother with this "least squares" method; it works, but is unnecessary. However, if you had more than two points you would soon discover the value of the method. BTW: it is almost as easy to do with 1000 points as with two points; all that happens is that you need to evaluate bigger sums. Try it out to see what I mean.

 

[Chestermiller]

Consider the following: If f(x) = Ax + B and D = ∑[y_i - f(x_i]², then

[tex]D=\sum_{i=1}^n(y_i-Ax_i-B)^2[/tex]
Take the partial derivative of D with respect to A and set it equal to zero.
Take the partial derivative of D with respect to B, and set it equal to zero.
Combine the coefficients of A and of B in the resulting equations.
Solve for A and B.

What you are doing is minimizing the deviation of the straight line from the data points (where the data points have experimental error, and thus do not all lie on anyone straight line).

 

[tmobilerocks]

Is this correct?

I took the partial of D with respect to A, and set it equal to zero. I obtained the result

0 = -2x_iy_i + 2Ax_ix_i - 2Bx_i

I then took the partial of D with respect to B, and set it equal to zero. The result I obtained was

0 = -2y_i + 2Ax_i + 2B

After combining these two expressions, and solving for A and B I obtained:

A = (-x_iy_i + y_i -Bx - B)/(x_i-x_ix_i)

B = (-x_iy_i + y_i - Ax_i + Ax_ix_i)/(1+x)

 

[voko]

I do not see how you solved for A and B if both sides of the equation contain them.

 

[tmobilerocks]

Am I on the right track? Is there a simpler way to solve this? Combining both of them makes it nearly impossible to solve

 

[voko]

What is so impossible about that?

The first equation can be rewritten as ## 0 = p + q A + r B ##, where ##p, q, r ## are all made of the known ##x_i, y_i## values. The second equation can be rewritten as ## 0 = u +v A + w B ## in a similar way. Surely you can then solve two linear equations for A and B.

 

[Chestermiller]

Is this correct?

I took the partial of D with respect to A, and set it equal to zero. I obtained the result

0 = -2x_iy_i + 2Ax_ix_i - 2Bx_i

I then took the partial of D with respect to B, and set it equal to zero. The result I obtained was

0 = -2y_i + 2Ax_i + 2B

After combining these two expressions, and solving for A and B I obtained:

A = (-x_iy_i + y_i -Bx - B)/(x_i-x_ix_i)

B = (-x_iy_i + y_i - Ax_i + Ax_ix_i)/(1+x)

In the following two equations, you need to have summation signs on each of the terms, where the summations are over all the data points.

0 = -2x_iy_i + 2Ax_ix_i - 2Bx_i

0 = -2y_i + 2Ax_i + 2B

Once you get the summations in there, you factor out the A and B, and solve these two equations for the two unknowns A and B.

Chet