Simple Linear Regression - Weighted Least Squares

$In\; figure\; 7\; the\; graph\; of\; Y\; versus\; X\; exhibits\; another\; type\; of\; assumption\; violation.$

The variability of the Ys appears to increase with increasing X. A linear least squares fit to these data results in the graph of studentized residuals versus X shown in figure 8. Seemingly the variability of the errors increases with X; the scatter of points does not at all approximate a horizontal band.

Under these circumstances weighted least squares methodology should be applied where each X, Y data point potentially receives a different weight. The appropriate weight to assign is one which is proportional to how well a given Y is known or inversely proportional to the variability of Y. In the case under consideration, figure 8 indicates that the variability of the errors (and thus Ys) increases almost linearly with X. A potential weight might thus be 1.0/X.

A weighted least squares problem may be converted to a ordinary least squares problem. If for the equation

Y_i = a + b X_i

the variability of the errors (and thus Y) is proportional to W_i (a weighted least squares problem) the equation

can be fit by ordinary least squares methodology. For the example under consideration W_i = X_i so that the appropriate equation is

that is, the response variable Y_i/(square root X_i) would be fit to two regressor variables 1.0/(square root X_i) and (square root X_i) with the intercept excluded from consideration. This is a multiple linear regression problem.

Note that applying a transformation changes the errors in the underlying equation as well. Residuals will need to be transformed to be properly interpreted. For the example just given one would plot residuals/(square root X_i) versus (square root X_i).

One special weighted least squares problem can be analyzed using the formulas presented earlier. When the variability of Y increases proportional to X² the appropriate equation to fit is

which can be rewritten as

Y' = b₀ + b₁ X'

with Y' = Y/X, X' = 1/X, b₀ = b, and b₁ = a. The equations presented earlier can be used to estimate b₀ and b₁ using Y' and X'. One must be careful, however, to ascribe the correct interpretation to bhat₀ and bhat₁.