39  LM6. Residuals

39.1 Residuals

Model \[Y = X \beta + \epsilon\]

\[\begin{equation*} \begin{pmatrix} Y_1\\ Y_2\\ \vdots \\ Y_n\\ \end{pmatrix} = \begin{pmatrix} 1 & x_{11} & x_{21} & \ldots & x_{k1} \\ 1 & x_{12} & x_{22} & \ldots & x_{k2} \\ \vdots & \vdots & \vdots & \vdots \\ 1 & x_{1n} & x_{2n} & \ldots & x_{kn} \\ \end{pmatrix} \begin{pmatrix} \beta_0\\ \beta_1\\ \vdots \\ \beta_k\\ \end{pmatrix} + \begin{pmatrix} \epsilon_1\\ \epsilon_2\\ \vdots \\ \epsilon_n\\ \end{pmatrix} \end{equation*}\]

\(\hat Y = X \hat \beta = X(X'X)^{-1}X' Y = H Y\)

\(e = Y - \hat Y = (I-H) Y\)

Prove

  • \(E(e) = 0\)

  • \(Var(e) = \sigma^2 (I-H)\)

  • Sum of the residuals is 0, \(\sum_{i=1}^n e_i = 0\)

  • Sample mean of the residuals is 0, \(\frac{1}{n} \sum_{i=1}^n e_i = 0\)

  • Mean of observed \(y\)s equal to the mean of the predicted \(y\)s, \[\bar y = \frac{1}{n} \sum_{i=1}^n y_i = \frac{1}{n} \sum_{i=1}^n \hat y_i = \bar {\hat y}\]

Residuals. Solutions

\(e = Y - \hat Y = (I-H) Y = (I-H)(X\beta+\epsilon)= (I-H)X\beta + (I-H)\epsilon =\)

\(= (I-X(X'X)^{-1}X')X\beta + (I-H)\epsilon = (X\beta-X\beta) + (I-H)\epsilon = (I-H)\epsilon\)

\(E(e) = 0\)


Since \(Var(\epsilon)=\sigma^2 I\),

\(Var(e) = Var((I-H)\epsilon) = (I-H)\sigma^2(I-H)'=\sigma^2(I-H)\)

Note that \(H\) and \(I-H\) are symmetric (\(A = A'\)) and idempotent (\(A^2=A\))


\[X'Y = (X'X) \hat \beta \mbox{ (Normal equations)}\] \[X'(X \hat \beta + e) = (X'X) \hat \beta\]

\[X'e = 0\]

\[\begin{equation*} \begin{pmatrix} 1 & 1 & \ldots & 1 \\ x_{11} & x_{12} & \ldots & x_{1n} \\ x_{21} & x_{22} & \ldots & x_{2n} \\ \vdots & \vdots & \vdots & \vdots \\ x_{k1} & x_{k2} & \ldots & x_{kn} \\ \end{pmatrix} \begin{pmatrix} e_1\\ e_2\\ \vdots \\ e_n\\ \end{pmatrix} = \begin{pmatrix} 1 e_1 + 1 e_2 + \ldots + 1 e_n \\ x_{11} e_1 + x_{12} e_2 + \ldots + x_{1n} e_n \\ x_{21} e_1 + x_{22} e_2 + \ldots + x_{2n} e_n \\ \vdots \\ x_{k1} e_1 + x_{k2} e_2 + \ldots + x_{kn} e_n \\ \end{pmatrix} = \begin{pmatrix} 0\\ 0\\ 0\\ \vdots \\ 0\\ \end{pmatrix} \end{equation*}\]

From this we can derive the properties:

  • The observed values of \(X\) are uncorrelated with the residuals.

\(X'e = 0\) implies that for every column \(x_k\) of \(X\), \(x_k' e = 0\). In other words, each regressor has zero sample correlation with the residuals.


  • Sum of the residuals is 0, \(\sum_{i=1}^n e_i = 0\).

If there is an intercept, then the first column in \(X\) will be a column of ones. This means that for the first element in \(X' e\) is zero. Then, \(e_1 + e_2 + \ldots + e_n = \sum e_i = 0\).


  • Sample mean of the residuals is 0, \(\frac{1}{n} \sum_{i=1}^n e_i = 0\).

This follows from the previous property \(\bar e = \frac{\sum e_i}{n} = 0\).


  • Mean of observed \(y\)s equal to the mean of the predicted \(y\)s \[\bar y = \frac{1}{n} \sum_{i=1}^n y_i = \frac{1}{n} \sum_{i=1}^n \hat y_i = \bar {\hat y}\]

\(\bar y = \bar {\hat y}\)

\(\sum e_i = \sum (y_i- \hat y_i) = \sum y_i - \sum \hat y_i = 0\) (since sum of residuals \(\sum e_i = 0\)).

Then, dividing by \(n\), \(\bar y = \bar {\hat y}\)


39.2 Analysis of variance

Prove \[SST = SSM + SSE\]

Analysis of variance. Solutions

SST

\[SST = \sum(Y_i - \bar Y)^2 = \sum(Y_i^2 - 2 Y_i \bar Y + \bar Y^2) = \sum Y_i^2 - 2 \bar Y \sum Y_i + n \bar Y^2 = \sum Y_i^2 - n \bar Y^2 = Y'Y - n \bar Y^2\] \((\sum \hat Y_i = n \bar Y)\)

SSM

\[SSM = \sum (\hat Y_i - \bar Y)^2 = \sum (\hat Y_i^2 - 2 \hat Y_i \bar Y + \bar Y^2) = \sum \hat Y_i^2 - 2 \bar Y \sum \hat Y_i + n \bar Y^2\]

\((\sum \hat Y_i = n \bar Y)\)

\[= \sum \hat Y_i^2 - n \bar Y^2 = \hat Y' \hat Y - n \bar Y^2 = \hat \beta' X' X \hat \beta - n \bar Y^2 = Y' X (X'X)^{-1} \ X'X \ (X'X)^{-1} X'Y - n \bar Y^2\]

\((X'X (X'X)^{-1} = I)\)

\[= Y' H Y - n \bar Y^2\]

Note that \(\hat \beta' X' X \hat \beta\) could also be expressed as \(\hat \beta' X' Y\) so another way of writing SSM is as \(\hat \beta' X' Y - n \bar Y^2\)

SSE

\[SSE = e'e = Y'(I-H)' (I-H) Y = Y' (I-H-H+HH)Y = Y' (I-H) Y\]

SST = SSM + SSE

\[SSM + SSE = Y' H Y - n \bar Y^2 + Y' (I-H)Y = Y'Y - n \bar Y^2 = SST\]