Parameter errors and correlation

The estimated parameter values are linear combinations of the observed values

$${\displaystyle {\hat {\boldsymbol {\beta }}}=(X^{\textsf {T}}WX)^{-1}X^{\textsf {T}}W\mathbf {y} .}$$

Therefore, an expression for the estimated variance-covariance matrix of the parameter estimates can be obtained by error propagation from the errors in the observations. Let the variance-covariance matrix for the observations be denoted by M and that of the estimated parameters by M^β. Then

$${\displaystyle M^{\beta }=\left(X^{\textsf {T}}WX\right)^{-1}X^{\textsf {T}}WMW^{\textsf {T}}X\left(X^{\textsf {T}}W^{\textsf {T}}X\right)^{-1}.}$$

When W = M⁻¹, this simplifies to

$${\displaystyle M^{\beta }=\left(X^{\textsf {T}}WX\right)^{-1}.}$$

When unit weights are used (W = I, the identity matrix), it is implied that the experimental errors are uncorrelated and all equal: M = σ²I, where σ² is the a priori variance of an observation. In any case, σ² is approximated by the reduced chi-squared ${\displaystyle \chi _{\nu }^{2}}$:

$${\displaystyle {\begin{aligned}M^{\beta }&=\chi _{\nu }^{2}\left(X^{\textsf {T}}WX\right)^{-1},\\\chi _{\nu }^{2}&=S/\nu ,\end{aligned}}}$$

where S is the minimum value of the weighted objective function:

$${\displaystyle S=r^{\textsf {T}}Wr=\left\|W^{\frac {1}{2}}\left(\mathbf {y} -X{\hat {\boldsymbol {\beta }}}\right)\right\|^{2}.}$$

The denominator, ${\displaystyle \nu =n-m}$, is the number of degrees of freedom; see effective degrees of freedom for generalizations for the case of correlated observations.

In all cases, the variance of the parameter estimate ${\displaystyle {\hat {\beta }}_{i}}$ is given by ${\displaystyle M_{ii}^{\beta }}$ and the covariance between the parameter estimates ${\displaystyle {\hat {\beta }}_{i}}$ and ${\displaystyle {\hat {\beta }}_{j}}$ is given by ${\displaystyle M_{ij}^{\beta }}$. The standard deviation is the square root of variance, ${\displaystyle \sigma _{i}={\sqrt {M_{ii}^{\beta }}}}$, and the correlation coefficient is given by ${\displaystyle \rho _{ij}=M_{ij}^{\beta }/(\sigma _{i}\sigma _{j})}$. These error estimates reflect only random errors in the measurements. The true uncertainty in the parameters is larger due to the presence of systematic errors, which, by definition, cannot be quantified. Note that even though the observations may be uncorrelated, the parameters are typically correlated.

Parameter confidence limits

It is often assumed, for want of any concrete evidence but often appealing to the central limit theorem—see Normal distribution#Occurrence and applications—that the error on each observation belongs to a normal distribution with a mean of zero and standard deviation ${\displaystyle \sigma }$. Under that assumption the following probabilities can be derived for a single scalar parameter estimate in terms of its estimated standard error ${\displaystyle se_{\beta }}$ (given here):

• 68% that the interval ${\displaystyle {\hat {\beta }}\pm se_{\beta }}$ encompasses the true coefficient value
• 95% that the interval ${\displaystyle {\hat {\beta }}\pm 2se_{\beta }}$ encompasses the true coefficient value
• 99% that the interval ${\displaystyle {\hat {\beta }}\pm 2.5se_{\beta }}$ encompasses the true coefficient value

The assumption is not unreasonable when n >> m. If the experimental errors are normally distributed the parameters will belong to a Student's t-distribution with n − m degrees of freedom. When n ≫ m Student's t-distribution approximates a normal distribution. Note, however, that these confidence limits cannot take systematic error into account. Also, parameter errors should be quoted to one significant figure only, as they are subject to sampling error.^[4]

When the number of observations is relatively small, Chebychev's inequality can be used for an upper bound on probabilities, regardless of any assumptions about the distribution of experimental errors: the maximum probabilities that a parameter will be more than 1, 2, or 3 standard deviations away from its expectation value are 100%, 25% and 11% respectively.

Residual values and correlation

The residuals are related to the observations by

$${\displaystyle \mathbf {\hat {r}} =\mathbf {y} -X{\hat {\boldsymbol {\beta }}}=\mathbf {y} -H\mathbf {y} =(I-H)\mathbf {y} ,}$$

where H is the idempotent matrix known as the hat matrix:

$${\displaystyle H=X\left(X^{\textsf {T}}WX\right)^{-1}X^{\textsf {T}}W,}$$

and I is the identity matrix. The variance-covariance matrix of the residuals, M ^r is given by

$${\displaystyle M^{\mathbf {r} }=(I-H)M(I-H)^{\textsf {T}}.}$$

Thus the residuals are correlated, even if the observations are not.

When ${\displaystyle W=M^{-1}}$,

$${\displaystyle M^{\mathbf {r} }=(I-H)M.}$$

The sum of weighted residual values is equal to zero whenever the model function contains a constant term. Left-multiply the expression for the residuals by X^T W^T:

$${\displaystyle X^{\textsf {T}}W{\hat {\mathbf {r} }}=X^{\textsf {T}}W\mathbf {y} -X^{\textsf {T}}WX{\hat {\boldsymbol {\beta }}}=X^{\textsf {T}}W\mathbf {y} -\left(X^{\rm {T}}WX\right)\left(X^{\textsf {T}}WX\right)^{-1}X^{\textsf {T}}W\mathbf {y} =\mathbf {0} .}$$

Say, for example, that the first term of the model is a constant, so that ${\displaystyle X_{i1}=1}$ for all i. In that case it follows that

$${\displaystyle \sum _{i}^{m}X_{i1}W_{i}{\hat {r}}_{i}=\sum _{i}^{m}W_{i}{\hat {r}}_{i}=0.}$$

Thus, in the motivational example, above, the fact that the sum of residual values is equal to zero is not accidental, but is a consequence of the presence of the constant term, α, in the model.

If experimental error follows a normal distribution, then, because of the linear relationship between residuals and observations, so should residuals,^[5] but since the observations are only a sample of the population of all possible observations, the residuals should belong to a Student's t-distribution. Studentized residuals are useful in making a statistical test for an outlier when a particular residual appears to be excessively large.