In statistics, a studentized residual is the quotient resulting from the division
of a residual by an estimate of its standard deviation. Typically the
standard deviations of residuals in a sample vary greatly from one data point
to another even when the errors all have the same standard deviation,
particularly in regression analysis; thus it does not make sense to compare
residuals at different data points without first studentizing. It is a form
of a Student's t-statistic, with the estimate of error varying between
points. This is an important technique in the
detection of outliers. It is among several named in honor of William Sealey
Gosset, who wrote under the pseudonym Student, and dividing by an estimate of
scale is called studentizing, in analogy with standardizing and normalizing
Motivation The key reason for studentizing is that,
in regression analysis of a multivariate distribution, the variances of the
residuals at different input variable values may differ, even if the variances
of the errors at these different input variable values are equal. The issue is
the difference between errors and residuals in statistics, particularly
the behavior of residuals in regressions.
Consider the simple linear regression model
Given a random sample, i = 1, ..., n, each pair satisfies
where the errors εi, are independent and all have the same variance σ2. The
residuals are not the true, and unobservable, errors, but rather are
estimates, based on the observable data, of the errors. When the method of least
squares is used to estimate α0 and α1, then the residuals , unlike the errors ,
cannot be independent since they satisfy the two constraints
and (Here εi is the ith error, and is the
ith residual.) Moreover, and most importantly, the
residuals, unlike the errors, do not all have the same variance: the variance
decreases as the corresponding x-value gets farther from the average x-value.
This is a feature of the regression better fitting values at the ends of the
domain, not the data itself, and is also reflected in the influence functions of
various data points on the regression coefficients: endpoints have more
influence. This can also be seen because the residuals at endpoints depend
greatly on the slope of a fitted line, while the residuals at the middle are
relatively insensitive to the slope. The fact that the variances of the residuals
differ, even though the variances of the true errors are all equal to each other,
is the principal reason for the need for studentization.
It is not simply a matter of the population parameters being unknown – it
is that regressions yield different residual distributions at different data
points, unlike point estimators of univariate distributions, which share a
common distribution for residuals. How to studentize
For this simple model, the design matrix is
and the hat matrix H is the matrix of the orthogonal projection onto the
column space of the design matrix: The leverage hii is the ith diagonal
entry in the hat matrix. The variance of the ith residual is
In case the design matrix X has only two columns, this is equal to
The corresponding studentized residual is then
where is an appropriate estimate of σ. Internal and external studentization
The usual estimate of σ2 is where m is the number of parameters in
the model. But it is desirable to exclude the ith observation from the
process of estimating the variance when one is considering whether the ith case
may be an outlier. Consequently, one may use the estimate
based on all but the ith residual. If the former estimate σ2 is used,
including the ith case, then the residual is said to be internally
studentized, . If the latter estimate is used instead, excluding the ith case,
then it is said to be externally studentized, .
Distribution If the errors are independent and
normally distributed with expected value 0 and variance σ2, then the probability
distribution of the ith externally studentized residual is a Student's
t-distribution with n − m − 1 degrees of freedom, and can range from to .
On the other hand, the internally studentized residuals are in the range ,
where ν = n − m is the number of residual degrees of freedom. If ti
represents the internally studentized residual, and again assuming that the
errors are independent identically distributed Gaussian variables, then:
where t is a random variable distributed as Student's t-distribution with ν − 1
degrees of freedom. In fact, this implies that ti /ν follows the beta
distribution B(1/2,(ν − 1)/2). The distribution above is sometimes referred
to as the tau distribution; it was first derived by Thompson in 1935.
When ν = 3, the internally studentized residuals are uniformly distributed
between and . If there is only one residual degree of freedom, the above
formula for the distribution of internally studentized residuals doesn't
apply. In this case, the ti are all either +1 or −1, with 50% chance for
each. The standard deviation of the
distribution of internally studentized residuals is always 1, but this does not
imply that the standard deviation of all the ti of a particular experiment is 1.
For instance, the internally studentized residuals when fitting a straight line
going through to the points,, are , and the standard deviation of these is not
1. See also
Normalization Samuelson's inequality
Standard score William Sealy Gosset
References Cook, R. Dennis; Weisberg, Sanford.
Residuals and Influence in Regression.. New York: Chapman and Hall. ISBN
041224280X. Retrieved 23 February 2013.