In statistics, sufficient dimension reduction (SDR) is a paradigm for analyzing data that combines the ideas of dimension reduction with the concept of sufficiency.

Dimension reduction has long been a primary goal of regression analysis. Given a response variable y and a p-dimensional predictor vector ${\displaystyle {\textbf {x}}}$, regression analysis aims to study the distribution of ${\displaystyle y\mid {\textbf {x}}}$, the conditional distribution of ${\displaystyle y}$ given ${\displaystyle {\textbf {x}}}$. A dimension reduction is a function ${\displaystyle R({\textbf {x}})}$ that maps ${\displaystyle {\textbf {x}}}$ to a subset of ${\displaystyle \mathbb {R} ^{k}}$, k < p, thereby reducing the dimension of ${\displaystyle {\textbf {x}}}$.^[1] For example, ${\displaystyle R({\textbf {x}})}$ may be one or more linear combinations of ${\displaystyle {\textbf {x}}}$.

A dimension reduction ${\displaystyle R({\textbf {x}})}$ is said to be sufficient if the distribution of ${\displaystyle y\mid R({\textbf {x}})}$ is the same as that of ${\displaystyle y\mid {\textbf {x}}}$. In other words, no information about the regression is lost in reducing the dimension of ${\displaystyle {\textbf {x}}}$ if the reduction is sufficient.^[1]

Graphical motivation

In a regression setting, it is often useful to summarize the distribution of ${\displaystyle y\mid {\textbf {x}}}$ graphically. For instance, one may consider a scatter plot of ${\displaystyle y}$ versus one or more of the predictors. A scatter plot that contains all available regression information is called a sufficient summary plot.

When ${\displaystyle {\textbf {x}}}$ is high-dimensional, particularly when ${\displaystyle p\geq 3}$, it becomes increasingly challenging to construct and visually interpret sufficiency summary plots without reducing the data. Even three-dimensional scatter plots must be viewed via a computer program, and the third dimension can only be visualized by rotating the coordinate axes. However, if there exists a sufficient dimension reduction ${\displaystyle R({\textbf {x}})}$ with small enough dimension, a sufficient summary plot of ${\displaystyle y}$ versus ${\displaystyle R({\textbf {x}})}$ may be constructed and visually interpreted with relative ease.

Hence sufficient dimension reduction allows for graphical intuition about the distribution of ${\displaystyle y\mid {\textbf {x}}}$, which might not have otherwise been available for high-dimensional data.

Most graphical methodology focuses primarily on dimension reduction involving linear combinations of ${\displaystyle {\textbf {x}}}$. The rest of this article deals only with such reductions.

Dimension reduction subspace

Suppose ${\displaystyle R({\textbf {x}})=A^{T}{\textbf {x}}}$ is a sufficient dimension reduction, where ${\displaystyle A}$ is a ${\displaystyle p\times k}$ matrix with rank ${\displaystyle k\leq p}$. Then the regression information for ${\displaystyle y\mid {\textbf {x}}}$ can be inferred by studying the distribution of ${\displaystyle y\mid A^{T}{\textbf {x}}}$, and the plot of ${\displaystyle y}$ versus ${\displaystyle A^{T}{\textbf {x}}}$ is a sufficient summary plot.

Without loss of generality, only the space spanned by the columns of ${\displaystyle A}$ need be considered. Let ${\displaystyle \eta }$ be a basis for the column space of ${\displaystyle A}$, and let the space spanned by ${\displaystyle \eta }$ be denoted by ${\displaystyle {\mathcal {S}}(\eta )}$. It follows from the definition of a sufficient dimension reduction that

${\displaystyle F_{y\mid x}=F_{y\mid \eta ^{T}x},}$

where ${\displaystyle F}$ denotes the appropriate distribution function. Another way to express this property is

${\displaystyle y\perp \!\!\!\perp {\textbf {x}}\mid \eta ^{T}{\textbf {x}},}$

or ${\displaystyle y}$ is conditionally independent of ${\displaystyle {\textbf {x}}}$, given ${\displaystyle \eta ^{T}{\textbf {x}}}$. Then the subspace ${\displaystyle {\mathcal {S}}(\eta )}$ is defined to be a dimension reduction subspace (DRS).^[2]

Central subspace

If a subspace ${\displaystyle {\mathcal {S}}}$ is a DRS for ${\displaystyle y\mid {\textbf {x}}}$, and if ${\displaystyle {\mathcal {S}}\subset {\mathcal {S}}_{drs}}$ for all other DRSs ${\displaystyle {\mathcal {S}}_{drs}}$, then it is a central dimension reduction subspace, or simply a central subspace, and it is denoted by ${\displaystyle {\mathcal {S}}_{y\mid x}}$. In other words, a central subspace for ${\displaystyle y\mid {\textbf {x}}}$ exists if and only if the intersection ${\textstyle \bigcap {\mathcal {S}}_{drs}}$ of all dimension reduction subspaces is also a dimension reduction subspace, and that intersection is the central subspace ${\displaystyle {\mathcal {S}}_{y\mid x}}$.^[2]

The central subspace ${\displaystyle {\mathcal {S}}_{y\mid x}}$ does not necessarily exist because the intersection ${\textstyle \bigcap {\mathcal {S}}_{drs}}$ is not necessarily a DRS. However, if ${\displaystyle {\mathcal {S}}_{y\mid x}}$ does exist, then it is also the unique minimum dimension reduction subspace.^[2]

Methods for dimension reduction

There are many existing methods for dimension reduction, both graphical and numeric. For example, sliced inverse regression (SIR) and sliced average variance estimation (SAVE) were introduced in the 1990s and continue to be widely used.^[3] Although SIR was originally designed to estimate an effective dimension reducing subspace, it is now understood that it estimates only the central subspace, which is generally different.

More recent methods for dimension reduction include likelihood-based sufficient dimension reduction,^[4] estimating the central subspace based on the inverse third moment (or kth moment),^[5] estimating the central solution space,^[6] graphical regression,^[2] envelope model, and the principal support vector machine.^[7] For more details on these and other methods, consult the statistical literature.

Principal components analysis (PCA) and similar methods for dimension reduction are not based on the sufficiency principle.

See also

• Dimension reduction
• Sliced inverse regression
• Principal component analysis
• Linear discriminant analysis
• Curse of dimensionality
• Multilinear subspace learning

Notes

References

External links

• Sufficient Dimension Reduction