Modes of variation in PCA

If a random vector ${\displaystyle \mathbf {X} =(X_{1},X_{2},\cdots ,X_{p})^{T}}$ has the mean vector ${\displaystyle {\boldsymbol {\mu }}_{p}}$, and the covariance matrix ${\displaystyle \mathbf {\Sigma } _{p\times p}}$ with eigenvalues ${\displaystyle \lambda _{1}\geq \lambda _{2}\geq \cdots \geq \lambda _{p}\geq 0}$ and corresponding orthonormal eigenvectors ${\displaystyle \mathbf {e} _{1},\mathbf {e} _{2},\cdots ,\mathbf {e} _{p}}$, by eigendecomposition of a real symmetric matrix, the covariance matrix ${\displaystyle \mathbf {\Sigma } }$ can be decomposed as

${\displaystyle \mathbf {\Sigma } =\mathbf {Q} \mathbf {\Lambda } \mathbf {Q} ^{T},}$

where ${\displaystyle \mathbf {Q} }$ is an orthogonal matrix whose columns are the eigenvectors of ${\displaystyle \mathbf {\Sigma } }$, and ${\displaystyle \mathbf {\Lambda } }$ is a diagonal matrix whose entries are the eigenvalues of ${\displaystyle \mathbf {\Sigma } }$. By the Karhunen–Loève expansion for random vectors, one can express the centered random vector in the eigenbasis

${\displaystyle \mathbf {X} -{\boldsymbol {\mu }}=\sum _{k=1}^{p}\xi _{k}\mathbf {e} _{k},}$

where ${\displaystyle \xi _{k}=\mathbf {e} _{k}^{T}(\mathbf {X} -{\boldsymbol {\mu }})}$ is the principal component^[3] associated with the ${\displaystyle k}$-th eigenvector ${\displaystyle \mathbf {e} _{k}}$, with the properties

${\displaystyle \operatorname {E} (\xi _{k})=0,\operatorname {Var} (\xi _{k})=\lambda _{k},}$ and ${\displaystyle \operatorname {E} (\xi _{k}\xi _{l})=0\ {\text{for}}\ l\neq k.}$

Then the ${\displaystyle k}$-th mode of variation of ${\displaystyle \mathbf {X} }$ is the set of vectors, indexed by ${\displaystyle \alpha }$,

${\displaystyle \mathbf {m} _{k,\alpha }={\boldsymbol {\mu }}\pm \alpha {\sqrt {\lambda _{k}}}\mathbf {e} _{k},\alpha \in [-A,A],}$

where ${\displaystyle A}$ is typically selected as ${\displaystyle 2\ {\text{or}}\ 3}$.

Modes of variation in FPCA

For a square-integrable random function ${\displaystyle X(t),t\in {\mathcal {T}}\subset R^{p}}$, where typically ${\displaystyle p=1}$ and ${\displaystyle {\mathcal {T}}}$ is an interval, denote the mean function by ${\displaystyle \mu (t)=\operatorname {E} (X(t))}$, and the covariance function by

${\displaystyle G(s,t)=\operatorname {Cov} (X(s),X(t))=\sum _{k=1}^{\infty }\lambda _{k}\varphi _{k}(s)\varphi _{k}(t),}$

where ${\displaystyle \lambda _{1}\geq \lambda _{2}\geq \cdots \geq 0}$ are the eigenvalues and ${\displaystyle \{\varphi _{1},\varphi _{2},\cdots \}}$ are the orthonormal eigenfunctions of the linear Hilbert–Schmidt operator

${\displaystyle G:L^{2}({\mathcal {T}})\rightarrow L^{2}({\mathcal {T}}),\,G(f)=\int _{\mathcal {T}}G(s,t)f(s)ds.}$

By the Karhunen–Loève theorem, one can express the centered function in the eigenbasis,

${\displaystyle X(t)-\mu (t)=\sum _{k=1}^{\infty }\xi _{k}\varphi _{k}(t),}$

where

${\displaystyle \xi _{k}=\int _{\mathcal {T}}(X(t)-\mu (t))\varphi _{k}(t)dt}$

is the ${\displaystyle k}$-th principal component with the properties

${\displaystyle \operatorname {E} (\xi _{k})=0,\operatorname {Var} (\xi _{k})=\lambda _{k},}$ and ${\displaystyle \operatorname {E} (\xi _{k}\xi _{l})=0{\text{ for }}l\neq k.}$

Then the ${\displaystyle k}$-th mode of variation of ${\displaystyle X(t)}$ is the set of functions, indexed by ${\displaystyle \alpha }$,

${\displaystyle m_{k,\alpha }(t)=\mu (t)\pm \alpha {\sqrt {\lambda _{k}}}\varphi _{k}(t),\ t\in {\mathcal {T}},\ \alpha \in [-A,A]}$

that are viewed simultaneously over the range of ${\displaystyle \alpha }$, usually for ${\displaystyle A=2\ {\text{or}}\ 3}$.^[2]

Modes of variation in PCA

Suppose the data ${\displaystyle \mathbf {x} _{1},\mathbf {x} _{2},\cdots ,\mathbf {x} _{n}}$ represent ${\displaystyle n}$ independent drawings from some ${\displaystyle p}$-dimensional population ${\displaystyle \mathbf {X} }$ with mean vector ${\displaystyle {\boldsymbol {\mu }}}$ and covariance matrix ${\displaystyle \mathbf {\Sigma } }$. These data yield the sample mean vector ${\displaystyle {\overline {\mathbf {x} }}}$, and the sample covariance matrix ${\displaystyle \mathbf {S} }$ with eigenvalue-eigenvector pairs ${\displaystyle ({\hat {\lambda }}_{1},{\hat {\mathbf {e} }}_{1}),({\hat {\lambda }}_{2},{\hat {\mathbf {e} }}_{2}),\cdots ,({\hat {\lambda }}_{p},{\hat {\mathbf {e} }}_{p})}$. Then the ${\displaystyle k}$-th mode of variation of ${\displaystyle \mathbf {X} }$ can be estimated by

${\displaystyle {\hat {\mathbf {m} }}_{k,\alpha }={\overline {\mathbf {x} }}\pm \alpha {\sqrt {{\hat {\lambda }}_{k}}}{\hat {\mathbf {e} }}_{k},\alpha \in [-A,A].}$

Modes of variation in FPCA

Consider ${\displaystyle n}$ realizations ${\displaystyle X_{1}(t),X_{2}(t),\cdots ,X_{n}(t)}$ of a square-integrable random function ${\displaystyle X(t),t\in {\mathcal {T}}}$ with the mean function ${\displaystyle \mu (t)=\operatorname {E} (X(t))}$ and the covariance function ${\displaystyle G(s,t)=\operatorname {Cov} (X(s),X(t))}$. Functional principal component analysis provides methods for the estimation of ${\displaystyle \mu (t)}$ and ${\displaystyle G(s,t)}$ in detail, often involving point wise estimate and interpolation. Substituting estimates for the unknown quantities, the ${\displaystyle k}$-th mode of variation of ${\displaystyle X(t)}$ can be estimated by

${\displaystyle {\hat {m}}_{k,\alpha }(t)={\hat {\mu }}(t)\pm \alpha {\sqrt {{\hat {\lambda }}_{k}}}{\hat {\varphi }}_{k}(t),t\in {\mathcal {T}},\alpha \in [-A,A].}$