Description

Instead of solving the original linear system ${\displaystyle Ax=b}$ for ${\displaystyle x}$, one may consider the right preconditioned system

$${\displaystyle AP^{-1}(Px)=b}$$

and solve

$${\displaystyle AP^{-1}y=b}$$

for ${\displaystyle y}$ and

$${\displaystyle Px=y}$$

for ${\displaystyle x}$.

Alternatively, one may solve the left preconditioned system

$${\displaystyle P^{-1}(Ax-b)=0.}$$

Both systems give the same solution as the original system as long as the preconditioner matrix ${\displaystyle P}$ is nonsingular. The left preconditioning is more traditional.

The two-sided preconditioned system

$${\displaystyle QAP^{-1}(Px)=Qb}$$

may be beneficial, e.g., to preserve the matrix symmetry: if the original matrix ${\displaystyle A}$ is real symmetric and real preconditioners ${\displaystyle Q}$ and ${\displaystyle P}$ satisfy ${\displaystyle Q^{T}=P^{-1}}$ then the preconditioned matrix ${\displaystyle QAP^{-1}}$ is also symmetric. The two-sided preconditioning is common for diagonal scaling where the preconditioners ${\displaystyle Q}$ and ${\displaystyle P}$ are diagonal and scaling is applied both to columns and rows of the original matrix ${\displaystyle A}$, e.g., in order to decrease the dynamic range of entries of the matrix.

The goal of preconditioning is reducing the condition number, e.g., of the left or right preconditioned system matrix ${\displaystyle P^{-1}A}$ or ${\displaystyle AP^{-1}}$. Small condition numbers benefit fast convergence of iterative solvers and improve stability of the solution with respect to perturbations in the system matrix and the right-hand side, e.g., allowing for more aggressive quantization of the matrix entries using lower computer precision.

The preconditioned matrix ${\displaystyle P^{-1}A}$ or ${\displaystyle AP^{-1}}$ is rarely explicitly formed. Only the action of applying the preconditioner solve operation ${\displaystyle P^{-1}}$ to a given vector may need to be computed.

Typically there is a trade-off in the choice of ${\displaystyle P}$. Since the operator ${\displaystyle P^{-1}}$ must be applied at each step of the iterative linear solver, it should have a small cost (computing time) of applying the ${\displaystyle P^{-1}}$ operation. The cheapest preconditioner would therefore be ${\displaystyle P=I}$ since then ${\displaystyle P^{-1}=I.}$ Clearly, this results in the original linear system and the preconditioner does nothing. At the other extreme, the choice ${\displaystyle P=A}$ gives ${\displaystyle P^{-1}A=AP^{-1}=I,}$ which has optimal condition number of 1, requiring a single iteration for convergence; however in this case ${\displaystyle P^{-1}=A^{-1},}$ and applying the preconditioner is as difficult as solving the original system. One therefore chooses ${\displaystyle P}$ as somewhere between these two extremes, in an attempt to achieve a minimal number of linear iterations while keeping the operator ${\displaystyle P^{-1}}$ as simple as possible. Some examples of typical preconditioning approaches are detailed below.

Preconditioned iterative methods

Preconditioned iterative methods for ${\displaystyle Ax-b=0}$ are, in most cases, mathematically equivalent to standard iterative methods applied to the preconditioned system ${\displaystyle P^{-1}(Ax-b)=0.}$ For example, the standard Richardson iteration for solving ${\displaystyle Ax-b=0}$ is

$${\displaystyle \mathbf {x} _{n+1}=\mathbf {x} _{n}-\gamma _{n}(A\mathbf {x} _{n}-\mathbf {b} ),\ n\geq 0.}$$

Applied to the preconditioned system ${\displaystyle P^{-1}(Ax-b)=0,}$ it turns into a preconditioned method

$${\displaystyle \mathbf {x} _{n+1}=\mathbf {x} _{n}-\gamma _{n}P^{-1}(A\mathbf {x} _{n}-\mathbf {b} ),\ n\geq 0.}$$

Examples of popular preconditioned iterative methods for linear systems include the preconditioned conjugate gradient method, the biconjugate gradient method, and generalized minimal residual method. Iterative methods, which use scalar products to compute the iterative parameters, require corresponding changes in the scalar product together with substituting ${\displaystyle P^{-1}(Ax-b)=0}$ for ${\displaystyle Ax-b=0.}$

Matrix splitting

A stationary iterative method is determined by the matrix splitting ${\displaystyle A=M-N}$ and the iteration matrix ${\displaystyle C=I-M^{-1}A}$. Assuming that

• the system matrix ${\displaystyle A}$ is symmetric positive-definite,
• the splitting matrix ${\displaystyle M}$ is symmetric positive-definite,
• the stationary iterative method is convergent, as determined by ${\displaystyle \rho (C)<1}$,

the condition number ${\displaystyle \kappa (M^{-1}A)}$ is bounded above by

$${\displaystyle \kappa (M^{-1}A)\leq {\frac {1+\rho (C)}{1-\rho (C)}}\,.}$$

Geometric interpretation

For a symmetric positive definite matrix ${\displaystyle A}$ the preconditioner ${\displaystyle P}$ is typically chosen to be symmetric positive definite as well. The preconditioned operator ${\displaystyle P^{-1}A}$ is then also symmetric positive definite, but with respect to the ${\displaystyle P}$-based scalar product. In this case, the desired effect in applying a preconditioner is to make the quadratic form of the preconditioned operator ${\displaystyle P^{-1}A}$ with respect to the ${\displaystyle P}$-based scalar product to be nearly spherical.^[1]

Variable and non-linear preconditioning

Denoting ${\displaystyle T=P^{-1}}$, we highlight that preconditioning is practically implemented as multiplying some vector ${\displaystyle r}$ by ${\displaystyle T}$, i.e., computing the product ${\displaystyle Tr.}$ In many applications, ${\displaystyle T}$ is not given as a matrix, but rather as an operator ${\displaystyle T(r)}$ acting on the vector ${\displaystyle r}$. Some popular preconditioners, however, change with ${\displaystyle r}$ and the dependence on ${\displaystyle r}$ may not be linear. Typical examples involve using non-linear iterative methods, e.g., the conjugate gradient method, as a part of the preconditioner construction. Such preconditioners may be practically very efficient, however, their behavior is hard to predict theoretically.

Random preconditioning

One interesting particular case of variable preconditioning is random preconditioning, e.g., multigrid preconditioning on random course grids.^[2] If used in gradient descent methods, random preconditioning can be viewed as an implementation of stochastic gradient descent and can lead to faster convergence, compared to fixed preconditioning, since it breaks the asymptotic "zig-zag" pattern of the gradient descent.

Spectrally equivalent preconditioning

The most common use of preconditioning is for iterative solution of linear systems resulting from approximations of partial differential equations. The better the approximation quality, the larger the matrix size is. In such a case, the goal of optimal preconditioning is, on the one side, to make the spectral condition number of ${\displaystyle P^{-1}A}$ to be bounded from above by a constant independent of the matrix size, which is called spectrally equivalent preconditioning by D'yakonov. On the other hand, the cost of application of the ${\displaystyle P^{-1}}$ should ideally be proportional (also independent of the matrix size) to the cost of multiplication of ${\displaystyle A}$ by a vector.

Examples

External links

• Preconditioned Conjugate Gradient – math-linux.com
• Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods

Spectral transformations

By analogy with linear systems, for an eigenvalue problem ${\displaystyle Ax=\lambda x}$ one may be tempted to replace the matrix ${\displaystyle A}$ with the matrix ${\displaystyle P^{-1}A}$ using a preconditioner ${\displaystyle P}$. However, this makes sense only if the seeking eigenvectors of ${\displaystyle A}$ and ${\displaystyle P^{-1}A}$ are the same. This is the case for spectral transformations.

The most popular spectral transformation is the so-called shift-and-invert transformation, where for a given scalar ${\displaystyle \alpha }$, called the shift, the original eigenvalue problem ${\displaystyle Ax=\lambda x}$ is replaced with the shift-and-invert problem ${\displaystyle (A-\alpha I)^{-1}x=\mu x}$. The eigenvectors are preserved, and one can solve the shift-and-invert problem by an iterative solver, e.g., the power iteration. This gives the Inverse iteration, which normally converges to the eigenvector, corresponding to the eigenvalue closest to the shift ${\displaystyle \alpha }$. The Rayleigh quotient iteration is a shift-and-invert method with a variable shift.

Spectral transformations are specific for eigenvalue problems and have no analogs for linear systems. They require accurate numerical calculation of the transformation involved, which becomes the main bottleneck for large problems.

General preconditioning

To make a close connection to linear systems, let us suppose that the targeted eigenvalue ${\displaystyle \lambda _{\star }}$ is known (approximately). Then one can compute the corresponding eigenvector from the homogeneous linear system ${\displaystyle (A-\lambda _{\star }I)x=0}$. Using the concept of left preconditioning for linear systems, we obtain ${\displaystyle T(A-\lambda _{\star }I)x=0}$, where ${\displaystyle T}$ is the preconditioner, which we can try to solve using the Richardson iteration

$${\displaystyle \mathbf {x} _{n+1}=\mathbf {x} _{n}-\gamma _{n}T(A-\lambda _{\star }I)\mathbf {x} _{n},\ n\geq 0.}$$

Practical preconditioning

Let us first replace the theoretical value ${\displaystyle \lambda _{\star }}$ in the Richardson iteration above with its current approximation ${\displaystyle \lambda _{n}}$ to obtain a practical algorithm

$${\displaystyle \mathbf {x} _{n+1}=\mathbf {x} _{n}-\gamma _{n}T(A-\lambda _{n}I)\mathbf {x} _{n},\ n\geq 0.}$$

A popular choice is ${\displaystyle \lambda _{n}=\rho (x_{n})}$ using the Rayleigh quotient function ${\displaystyle \rho (\cdot )}$. Practical preconditioning may be as trivial as just using ${\displaystyle T=(\operatorname {diag} (A))^{-1}}$ or ${\displaystyle T=(\operatorname {diag} (A-\lambda _{n}I))^{-1}.}$ For some classes of eigenvalue problems the efficiency of ${\displaystyle T\approx A^{-1}}$ has been demonstrated, both numerically and theoretically. The choice ${\displaystyle T\approx A^{-1}}$ allows one to easily utilize for eigenvalue problems the vast variety of preconditioners developed for linear systems.

Due to the changing value ${\displaystyle \lambda _{n}}$, a comprehensive theoretical convergence analysis is much more difficult, compared to the linear systems case, even for the simplest methods, such as the Richardson iteration.

External links

• Templates for the Solution of Algebraic Eigenvalue Problems: a Practical Guide

Description

For example, to find a local minimum of a real-valued function ${\displaystyle F(\mathbf {x} )}$ using gradient descent, one takes steps proportional to the negative of the gradient ${\displaystyle -\nabla F(\mathbf {a} )}$ (or of the approximate gradient) of the function at the current point:

$${\displaystyle \mathbf {x} _{n+1}=\mathbf {x} _{n}-\gamma _{n}\nabla F(\mathbf {x} _{n}),\ n\geq 0.}$$

The preconditioner is applied to the gradient:

$${\displaystyle \mathbf {x} _{n+1}=\mathbf {x} _{n}-\gamma _{n}P^{-1}\nabla F(\mathbf {x} _{n}),\ n\geq 0.}$$

Preconditioning here can be viewed as changing the geometry of the vector space with the goal to make the level sets look like circles.^[5] In this case the preconditioned gradient aims closer to the point of the extrema as on the figure, which speeds up the convergence.

Connection to linear systems

The minimum of a quadratic function

$${\displaystyle F(\mathbf {x} )={\tfrac {1}{2}}\mathbf {x} ^{T}A\mathbf {x} -\mathbf {x} ^{T}\mathbf {b} ,}$$

where ${\displaystyle \mathbf {x} }$ and ${\displaystyle \mathbf {b} }$ are real column-vectors and ${\displaystyle A}$ is a real symmetric positive-definite matrix, is exactly the solution of the linear equation ${\displaystyle A\mathbf {x} =\mathbf {b} }$. Since ${\displaystyle \nabla F(\mathbf {x} )=A\mathbf {x} -\mathbf {b} }$, the preconditioned gradient descent method of minimizing ${\displaystyle F(\mathbf {x} )}$ is

$${\displaystyle \mathbf {x} _{n+1}=\mathbf {x} _{n}-\gamma _{n}P^{-1}(A\mathbf {x} _{n}-\mathbf {b} ),\ n\geq 0.}$$

This is the preconditioned Richardson iteration for solving a system of linear equations.

Connection to eigenvalue problems

The minimum of the Rayleigh quotient

$${\displaystyle \rho (\mathbf {x} )={\frac {\mathbf {x} ^{T}A\mathbf {x} }{\mathbf {x} ^{T}\mathbf {x} }},}$$

where ${\displaystyle \mathbf {x} }$ is a real non-zero column-vector and ${\displaystyle A}$ is a real symmetric positive-definite matrix, is the smallest eigenvalue of ${\displaystyle A}$, while the minimizer is the corresponding eigenvector. Since ${\displaystyle \nabla \rho (\mathbf {x} )}$ is proportional to ${\displaystyle A\mathbf {x} -\rho (\mathbf {x} )\mathbf {x} }$, the preconditioned gradient descent method of minimizing ${\displaystyle \rho (\mathbf {x} )}$ is

$${\displaystyle \mathbf {x} _{n+1}=\mathbf {x} _{n}-\gamma _{n}P^{-1}(A\mathbf {x} _{n}-\rho (\mathbf {x_{n}} )\mathbf {x_{n}} ),\ n\geq 0.}$$

This is an analog of preconditioned Richardson iteration for solving eigenvalue problems.

Variable preconditioning

In many cases, it may be beneficial to change the preconditioner at some or even every step of an iterative algorithm in order to accommodate for a changing shape of the level sets, as in

$${\displaystyle \mathbf {x} _{n+1}=\mathbf {x} _{n}-\gamma _{n}P_{n}^{-1}\nabla F(\mathbf {x} _{n}),\ n\geq 0.}$$

One should have in mind, however, that constructing an efficient preconditioner is very often computationally expensive. The increased cost of updating the preconditioner can easily override the positive effect of faster convergence. If ${\displaystyle P_{n}^{-1}=H_{n}}$, a BFGS approximation of the inverse hessian matrix, this method is referred to as a Quasi-Newton method.