Linear speedup

An adjoint solver allows one to compute the gradient of the quantity of interest with respect to all design parameters at the cost of one additional solve. This, potentially, leads to a linear speedup: the computational cost of constructing an accurate surrogate decrease, and the resulting computational speedup ${\displaystyle s}$ scales linearly with the number ${\displaystyle d}$ of design parameters.

The reasoning behind this linear speedup is straightforward. Assume we run ${\displaystyle N}$ primal solves and ${\displaystyle N}$ adjoint solves, at a total cost of ${\displaystyle 2N}$. This results in ${\displaystyle N+dN}$ data; ${\displaystyle N}$ values for the quantity of interest and ${\displaystyle d}$ partial derivatives in each of the ${\displaystyle N}$ gradients. Now assume that each partial derivative provides as much information for our surrogate as a single primal solve. Then, the total cost of getting the same amount of information from primal solves only is ${\displaystyle N+dN}$. The speedup is the ratio of these costs: ^{[2] [3]}

${\displaystyle s={\frac {N+dN}{2N}}={\frac {1}{2}}+{\frac {1}{2}}d.}$

A linear speedup has been demonstrated for a fluid-structure interaction problem ^[2] and for a transonic airfoil.^[3]

Noise

One issue with adjoint-based gradients in CFD is that they can be particularly noisy. ^{[4] [5]} When derived in a Bayesian framework, GEK allows one to incorporate not only the gradient information, but also the uncertainty in that gradient information.^[6]

Kriging

Along the lines of, ^[7] we are interested in the output ${\displaystyle X}$ of our computer simulation, for which we assume the normal prior probability distribution:

${\displaystyle X\sim {\mathcal {N}}(\mu ,P),}$

with prior mean ${\displaystyle \mu }$ and prior covariance matrix ${\displaystyle P}$. The observations ${\displaystyle y}$ have the normal likelihood:

${\displaystyle Y\mid x\sim {\mathcal {N}}(Hx,R),}$

with ${\displaystyle H}$ the observation matrix and ${\displaystyle R}$ the observation error covariance matrix, which contains the observation uncertainties. After applying Bayes' Theorem we obtain a normally distributed posterior probability distribution, with Kriging mean:

${\displaystyle \operatorname {E} (X\mid y)=\mu +K(y-H\mu ),}$

and Kriging covariance:

${\displaystyle \operatorname {cov} (X\mid y)=(I-KH)P,}$

where we have the gain matrix:

${\displaystyle K=PH^{T}(R+HPH^{T})^{-1}.}$

In Kriging, the prior covariance matrix ${\displaystyle P}$ is generated from a covariance function. One example of a covariance function is the Gaussian covariance:

${\displaystyle P_{ij}=\sigma ^{2}\exp \left(-\sum _{k}{\frac {|\xi _{jk}-\xi _{ik}|^{2}}{2\theta _{k}^{2}}}\right),}$

where we sum over the dimensions ${\displaystyle k}$ and ${\displaystyle \xi }$ are the input parameters. The hyperparameters ${\displaystyle \mu }$, ${\displaystyle \sigma }$ and ${\displaystyle \theta }$ can be estimated from a Maximum Likelihood Estimate (MLE).^{[6] [8] [9]}

Indirect GEK

There are several ways of implementing GEK. The first method, indirect GEK, defines a small but finite stepsize ${\displaystyle h}$, and uses the gradient information to append synthetic data to the observations ${\displaystyle y}$, see for example.^[8] Indirect Kriging is sensitive to the choice of the step-size ${\displaystyle h}$ and cannot include observation uncertainties.

Direct GEK (through prior covariance matrix)

Direct GEK is a form of co-Kriging, where we add the gradient information as co-variables. This can be done by modifying the prior covariance ${\displaystyle P}$ or by modifying the observation matrix ${\displaystyle H}$; both approaches lead to the same GEK predictor. When we construct direct GEK through the prior covariance matrix, we append the partial derivatives to ${\displaystyle y}$, and modify the prior covariance matrix ${\displaystyle P}$ such that it also contains the derivatives (and second derivatives) of the covariance function, see for example ^[10] .^[6] The main advantages of direct GEK over indirect GEK are: 1) we do not have to choose a step-size, 2) we can include observation uncertainties for the gradients in ${\displaystyle R}$, and 3) it is less susceptible to poor conditioning of the gain matrix ${\displaystyle K}$. ^{[6] [8]}

Direct GEK (through observation matrix)

Another way of arriving at the same direct GEK predictor is to append the partial derivatives to the observations ${\displaystyle y}$ and include partial derivative operators in the observation matrix ${\displaystyle H}$, see for example.^[11]

Gradient-enhanced kriging for high-dimensional problems (Indirect method)

Current gradient-enhanced kriging methods do not scale well with the number of sampling points due to the rapid growth in the size of the correlation matrix, where new information is added for each sampling point in each direction of the design space. Furthermore, they do not scale well with the number of independent variables due to the increase in the number of hyperparameters that needs to be estimated. To address this issue, a new gradient-enhanced surrogate model approach that drastically reduced the number of hyperparameters through the use of the partial-least squares method that maintains accuracy is developed. In addition, this method is able to control the size of the correlation matrix by adding only relevant points defined through the information provided by the partial-least squares method. For more details, see.^[12] This approach is implemented into the Surrogate Modeling Toolbox (SMT) in Python (https://github.com/SMTorg/SMT), and it runs on Linux, macOS, and Windows. SMT is distributed under the New BSD license.

Augmented gradient-enhanced kriging (direct method)

A universal augmented framework is proposed in ^[9] to append derivatives of any order to the observations. This method can be viewed as a generalization of Direct GEK that takes into account higher-order derivatives. Also, the observations and derivatives are not required to be measured at the same location under this framework.