Search gradients

Let ${\displaystyle \theta }$ denote the parameters of the search distribution ${\displaystyle \pi (x\,|\,\theta )}$ and ${\displaystyle f(x)}$ the fitness function evaluated at ${\displaystyle x}$. NES then pursues the objective of maximizing the expected fitness under the search distribution

${\displaystyle J(\theta )=\operatorname {E} _{\theta }[f(x)]=\int f(x)\;\pi (x\,|\,\theta )\;dx}$

through gradient ascent. The gradient can be rewritten as

${\displaystyle \nabla _{\theta }J(\theta )=\nabla _{\theta }\int f(x)\;\pi (x\,|\,\theta )\;dx}$

${\displaystyle =\int f(x)\;\nabla _{\theta }\pi (x\,|\,\theta )\;dx}$

${\displaystyle =\int f(x)\;\nabla _{\theta }\pi (x\,|\,\theta )\;{\frac {\pi (x\,|\,\theta )}{\pi (x\,|\,\theta )}}\;dx}$

${\displaystyle =\int {\Big [}f(x)\;\nabla _{\theta }\log \pi (x\,|\,\theta ){\Big ]}\;\pi (x\,|\,\theta )\;dx}$

${\displaystyle =\operatorname {E} _{\theta }\left[f(x)\;\nabla _{\theta }\log \pi (x\,|\,\theta )\right]}$

that is, the expected value of ${\displaystyle f(x)}$ times the log-derivatives at ${\displaystyle x}$. In practice, it is possible to use the Monte Carlo approximation based on a finite number of ${\displaystyle \lambda }$ samples

${\displaystyle \nabla _{\theta }J(\theta )\approx {\frac {1}{\lambda }}\sum _{k=1}^{\lambda }f(x_{k})\;\nabla _{\theta }\log \pi (x_{k}\,|\,\theta )}$.

Finally, the parameters of the search distribution can be updated iteratively

${\displaystyle \theta \leftarrow \theta +\eta \nabla _{\theta }J(\theta )}$

Natural gradient ascent

Instead of using the plain stochastic gradient for updates, NES follows the natural gradient, which has been shown to possess numerous advantages over the plain (vanilla) gradient, e.g.:

• the gradient direction is independent of the parameterization of the search distribution
• the updates magnitudes are automatically adjusted based on uncertainty, in turn speeding convergence on plateaus and ridges.

The NES update is therefore

${\displaystyle \theta \leftarrow \theta +\eta \mathbf {F} ^{-1}\nabla _{\theta }J(\theta )}$,

where ${\displaystyle \mathbf {F} }$ is the Fisher information matrix. The Fisher matrix can sometimes be computed exactly, otherwise it is estimated from samples, reusing the log-derivatives ${\displaystyle \nabla _{\theta }\log \pi (x|\theta )}$.

Fitness shaping

NES utilizes rank-based fitness shaping in order to render the algorithm more robust, and invariant under monotonically increasing transformations of the fitness function. For this purpose, the fitness of the population is transformed into a set of utility values ${\displaystyle u_{1}\geq \dots \geq u_{\lambda }}$. Let ${\displaystyle x_{i}}$ denote the i^th best individual. Replacing fitness with utility, the gradient estimate becomes

${\displaystyle \nabla _{\theta }J(\theta )=\sum _{k=1}^{\lambda }u_{k}\;\nabla _{\theta }\log \pi (x_{k}\,|\,\theta )}$.

The choice of utility function is a free parameter of the algorithm.

Pseudocode

input: ${\displaystyle f,\;\;\theta _{init}}$

1  repeat
   
2     for  ${\displaystyle k=1\ldots \lambda }$ do                                              // λ is the population size
       
3         draw sample ${\displaystyle x_{k}\sim \pi (\cdot |\theta )}$
       
4         evaluate fitness ${\displaystyle f(x_{k})}$
       
5         calculate log-derivatives ${\displaystyle \nabla _{\theta }\log \pi (x_{k}|\theta )}$
       
6     end
   
7     assign the utilities ${\displaystyle u_{k}}$                                          // based on rank
   
8     estimate the gradient ${\displaystyle \nabla _{\theta }J\leftarrow {\frac {1}{\lambda }}\sum _{k=1}^{\lambda }u_{k}\cdot \nabla _{\theta }\log \pi (x_{k}|\theta )}$
   
9     estimate ${\displaystyle \mathbf {F} \leftarrow {\frac {1}{\lambda }}\sum _{k=1}^{\lambda }\nabla _{\theta }\log \pi (x_{k}|\theta )\nabla _{\theta }\log \pi (x_{k}|\theta )^{\top }}$           // or compute it exactly 
   
10    update parameters ${\displaystyle \theta \leftarrow \theta +\eta \cdot \mathbf {F} ^{-1}\nabla _{\theta }J}$                        // η is the learning rate

11 until stopping criterion is met