Recurrent network model

A generic recurrent network has hidden states ${\displaystyle h_{1},h_{2},...}$ inputs ${\displaystyle u_{1},u_{2},...}$, and outputs ${\displaystyle x_{1},x_{2},...}$. Let it be parametrized by ${\displaystyle \theta }$, so that the system evolves as

$${\displaystyle (h_{t},x_{t})=F(h_{t-1},u_{t},\theta )}$$

Often, the output ${\displaystyle x_{t}}$ is a function of ${\displaystyle h_{t}}$, as some ${\displaystyle x_{t}=G(h_{t})}$. The vanishing gradient problem already presents itself clearly when ${\displaystyle x_{t}=h_{t}}$, so we simplify our notation to the special case with:

$${\displaystyle x_{t}=F(x_{t-1},u_{t},\theta )}$$

Now, take its differential:

$${\displaystyle {\begin{aligned}dx_{t}&=\nabla _{\theta }F(x_{t-1},u_{t},\theta )d\theta +\nabla _{x}F(x_{t-1},u_{t},\theta )dx_{t-1}\\&=\nabla _{\theta }F(x_{t-1},u_{t},\theta )d\theta +\nabla _{x}F(x_{t-1},u_{t},\theta )(\nabla _{\theta }F(x_{t-2},u_{t-1},\theta )d\theta +\nabla _{x}F(x_{t-2},u_{t-1},\theta )dx_{t-2})\\&=\cdots \\&=\left(\nabla _{\theta }F(x_{t-1},u_{t},\theta )+\nabla _{x}F(x_{t-1},u_{t},\theta )\nabla _{\theta }F(x_{t-2},u_{t-1},\theta )+\cdots \right)d\theta \end{aligned}}}$$

Training the network requires us to define a loss function to be minimized. Let it be ${\displaystyle L(x_{T},u_{1},...,u_{T})}$^{[note 1]}, then minimizing it by gradient descent gives

${\displaystyle dL=\nabla _{x}L(x_{T},u_{1},...,u_{T})\left(\nabla _{\theta }F(x_{t-1},u_{t},\theta )+\nabla _{x}F(x_{t-1},u_{t},\theta )\nabla _{\theta }F(x_{t-2},u_{t-1},\theta )+\cdots \right)d\theta }$

(loss differential)

$${\displaystyle \Delta \theta =-\eta \cdot \left[\nabla _{x}L(x_{T})\left(\nabla _{\theta }F(x_{t-1},u_{t},\theta )+\nabla _{x}F(x_{t-1},u_{t},\theta )\nabla _{\theta }F(x_{t-2},u_{t-1},\theta )+\cdots \right)\right]^{T}}$$

where ${\displaystyle \eta }$ is the learning rate. The vanishing/exploding gradient problem appears because there are repeated multiplications, of the form

$${\displaystyle \nabla _{x}F(x_{t-1},u_{t},\theta )\nabla _{x}F(x_{t-2},u_{t-1},\theta )\nabla _{x}F(x_{t-3},u_{t-2},\theta )\cdots }$$

Example: recurrent network with sigmoid activation

For a concrete example, consider a typical recurrent network defined by

$${\displaystyle x_{t}=F(x_{t-1},u_{t},\theta )=W_{rec}\sigma (x_{t-1})+W_{in}u_{t}+b}$$

where ${\displaystyle \theta =(W_{rec},W_{in})}$ is the network parameter, ${\displaystyle \sigma }$ is the sigmoid activation function^{[note 2]}, applied to each vector coordinate separately, and ${\displaystyle b}$ is the bias vector. Then, ${\displaystyle \nabla _{x}F(x_{t-1},u_{t},\theta )=W_{rec}\mathop {diag} (\sigma '(x_{t-1}))}$, and so

$${\displaystyle {\begin{aligned}\nabla _{x}F(x_{t-1},u_{t},\theta )&\nabla _{x}F(x_{t-2},u_{t-1},\theta )\cdots \nabla _{x}F(x_{t-k},u_{t-k+1},\theta )\\=W_{rec}\mathop {diag} (\sigma '(x_{t-1}))&W_{rec}\mathop {diag} (\sigma '(x_{t-2}))\cdots W_{rec}\mathop {diag} (\sigma '(x_{t-k}))\end{aligned}}}$$

Since ${\displaystyle |\sigma '|\leq 1}$, the operator norm of the above multiplication is bounded above by ${\displaystyle \|W_{rec}\|^{k}}$. So if the spectral radius of ${\displaystyle W_{rec}}$ is ${\displaystyle \gamma <1}$, then at large ${\displaystyle k}$, the above multiplication has operator norm bounded above by ${\displaystyle \gamma ^{k}\to 0}$. This is the prototypical vanishing gradient problem. The effect of a vanishing gradient is that the network cannot learn long-range effects. Recall Equation (loss differential):

$${\displaystyle \nabla _{\theta }L=\nabla _{x}L(x_{T},u_{1},...,u_{T})\left(\nabla _{\theta }F(x_{t-1},u_{t},\theta )+\nabla _{x}F(x_{t-1},u_{t},\theta )\nabla _{\theta }F(x_{t-2},u_{t-1},\theta )+\cdots \right)}$$

The components of ${\displaystyle \nabla _{\theta }F(x,u,\theta )}$ are just components of ${\displaystyle \sigma (x)}$ and ${\displaystyle u}$, so if ${\displaystyle u_{t},u_{t-1},...}$ are bounded, then ${\displaystyle \|\nabla _{\theta }F(x_{t-k-1},u_{t-k},\theta )\|}$ is also bounded by some ${\displaystyle M>0}$, and so the terms in ${\displaystyle \nabla _{\theta }L}$ decay as ${\displaystyle M\gamma ^{k}}$. This means that, effectively, ${\displaystyle \nabla _{\theta }L}$ is affected only by the first ${\displaystyle O(\gamma ^{-1})}$ terms in the sum.

If ${\displaystyle \gamma \geq 1}$, the above analysis does not quite work.^{[note 3]} For the prototypical exploding gradient problem, the next model is clearer.

Dynamical systems model

Bifurcation diagram of the one-neuron recurrent network. Horizontal axis is b, and vertical axis is x. The black curve is the set of stable and unstable equilibria. Notice that the system exhibits hysteresis, and can be used as a one-bit memory.

Following (Doya, 1993),^[6] consider this one-neuron recurrent network with sigmoid activation:

$${\displaystyle x_{t+1}=(1-\epsilon )x_{t}+\epsilon \sigma (wx_{t}+b)+\epsilon w'u_{t}}$$

At the small ${\displaystyle \epsilon }$ limit, the dynamics of the network becomes

$${\displaystyle {\frac {dx}{dt}}=-x(t)+\sigma (wx(t)+b)+w'u(t)}$$

Consider first the autonomous case, with ${\displaystyle u=0}$. Set ${\displaystyle w=5.0}$, and vary ${\displaystyle b}$ in ${\displaystyle [-3,-2]}$. As ${\displaystyle b}$ decreases, the system has 1 stable point, then has 2 stable points and 1 unstable point, and finally has 1 stable point again. Explicitly, the stable points are ${\displaystyle (x,b)=\left(x,\ln \left({\frac {x}{1-x}}\right)-5x\right)}$.

Now consider ${\displaystyle {\frac {\Delta x(T)}{\Delta x(0)}}}$ and ${\displaystyle {\frac {\Delta x(T)}{\Delta b}}}$, where ${\displaystyle T}$ is large enough that the system has settled into one of the stable points.

If ${\displaystyle (x(0),b)}$ puts the system very close to an unstable point, then a tiny variation in ${\displaystyle x(0)}$ or ${\displaystyle b}$ would make ${\displaystyle x(T)}$ move from one stable point to the other. This makes ${\displaystyle {\frac {\Delta x(T)}{\Delta x(0)}}}$ and ${\displaystyle {\frac {\Delta x(T)}{\Delta b}}}$ both very large, a case of the exploding gradient.

If ${\displaystyle (x(0),b)}$ puts the system far from an unstable point, then a small variation in ${\displaystyle x(0)}$ would have no effect on ${\displaystyle x(T)}$, making ${\displaystyle {\frac {\Delta x(T)}{\Delta x(0)}}=0}$, a case of the vanishing gradient.

Note that in this case, ${\displaystyle {\frac {\Delta x(T)}{\Delta b}}\approx {\frac {\partial x(T)}{\partial b}}=\left({\frac {1}{x(T)(1-x(T))}}-5\right)^{-1}}$ neither decays to zero nor blows up to infinity. Indeed, it's the only well-behaved gradient, which explains why early researches focused on learning or designing recurrent networks systems that could perform long-ranged computations (such as outputting the first input it sees at the very end of an episode) by shaping its stable attractors.^[7]

For the general case, the intuition still holds (^[5] Figures 3, 4, and 5).

Geometric model

Continue using the above one-neuron network, fixing ${\displaystyle w=5,x(0)=0.5,u(t)=0}$, and consider a loss function defined by ${\displaystyle L(x(T))=(0.855-x(T))^{2}}$. This produces a rather pathological loss landscape: as ${\displaystyle b}$ approach ${\displaystyle -2.5}$ from above, the loss approaches zero, but as soon as ${\displaystyle b}$ crosses ${\displaystyle -2.5}$, the attractor basin changes, and loss jumps to 0.50.^{[note 4]}

Consequently, attempting to train ${\displaystyle b}$ by gradient descent would "hit a wall in the loss landscape", and cause exploding gradient. A slightly more complex situation is plotted in,^[5] Figures 6.

Batch normalization

Batch normalization is a standard method for solving both the exploding and the vanishing gradient problems.^[8][9]

Multi-level hierarchy

One is Jürgen Schmidhuber's multi-level hierarchy of networks (1992) pre-trained one level at a time through unsupervised learning, fine-tuned through backpropagation.^[10] Here each level learns a compressed representation of the observations that is fed to the next level.

Long short-term memory

Another technique particularly used for recurrent neural networks is the long short-term memory (LSTM) network of 1997 by Hochreiter & Schmidhuber.^[13] In 2009, deep multidimensional LSTM networks demonstrated the power of deep learning with many nonlinear layers, by winning three ICDAR 2009 competitions in connected handwriting recognition, without any prior knowledge about the three different languages to be learned.^[14][15]

Faster hardware

Hardware advances have meant that from 1991 to 2015, computer power (especially as delivered by GPUs) has increased around a million-fold, making standard backpropagation feasible for networks several layers deeper than when the vanishing gradient problem was recognized. Schmidhuber notes that this "is basically what is winning many of the image recognition competitions now", but that it "does not really overcome the problem in a fundamental way"^[16] since the original models tackling the vanishing gradient problem by Hinton and others were trained in a Xeon processor, not GPUs.^[11]

Residual networks

One of the newest and most effective ways to resolve the vanishing gradient problem is with residual neural networks,^[17] or ResNets (not to be confused with recurrent neural networks). ResNets refer to neural networks where skip connections or residual connections are part of the network architecture. These skip connections allow gradient information to pass through the layers, by creating "highways" of information, where the output of a previous layer/activation is added to the output of a deeper layer. This allows information from the earlier parts of the network to be passed to the deeper parts of the network, helping maintain signal propagation even in deeper networks. Skip connections are a critical component of what allowed successful training of deeper neural networks.

ResNets yielded lower training error (and test error) than their shallower counterparts simply by reintroducing outputs from shallower layers in the network to compensate for the vanishing data.^[17] Note that ResNets are an ensemble of relatively shallow nets and do not resolve the vanishing gradient problem by preserving gradient flow throughout the entire depth of the network – rather, they avoid the problem simply by constructing ensembles of many short networks together. (Ensemble by Construction^[18])

Other activation functions

Rectifiers such as ReLU suffer less from the vanishing gradient problem, because they only saturate in one direction.^[19]

Weight initialization

Weight initialization is another approach that has been proposed to reduce the vanishing gradient problem in deep networks.

Kumar suggested that the distribution of initial weights should vary according to activation function used and proposed to initialize the weights in networks with the logistic activation function using a Gaussian distribution with a zero mean and a standard deviation of 3.6/sqrt(N), where N is the number of neurons in a layer.^[20]

Recently, Yilmaz and Poli^[21] performed a theoretical analysis on how gradients are affected by the mean of the initial weights in deep neural networks using the logistic activation function and found that gradients do not vanish if the mean of the initial weights is set according to the formula: max(−1,-8/N). This simple strategy allows networks with 10 or 15 hidden layers to be trained very efficiently and effectively using the standard backpropagation.

Other

Behnke relied only on the sign of the gradient (Rprop) when training his Neural Abstraction Pyramid^[22] to solve problems like image reconstruction and face localization.

Neural networks can also be optimized by using a universal search algorithm on the space of neural network's weights, e.g., random guess or more systematically genetic algorithm. This approach is not based on gradient and avoids the vanishing gradient problem.^[23]