Neuron

A neuron with label ${\displaystyle j}$ receiving an input ${\displaystyle p_{j}(t)}$ from predecessor neurons consists of the following components:^[1]

• an activation ${\displaystyle a_{j}(t)}$, the neuron's state, depending on a discrete time parameter,
• an optional threshold ${\displaystyle \theta _{j}}$, which stays fixed unless changed by learning,
• an activation function ${\displaystyle f}$ that computes the new activation at a given time ${\displaystyle t+1}$ from ${\displaystyle a_{j}(t)}$, ${\displaystyle \theta _{j}}$ and the net input ${\displaystyle p_{j}(t)}$ giving rise to the relation

${\displaystyle a_{j}(t+1)=f(a_{j}(t),p_{j}(t),\theta _{j}),}$

• and an output function ${\displaystyle f_{\text{out}}}$ computing the output from the activation

${\displaystyle o_{j}(t)=f_{\text{out}}(a_{j}(t)).}$

Often the output function is simply the identity function.

An input neuron has no predecessor but serves as input interface for the whole network. Similarly an output neuron has no successor and thus serves as output interface of the whole network.

Propagation function

The propagation function computes the input ${\displaystyle p_{j}(t)}$ to the neuron ${\displaystyle j}$ from the outputs ${\displaystyle o_{i}(t)}$and typically has the form^[1]

${\displaystyle p_{j}(t)=\sum _{i}o_{i}(t)w_{ij}.}$

Bias

A bias term can be added, changing the form to the following:^[2]

${\displaystyle p_{j}(t)=\sum _{i}o_{i}(t)w_{ij}+w_{0j},}$ where ${\displaystyle w_{0j}}$ is a bias.

Algorithm

Let ${\displaystyle N}$ be a network with ${\displaystyle e}$ connections, ${\displaystyle m}$ inputs and ${\displaystyle n}$ outputs.

Below, ${\displaystyle x_{1},x_{2},\dots }$ denote vectors in ${\displaystyle \mathbb {R} ^{m}}$, ${\displaystyle y_{1},y_{2},\dots }$ vectors in ${\displaystyle \mathbb {R} ^{n}}$, and ${\displaystyle w_{0},w_{1},w_{2},\ldots }$ vectors in ${\displaystyle \mathbb {R} ^{e}}$. These are called inputs, outputs and weights, respectively.

The network corresponds to a function ${\displaystyle y=f_{N}(w,x)}$ which, given a weight ${\displaystyle w}$, maps an input ${\displaystyle x}$ to an output ${\displaystyle y}$.

In supervised learning, a sequence of training examples ${\displaystyle (x_{1},y_{1}),\dots ,(x_{p},y_{p})}$ produces a sequence of weights ${\displaystyle w_{0},w_{1},\dots ,w_{p}}$ starting from some initial weight ${\displaystyle w_{0}}$, usually chosen at random.

These weights are computed in turn: first compute ${\displaystyle w_{i}}$ using only ${\displaystyle (x_{i},y_{i},w_{i-1})}$ for ${\displaystyle i=1,\dots ,p}$. The output of the algorithm is then ${\displaystyle w_{p}}$, giving a new function ${\displaystyle x\mapsto f_{N}(w_{p},x)}$. The computation is the same in each step, hence only the case ${\displaystyle i=1}$ is described.

${\displaystyle w_{1}}$ is calculated from ${\displaystyle (x_{1},y_{1},w_{0})}$ by considering a variable weight ${\displaystyle w}$ and applying gradient descent to the function ${\displaystyle w\mapsto E(f_{N}(w,x_{1}),y_{1})}$ to find a local minimum, starting at ${\displaystyle w=w_{0}}$.

This makes ${\displaystyle w_{1}}$ the minimizing weight found by gradient descent.

Propagation

Propagation involves the following steps:

• Propagation forward through the network to generate the output value(s)
• Calculation of the cost (error term)
• Propagation of the output activations back through the network using the training pattern target to generate the deltas (the difference between the targeted and actual output values) of all output and hidden neurons.

Weight update

For each weight:

• Multiply the weight's output delta and input activation to find the gradient of the weight.
• Subtract the ratio (percentage) of the weight's gradient from the weight.

The learning rate is the ratio (percentage) that influences the speed and quality of learning. The greater the ratio, the faster the neuron trains, but the lower the ratio, the more accurate the training. The sign of the gradient of a weight indicates whether the error varies directly with or inversely to the weight. Therefore, the weight must be updated in the opposite direction, "descending" the gradient.

Learning is repeated (on new batches) until the network performs adequately.

Pseudocode

Pseudocode for a stochastic gradient descent algorithm for training a three-layer network (one hidden layer):

initialize network weights (often small random values)
do
    for each training example named ex do
        prediction = neural-net-output(network, ex)  // forward pass
        actual = teacher-output(ex)
        compute error (prediction - actual) at the output units
        compute ${\displaystyle \Delta w_{h}}$ for all weights from hidden layer to output layer  // backward pass
        compute ${\displaystyle \Delta w_{i}}$ for all weights from input layer to hidden layer   // backward pass continued
        update network weights // input layer not modified by error estimate
until error rate becomes acceptably low
return the network

The lines labeled "backward pass" can be implemented using the backpropagation algorithm, which calculates the gradient of the error of the network regarding the network's modifiable weights.^[5]