Prefix sum

In the beginning of a prefix sum operation, each processing element ${\displaystyle i}$ owns a message ${\displaystyle m_{i}}$. The goal is to compute ${\displaystyle \bigoplus _{0\leq j\leq i}m_{j}}$, where ${\displaystyle \oplus }$ is an associative operation. The following pseudo code describes the algorithm.

Input: message ${\displaystyle m_{i}}$ of processor ${\displaystyle i}$.
Output: prefix sum ${\displaystyle \bigoplus _{0\leq j\leq i}m_{j}}$ of processor ${\displaystyle i}$.
${\displaystyle x:=m_{i}}$ 
${\displaystyle \sigma :=m_{i}}$
for ${\displaystyle 0\leq k\leq d-1}$ do
    ${\displaystyle y:=i{\text{ XOR }}2^{k}}$
    Send ${\displaystyle \sigma }$ to ${\displaystyle y}$
    Receive ${\displaystyle m}$ from ${\displaystyle y}$
    ${\displaystyle \sigma :=\sigma \oplus m}$
    if bit ${\displaystyle k}$ in ${\displaystyle i}$ is set then ${\displaystyle x:=x\oplus m}$
endfor

The algorithm works as follows. Observe that hypercubes of dimension ${\displaystyle d}$ can be split into two hypercubes of dimension ${\displaystyle d-1}$. Refer to the sub cube containing nodes with a leading 0 as the 0-sub cube and the sub cube consisting of nodes with a leading 1 as 1-sub cube. Once both sub cubes have calculated the prefix sum, the sum over all elements in the 0-sub cube has to be added to the every element in the 1-sub cube, since every processing element in the 0-sub cube has a lower rank than the processing elements in the 1-sub cube. The pseudo code stores the prefix sum in variable ${\displaystyle x}$ and the sum over all nodes in a sub cube in variable ${\displaystyle \sigma }$. This makes it possible for all nodes in 1-sub cube to receive the sum over the 0-sub cube in every step.

This results in a factor of ${\displaystyle \log p}$ for ${\displaystyle T_{\text{start}}}$ and a factor of ${\displaystyle n\log p}$ for ${\displaystyle T_{\text{byte}}}$: ${\displaystyle T(n,p)=(T_{\text{start}}+nT_{\text{byte}})\log p}$.

Example for a prefix sum calculation. Upper number: tentatetive prefix sum (variable ${\displaystyle x}$). Lower number: sum over all elements in the sub cube (variable ${\displaystyle \sigma }$).

All-gather / all-reduce

All-gather operations start with each processing element having a message ${\displaystyle m_{i}}$. The goal of the operation is for each processing element to know the messages of all other processing elements, i.e. ${\displaystyle x:=m_{0}\cdot m_{1}\dots m_{p}}$ where ${\displaystyle \cdot }$ is concatenation. The operation can be implemented following the algorithm template.

Input: message ${\displaystyle x:=m_{i}}$ at processing unit ${\displaystyle i}$.
Output: all messages ${\displaystyle m_{1}\cdot m_{2}\dots m_{p}}$.
${\displaystyle x:=m_{i}}$
for ${\displaystyle 0\leq k<d}$ do
    ${\displaystyle y:=i{\text{ XOR }}2^{k}}$
    Send ${\displaystyle x}$ to ${\displaystyle y}$
    Receive ${\displaystyle x'}$ from ${\displaystyle y}$
    ${\displaystyle x:=x\cdot x'}$
endfor

With each iteration, the transferred message doubles in length. This leads to a runtime of ${\displaystyle T(n,p)\approx \sum _{j=0}^{d-1}(T_{\text{start}}+n\cdot 2^{j}T_{\text{byte}})=\log(p)T_{\text{start}}+(p-1)nT_{\text{byte}}}$.

The same principle can be applied to the All-Reduce operations, but instead of concatenating the messages, it performs a reduction operation on the two messages. So it is a Reduce operation, where all processing units know the result. Compared to a normal reduce operation followed by a broadcast, All-Reduce in hypercubes reduces the number of communication steps.

All-to-all

Here every processing element has a unique message for all other processing elements.

Input: message ${\displaystyle m_{ij}}$ at processing element ${\displaystyle i}$ to processing element ${\displaystyle j}$.
for ${\displaystyle d>k\geq 0}$ do
    Receive from processing element ${\displaystyle i{\text{ XOR }}2^{k}}$:
        all messages for my ${\displaystyle k}$-dimensional sub cube
    Send to processing element ${\displaystyle i{\text{ XOR }}2^{k}}$:
        all messages for its ${\displaystyle k}$-dimensional sub cube
endfor

With each iteration a messages comes closer to its destination by one dimension, if it hasn't arrived yet. Hence, all messages have reached their target after at most ${\displaystyle d=\log {p}}$ steps. In every step, ${\displaystyle p/2}$ messages are sent: in the first iteration, half of the messages aren't meant for the own sub cube. In every following step, the sub cube is only half the size as before, but in the previous step exactly the same number of messages arrived from another processing element.

This results in a run-time of ${\displaystyle T(n,p)\approx \log {p}(T_{\text{start}}+{\frac {p}{2}}nT_{\text{byte}})}$.

Runtime

In each step, the source node sends one of its ${\displaystyle k}$ chunks to a binomial tree. Broadcasting the chunk within the binomial tree takes ${\displaystyle d}$ steps. Thus, it takes ${\displaystyle k}$ steps to distribute all chunks and additionally ${\displaystyle d}$ steps until the last binomial tree broadcast has finished, resulting in ${\displaystyle k+d}$ steps overall. Therefore, the runtime for a message of length ${\displaystyle n}$ is ${\displaystyle T(n,p,k)=\left({\frac {n}{k}}T_{\text{byte}}+T_{\text{start}}\right)(k+d)}$. With the optimal chunk size ${\displaystyle k^{*}={\sqrt {\frac {nd\cdot T_{\text{byte}}}{T_{\text{start}}}}}}$, the optimal runtime of the algorithm is ${\displaystyle T^{*}(n,p)=n\cdot T_{\text{byte}}+\log(p)\cdot T_{\text{start}}+{\sqrt {n\log(p)\cdot T_{\text{start}}\cdot T_{\text{byte}}}}}$.

Construction of the binomial trees

A ${\displaystyle 3}$-dimensional hypercubes with three ESBT embedded.

This section describes how to construct the binomial trees systematically. First, construct a single binomial spanning tree von ${\displaystyle 2^{d}}$ nodes as follows. Number the nodes from ${\displaystyle 0}$ to ${\displaystyle 2^{d}-1}$ and consider their binary representation. Then the children of each nodes are obtained by negating single leading zeroes. This results in a single binomial spanning tree. To obtain ${\displaystyle d}$ edge-disjoint copies of the tree, translate and rotate the nodes: for the ${\displaystyle k}$-th copy of the tree, apply a XOR operation with ${\displaystyle 2^{k}}$ to each node. Subsequently, right-rotate all nodes by ${\displaystyle k}$ digits. The resulting binomial trees are edge-disjoint and therefore fulfill the requirements for the ESBT-broadcasting algorithm.