Inputs are initial passed by some totally linked layer, to your double-layer residual multihead attention as demonstrated in Fig. 7. Residual networks (Kaiming He, 2016), include feedforward to stop neurons from enduring exploding or vanishing gradients all through the learning approach. The fully related levels from the residual block (dashed box)