R24 Q5a

Answer — step by step

A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.

Diagram: The shortcut adds the input to the learned residual before the next stage. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W_s x for projection. | Add matching shapes; do not concatenate.

LaTeX: \frac{\partial L}{\partial x}=\frac{\partial L}{\partial y}\left(\frac{\partial F}{\partial x}+I\right)

The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.

Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].

A ResNet is a suitable architecture to investigate because skip additions provide shorter gradient paths. First verify the diagnosis with gradients and training curves; slow training alone does not establish vanishing gradients. Match projection shortcuts when spatial size/channels change and compare validation performance after the redesign.