F23 Q6b

Answer — step by step

A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.

Diagram: The shortcut adds the input to the learned residual before the next stage. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W_s x for projection. | Add matching shapes; do not concatenate.

LaTeX: \frac{\partial L}{\partial x}=\frac{\partial L}{\partial y}\left(\frac{\partial F}{\partial x}+I\right)

The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.

Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].