Deep Learning · TEE · section

ResNet adds a shortcut

CNN architectures and transfer learning

Definition. A residual network uses shortcut connections to learn a residual mapping relative to a block's input. A basic residual block adds the block transformation F(x) to a shortcut representation of x.

$y=F(x)+x$ LaTeX: y=F(x)+x

If the desired mapping is H(x), the residual branch represents F(x) = H(x) − x. When the useful mapping is close to identity, a small residual can be easier to optimize than reconstructing identity through the entire branch.

The two paths must have equal output shape before elementwise addition. Tap the diagram to enlarge.

Effect on gradient propagation

For the identity-shortcut formulation, the derivative includes a direct identity contribution.

$\frac{\partial y}{\partial x}=\frac{\partial F}{\partial x}+I$ LaTeX: \frac{\partial y}{\partial x}=\frac{\partial F}{\partial x}+I

This provides a path for information and derivatives that does not require passing through every weight layer in the residual branch. It helps optimize deep networks and addresses the degradation problem. It does not imply that every possible ResNet is immune to vanishing gradients.

Projection when dimensions differ

If channels or spatial dimensions change, use a projection shortcut, often a 1 × 1 convolution with suitable stride.

$y=F(x)+W_sx$ LaTeX: y=F(x)+W_sx

An identity shortcut has no learned parameters. A projection does. For Cᵢ input channels and Cₒ output channels, a biased 1 × 1 projection has CᵢCₒ + Cₒ parameters. The residual branch's output must match the projection output for addition.

Question. Let x = [1, 2, 3] and F(x) = [0.1, −0.2, 0.3]. Find the residual sum.

Given

x = [1,2,3]; residual branch F(x) = [0.1,−0.2,0.3].

To find

Find the three components of x+F(x).

Formula — which one and why

Residual addition. Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested.

x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition.

$y=x+F(x)$ LaTeX: y=x+F(x)

Yaad rakhne ka cue: Same position ke numbers add; shape same rehni chahiye.

Solution — step by step

Step 1: add the first matching pair.

$1+0.1=1.1$ LaTeX: 1+0.1=1.1

Step 2: add the second matching pair.

$2+(-0.2)=1.8$ LaTeX: 2+(-0.2)=1.8

Step 3: add the third matching pair.

$3+0.3=3.3$ LaTeX: 3+0.3=3.3

Step 4: assemble in the original order.

$y=[1.1,1.8,3.3]$ LaTeX: y=[1.1,1.8,3.3]

The vector length remains 3. Concatenating these vectors would give length 6 and would implement a different operation. Any post-addition activation specified in the architecture is applied after this sum.