Deep Learning · TEE · section

Residual-block order and architecture qualifications

CNN architectures and transfer learning

The original post-activation basic block has this sequence: convolution → BN → ReLU → convolution → BN → add shortcut → ReLU. Thus the second branch convolution is not followed by a separate ReLU before the addition in this particular block. A pre-activation ResNet is another design; use the stated diagram.

Diagram: Add the shortcut to the residual branch, then apply the post-addition ReLU when the specified block requires it.. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W_s x for projection. | Add matching shapes; do not concatenate.

An identity shortcut has no learned parameters. A projection shortcut changes the shape using learned weights. Original ImageNet ResNets include an initial max-pooling operation; a basic residual block does not need a max-pooling layer inside it. ResNet-18/34 use two-convolution basic blocks, whereas deeper standard variants use bottleneck blocks. Avoid treating every ResNet as the same two-layer diagram.

The main advantage is easier optimization of deep networks: shortcuts provide an additional information and gradient path. This reduces the degradation problem but does not prove immunity to every vanishing/exploding-gradient case. Large networks still need compute, memory, suitable data and regularization. Reusing early features can help; it does not guarantee that every deeper model generalizes better.