R25 Q5a · ResNet shortcut and Inception1×1

Answer — step by step

Definition. A residual network uses shortcut connections to learn a residual mapping relative to a block's input. A basic residual block adds the block transformation F(x) to a shortcut representation of x.

LaTeX: y=F(x)+x

If the desired mapping is H(x), the residual branch represents F(x) = H(x) − x. When the useful mapping is close to identity, a small residual can be easier to optimize than reconstructing identity through the entire branch.

Diagram: The two paths must have equal output shape before elementwise addition.. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W_s x for projection. | Add matching shapes; do not concatenate.

Effect on gradient propagation

For the identity-shortcut formulation, the derivative includes a direct identity contribution.

LaTeX: \frac{\partial y}{\partial x}=\frac{\partial F}{\partial x}+I

This provides a path for information and derivatives that does not require passing through every weight layer in the residual branch. It helps optimize deep networks and addresses the degradation problem. It does not imply that every possible ResNet is immune to vanishing gradients.

Projection when dimensions differ

If channels or spatial dimensions change, use a projection shortcut, often a 1 × 1 convolution with suitable stride.

LaTeX: y=F(x)+W_sx

An identity shortcut has no learned parameters. A projection does. For Cᵢ input channels and Cₒ output channels, a biased 1 × 1 projection has CᵢCₒ + Cₒ parameters. The residual branch's output must match the projection output for addition.

Question. Let x = [1, 2, 3] and F(x) = [0.1, −0.2, 0.3]. Find the residual sum.

Given

x = [1,2,3]; residual branch F(x) = [0.1,−0.2,0.3].

To find

Find the three components of x+F(x).

Formula — which one and why

Residual addition. Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested.

x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition.

LaTeX: y=x+F(x)

Yaad rakhne ka cue: Same position ke numbers add; shape same rehni chahiye.

Solution — step by step

Step 1: add the first matching pair.

LaTeX: 1+0.1=1.1

Step 2: add the second matching pair.

LaTeX: 2+(-0.2)=1.8

Step 3: add the third matching pair.

LaTeX: 3+0.3=3.3

Step 4: assemble in the original order.

LaTeX: y=[1.1,1.8,3.3]

The vector length remains 3. Concatenating these vectors would give length 6 and would implement a different operation. Any post-addition activation specified in the architecture is applied after this sum.

Definition. A 1 × 1 convolution combines the input channels independently at each spatial location. A bottleneck uses fewer output channels before a larger spatial convolution, reducing its parameter and multiplication counts.

Diagram: Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size.. Labels: Compare the two paths in the worked question | Compare the two paths in the worked question | Input 28 × 28 × 192. Output 28 × 28 × 32. Count multiplications (MACs). | DIRECT | One 5 × 5 convolution | 28 × 28 × 32 × 25 × 192 | = 120,422,400 multiplications | REDUCE | 1 × 1: 192 → 16 | 28 × 28 × 16 × 192 | = 2,408,448 | 5 × 5: 16 → 32 | 28 × 28 × 32 × 25 × 16 | = 10,035,200 | Total with reduction = 12,443,648 MACs | Reducing depth to 16 before the 5 × 5 filter saves about 89.67% of these | multiplications.

Question. Input is 28 × 28 × 192. Compare a direct same-padded 5 × 5 convolution with 32 filters against a 1 × 1 reduction to 16 channels followed by the same 5 × 5 convolution. Count MACs and ignore bias additions.

Given

Both paths produce 28 × 28 × 32. The direct layer receives 192 channels; the bottleneck's spatial layer receives 16 channels. Stride is 1 and padding preserves spatial size.

To find

Direct MACs, total two-stage bottleneck MACs and percentage saved.

Formula — which one and why

Each output scalar uses one product per input-channel kernel entry. Multiply that cost by the number of output positions and channels. The two bottleneck layers require separate counts, which are added.

LaTeX: \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}}

One MAC counts one multiply–accumulate pair. A convention counting multiplication and addition separately gives approximately twice as many FLOPs.

Solution — step by step

Step 1: direct layer. Each output uses 5 × 5 × 192 = 4,800 products. There are 28 × 28 × 32 = 25,088 outputs.

LaTeX: \mathrm{MACs}_{\mathrm{direct}}=25088(4800)=120422400

Step 2: 1 × 1 reduction. There are 16 receiving filters, each spanning all 192 original channels.

LaTeX: \mathrm{MACs}_{1\times1}=28(28)(16)(192)=2408448

Step 3: reduced-depth spatial layer. Its 32 filters now span 16 channels.

LaTeX: \mathrm{MACs}_{5\times5}=28(28)(32)(25)(16)=10035200

Step 4: add the two bottleneck costs.

LaTeX: \mathrm{MACs}_{\mathrm{bottleneck}}=2408448+10035200=12443648

Step 5: calculate the saving.

LaTeX: \mathrm{saving}=120422400-12443648=107978752

LaTeX: \mathrm{saving\ percentage}=100\frac{107978752}{120422400}\approx89.666667\%

Final answer: direct = 120,422,400 MACs; bottleneck = 12,443,648 MACs, saving about 89.67%. Spatial size is unchanged; the saving comes from reducing the channels entering the expensive 5 × 5 layer. If the reduction width is not supplied, use a symbol for it and label any numerical choice as an assumption.

A second bottleneck with different dimensions

Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with a 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.

Given

Spatial output = 8 × 8; original channels = 16; final filters = 8; bottleneck channels = 4; stride = 1.

To find

Direct MACs, bottleneck MACs and saving.

Formula — which one and why

Use the same MAC formula above once for the direct path and once for each bottleneck stage. Every branch must include all of its layers.

Solution — step by step

Step 1: direct spatial convolution.

LaTeX: 8(8)(8)(3)(3)(16)=73728

Step 2: 1 × 1 reduction.

LaTeX: 8(8)(4)(1)(1)(16)=4096

Step 3: reduced-depth 3 × 3 convolution.

LaTeX: 8(8)(8)(3)(3)(4)=18432

Step 4: add the stages and subtract from the direct cost.

LaTeX: 4096+18432=22528

LaTeX: 73728-22528=51200

Final answer: direct = 73,728 MACs; bottleneck = 22,528 MACs; saving = 51,200 MACs, about 69.44%. Both paths produce 8 × 8 × 8, but they are different learned transformations and need not produce identical feature values.