Deep Learning · TEE · section

Why one by one convolution saves work

CNN architectures and transfer learning

Definition. A 1 × 1 convolution combines the input channels independently at each spatial location. A bottleneck uses fewer output channels before a larger spatial convolution, reducing its parameter and multiplication counts.

Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size. Tap the diagram to enlarge.

Question. Input is 28 × 28 × 192. Compare a direct same-padded 5 × 5 convolution with 32 filters against a 1 × 1 reduction to 16 channels followed by the same 5 × 5 convolution. Count MACs and ignore bias additions.

Given

Both paths produce 28 × 28 × 32. The direct layer receives 192 channels; the bottleneck's spatial layer receives 16 channels. Stride is 1 and padding preserves spatial size.

To find

Direct MACs, total two-stage bottleneck MACs and percentage saved.

Formula — which one and why

Each output scalar uses one product per input-channel kernel entry. Multiply that cost by the number of output positions and channels. The two bottleneck layers require separate counts, which are added.

One MAC counts one multiply–accumulate pair. A convention counting multiplication and addition separately gives approximately twice as many FLOPs.

Solution — step by step

Step 1: direct layer. Each output uses 5 × 5 × 192 = 4,800 products. There are 28 × 28 × 32 = 25,088 outputs.

$\mathrm{MACs}_{\mathrm{direct}}=25088(4800)=120422400$ LaTeX: \mathrm{MACs}_{\mathrm{direct}}=25088(4800)=120422400

Step 2: 1 × 1 reduction. There are 16 receiving filters, each spanning all 192 original channels.

$\mathrm{MACs}_{1\times1}=28(28)(16)(192)=2408448$ LaTeX: \mathrm{MACs}_{1\times1}=28(28)(16)(192)=2408448

Step 3: reduced-depth spatial layer. Its 32 filters now span 16 channels.

$\mathrm{MACs}_{5\times5}=28(28)(32)(25)(16)=10035200$ LaTeX: \mathrm{MACs}_{5\times5}=28(28)(32)(25)(16)=10035200

Step 4: add the two bottleneck costs.

$\mathrm{MACs}_{\mathrm{bottleneck}}=2408448+10035200=12443648$ LaTeX: \mathrm{MACs}_{\mathrm{bottleneck}}=2408448+10035200=12443648

Step 5: calculate the saving.

$\mathrm{saving}=120422400-12443648=107978752$ LaTeX: \mathrm{saving}=120422400-12443648=107978752

$\mathrm{saving\ percentage}=100\frac{107978752}{120422400}\approx89.666667\%$ LaTeX: \mathrm{saving\ percentage}=100\frac{107978752}{120422400}\approx89.666667\%

Final answer: direct = 120,422,400 MACs; bottleneck = 12,443,648 MACs, saving about 89.67%. Spatial size is unchanged; the saving comes from reducing the channels entering the expensive 5 × 5 layer. If the reduction width is not supplied, use a symbol for it and label any numerical choice as an assumption.

A second bottleneck with different dimensions

Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with a 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.

Given

Spatial output = 8 × 8; original channels = 16; final filters = 8; bottleneck channels = 4; stride = 1.

To find

Direct MACs, bottleneck MACs and saving.

Formula — which one and why

Use the same MAC formula above once for the direct path and once for each bottleneck stage. Every branch must include all of its layers.

Solution — step by step

Step 1: direct spatial convolution.

$8(8)(8)(3)(3)(16)=73728$ LaTeX: 8(8)(8)(3)(3)(16)=73728

Step 2: 1 × 1 reduction.

$8(8)(4)(1)(1)(16)=4096$ LaTeX: 8(8)(4)(1)(1)(16)=4096

Step 3: reduced-depth 3 × 3 convolution.

$8(8)(8)(3)(3)(4)=18432$ LaTeX: 8(8)(8)(3)(3)(4)=18432

Step 4: add the stages and subtract from the direct cost.

$4096+18432=22528$ LaTeX: 4096+18432=22528

$73728-22528=51200$ LaTeX: 73728-22528=51200

Final answer: direct = 73,728 MACs; bottleneck = 22,528 MACs; saving = 51,200 MACs, about 69.44%. Both paths produce 8 × 8 × 8, but they are different learned transformations and need not produce identical feature values.