Deep Learning · TEE · section

Drill 39 Bottleneck cost with new values

Numerical practice from direct to mixed

Question

Question. A 16 × 16 × 64 map feeds 32 same-padded 3 × 3 filters. Compare direct MACs with an 8-channel 1 × 1 bottleneck first.

Answer — step by step

Given

Input = 16 × 16 × 64.

Direct branch: 32 same-padded 3 × 3 filters.

Alternative: 1 × 1 reduction to 8 channels, then the same 32-filter 3 × 3 layer.

Spatial size remains 16 × 16; exclude bias additions.

To find

Find both MAC counts, absolute saving and percentage saving.

Formula — which one and why

Multiply–accumulate count. Each output position and output channel uses K²C_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution.

H_out,W_out = output map size; C_in,C_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded.

\mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}}LaTeX: \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}}
\mathrm{saving\ percentage}=100\frac{\mathrm{direct}-\mathrm{reduced}}{\mathrm{direct}}LaTeX: \mathrm{saving\ percentage}=100\frac{\mathrm{direct}-\mathrm{reduced}}{\mathrm{direct}}

Yaad rakhne ka cue: Har output cell ka kaam × kitne output cells.

Solution — step by step

Meaning and formula

A MAC is one multiply–accumulate operation. For the convention used here, count the weight–input products over every output location. Bias additions are omitted. A 1 × 1 bottleneck reduces channels before the expensive 3 × 3 layer.

\mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}}LaTeX: \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}}

Step 1: direct 3 × 3 convolution

Each output uses 3 × 3 × 64 = 576 weight–input pairs. There are 16 × 16 × 32 = 8,192 outputs.

\mathrm{MACs}_{\mathrm{direct}}=8192\times576=4{,}718{,}592LaTeX: \mathrm{MACs}_{\mathrm{direct}}=8192\times576=4{,}718{,}592

Step 2: 1 × 1 reduction from 64 to 8 channels

\mathrm{MACs}_{1\times1}=16\times16\times8\times1\times1\times64=131{,}072LaTeX: \mathrm{MACs}_{1\times1}=16\times16\times8\times1\times1\times64=131{,}072

Step 3: 3 × 3 convolution now receives only 8 channels

\mathrm{MACs}_{3\times3}=16\times16\times32\times3\times3\times8=589{,}824LaTeX: \mathrm{MACs}_{3\times3}=16\times16\times32\times3\times3\times8=589{,}824

Step 4: add the two bottleneck stages

\mathrm{MACs}_{\mathrm{bottleneck}}=131{,}072+589{,}824=720{,}896LaTeX: \mathrm{MACs}_{\mathrm{bottleneck}}=131{,}072+589{,}824=720{,}896

Step 5: calculate the saving

\mathrm{saving}=4{,}718{,}592-720{,}896=3{,}997{,}696LaTeX: \mathrm{saving}=4{,}718{,}592-720{,}896=3{,}997{,}696
\mathrm{saving\ percent}=\frac{3{,}997{,}696}{4{,}718{,}592}\times100=84.722222\%LaTeX: \mathrm{saving\ percent}=\frac{3{,}997{,}696}{4{,}718{,}592}\times100=84.722222\%

Final answer: direct 4,718,592 MACs; bottleneck 720,896 MACs; approximately 84.72% fewer. The output shape stays 16 × 16 × 32.