Deep Learning · TEE · formulas

Convolution multiply-accumulate count

A parameter count counts learned numbers. A multiply-accumulate count counts how many times input–weight products are used. One filter is reused at every output position, so computation includes output height and width even though its parameter count does not.

MAC=H_{out}W_{out}K_hK_wC_{in}C_{out}LaTeX: MAC=H_{out}W_{out}K_hK_wC_{in}C_{out}

Why one by one convolution saves work

Definition. A 1 × 1 convolution combines the input channels independently at each spatial location. A bottleneck uses fewer output channels before a larger spatial convolution, reducing its parameter and multiplication counts.

Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size. Tap the diagram to enlarge.

Question. Input is 28 × 28 × 192. Compare a direct same-padded 5 × 5 convolution with 32 filters against a 1 × 1 reduction to 16 channels followed by the same 5 × 5 convolution. Count MACs and ignore bias additions.

Given

Both paths produce 28 × 28 × 32. The direct layer receives 192 channels; the bottleneck's spatial layer receives 16 channels. Stride is 1 and padding preserves spatial size.

To find

Direct MACs, total two-stage bottleneck MACs and percentage saved.

Formula — which one and why

Each output scalar uses one product per input-channel kernel entry. Multiply that cost by the number of output positions and channels. The two bottleneck layers require separate counts, which are added.

\mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}}LaTeX: \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}}

One MAC counts one multiply–accumulate pair. A convention counting multiplication and addition separately gives approximately twice as many FLOPs.

Solution — step by step

Step 1: direct layer. Each output uses 5 × 5 × 192 = 4,800 products. There are 28 × 28 × 32 = 25,088 outputs.

\mathrm{MACs}_{\mathrm{direct}}=25088(4800)=120422400LaTeX: \mathrm{MACs}_{\mathrm{direct}}=25088(4800)=120422400

Step 2: 1 × 1 reduction. There are 16 receiving filters, each spanning all 192 original channels.

\mathrm{MACs}_{1\times1}=28(28)(16)(192)=2408448LaTeX: \mathrm{MACs}_{1\times1}=28(28)(16)(192)=2408448

Step 3: reduced-depth spatial layer. Its 32 filters now span 16 channels.

\mathrm{MACs}_{5\times5}=28(28)(32)(25)(16)=10035200LaTeX: \mathrm{MACs}_{5\times5}=28(28)(32)(25)(16)=10035200

Step 4: add the two bottleneck costs.

\mathrm{MACs}_{\mathrm{bottleneck}}=2408448+10035200=12443648LaTeX: \mathrm{MACs}_{\mathrm{bottleneck}}=2408448+10035200=12443648

Step 5: calculate the saving.

\mathrm{saving}=120422400-12443648=107978752LaTeX: \mathrm{saving}=120422400-12443648=107978752
\mathrm{saving\ percentage}=100\frac{107978752}{120422400}\approx89.666667\%LaTeX: \mathrm{saving\ percentage}=100\frac{107978752}{120422400}\approx89.666667\%

Final answer: direct = 120,422,400 MACs; bottleneck = 12,443,648 MACs, saving about 89.67%. Spatial size is unchanged; the saving comes from reducing the channels entering the expensive 5 × 5 layer. If the reduction width is not supplied, use a symbol for it and label any numerical choice as an assumption.

A second bottleneck with different dimensions

Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with a 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.

Given

Spatial output = 8 × 8; original channels = 16; final filters = 8; bottleneck channels = 4; stride = 1.

To find

Direct MACs, bottleneck MACs and saving.

Formula — which one and why

Use the same MAC formula above once for the direct path and once for each bottleneck stage. Every branch must include all of its layers.

Solution — step by step

Step 1: direct spatial convolution.

8(8)(8)(3)(3)(16)=73728LaTeX: 8(8)(8)(3)(3)(16)=73728

Step 2: 1 × 1 reduction.

8(8)(4)(1)(1)(16)=4096LaTeX: 8(8)(4)(1)(1)(16)=4096

Step 3: reduced-depth 3 × 3 convolution.

8(8)(8)(3)(3)(4)=18432LaTeX: 8(8)(8)(3)(3)(4)=18432

Step 4: add the stages and subtract from the direct cost.

4096+18432=22528LaTeX: 4096+18432=22528
73728-22528=51200LaTeX: 73728-22528=51200

Final answer: direct = 73,728 MACs; bottleneck = 22,528 MACs; saving = 51,200 MACs, about 69.44%. Both paths produce 8 × 8 × 8, but they are different learned transformations and need not produce identical feature values.

The faculty's 480-channel bottleneck numerical

Both paths start from 14 × 14 × 480 and end at 14 × 14 × 48. The bottleneck cost includes both layers. Tap the diagram to enlarge.

Question (practice, 3 marks). Compare a direct same-padded 5 × 5 convolution with 48 filters on a 14 × 14 × 480 input against a 1 × 1 reduction to 16 channels followed by the same 5 × 5, 48-filter layer. Use stride 1. Count MACs, ignore bias additions, and find the percentage reduction.

Given

Both paths produce 14 × 14 × 48. Direct spatial filters span 480 channels; reduced spatial filters span 16. The reduction has 16 output filters.

To find

MACs for the direct path, both bottleneck stages, their total and percentage saved.

Formula — which one and why

M=H_oW_oC_oK^2C_iLaTeX: M=H_oW_oC_oK^2C_i

M is the MAC count. H_oW_oC_o counts output values; K²C_i is the products used for each output. One MAC means one multiply-accumulate pair. Count both layers of the bottleneck.

\mathrm{saving}\%=100\frac{M_{\mathrm{direct}}-M_{\mathrm{bottleneck}}}{M_{\mathrm{direct}}}LaTeX: \mathrm{saving}\%=100\frac{M_{\mathrm{direct}}-M_{\mathrm{bottleneck}}}{M_{\mathrm{direct}}}

Solution — step by step

Step 1: direct output count. 14 × 14 × 48 = 9,408.

Step 2: products per direct output. 5 × 5 × 480 = 12,000.

Step 3: direct MACs.

9408(12000)=112896000LaTeX: 9408(12000)=112896000

Step 4: reduction-layer MACs. Each of 14 × 14 × 16 outputs combines 480 values.

14(14)(16)(480)=1505280LaTeX: 14(14)(16)(480)=1505280

Step 5: reduced spatial-layer MACs. Each of 9,408 outputs now combines 5 × 5 × 16 = 400 values.

9408(400)=3763200LaTeX: 9408(400)=3763200

Step 6: add both bottleneck costs.

1505280+3763200=5268480LaTeX: 1505280+3763200=5268480

Step 7: calculate the saving.

112896000-5268480=107627520LaTeX: 112896000-5268480=107627520
100\frac{107627520}{112896000}\approx95.333333\%LaTeX: 100\frac{107627520}{112896000}\approx95.333333\%

Final answer: direct = 112.896 million MACs; bottleneck = 5.26848 million MACs; saving ≈ 95.33%. The lecture's 112.9M and 5.3M are rounded versions. This arithmetic proves reduced computation; equal prediction accuracy is not guaranteed by the count alone.

Drill 39 Bottleneck cost with new values

Question

Question. A 16 × 16 × 64 map feeds 32 same-padded 3 × 3 filters. Compare direct MACs with an 8-channel 1 × 1 bottleneck first.

Answer — step by step

Given

Input = 16 × 16 × 64.

Direct branch: 32 same-padded 3 × 3 filters.

Alternative: 1 × 1 reduction to 8 channels, then the same 32-filter 3 × 3 layer.

Spatial size remains 16 × 16; exclude bias additions.

To find

Find both MAC counts, absolute saving and percentage saving.

Formula — which one and why

Multiply–accumulate count. Each output position and output channel uses K²C_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution.

H_out,W_out = output map size; C_in,C_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded.

\mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}}LaTeX: \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}}
\mathrm{saving\ percentage}=100\frac{\mathrm{direct}-\mathrm{reduced}}{\mathrm{direct}}LaTeX: \mathrm{saving\ percentage}=100\frac{\mathrm{direct}-\mathrm{reduced}}{\mathrm{direct}}

Yaad rakhne ka cue: Har output cell ka kaam × kitne output cells.

Solution — step by step

Meaning and formula

A MAC is one multiply–accumulate operation. For the convention used here, count the weight–input products over every output location. Bias additions are omitted. A 1 × 1 bottleneck reduces channels before the expensive 3 × 3 layer.

\mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}}LaTeX: \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}}

Step 1: direct 3 × 3 convolution

Each output uses 3 × 3 × 64 = 576 weight–input pairs. There are 16 × 16 × 32 = 8,192 outputs.

\mathrm{MACs}_{\mathrm{direct}}=8192\times576=4{,}718{,}592LaTeX: \mathrm{MACs}_{\mathrm{direct}}=8192\times576=4{,}718{,}592

Step 2: 1 × 1 reduction from 64 to 8 channels

\mathrm{MACs}_{1\times1}=16\times16\times8\times1\times1\times64=131{,}072LaTeX: \mathrm{MACs}_{1\times1}=16\times16\times8\times1\times1\times64=131{,}072

Step 3: 3 × 3 convolution now receives only 8 channels

\mathrm{MACs}_{3\times3}=16\times16\times32\times3\times3\times8=589{,}824LaTeX: \mathrm{MACs}_{3\times3}=16\times16\times32\times3\times3\times8=589{,}824

Step 4: add the two bottleneck stages

\mathrm{MACs}_{\mathrm{bottleneck}}=131{,}072+589{,}824=720{,}896LaTeX: \mathrm{MACs}_{\mathrm{bottleneck}}=131{,}072+589{,}824=720{,}896

Step 5: calculate the saving

\mathrm{saving}=4{,}718{,}592-720{,}896=3{,}997{,}696LaTeX: \mathrm{saving}=4{,}718{,}592-720{,}896=3{,}997{,}696
\mathrm{saving\ percent}=\frac{3{,}997{,}696}{4{,}718{,}592}\times100=84.722222\%LaTeX: \mathrm{saving\ percent}=\frac{3{,}997{,}696}{4{,}718{,}592}\times100=84.722222\%

Final answer: direct 4,718,592 MACs; bottleneck 720,896 MACs; approximately 84.72% fewer. The output shape stays 16 × 16 × 32.

Bottleneck computation on an 8 × 8 input

Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.

Given

Spatial output = 8 × 8 in both stages.

Input channels = 16; output filters = 8; main kernel = 3 × 3.

Bottleneck uses 4 filters of 1 × 1.

To find

Direct MACs, two-stage bottleneck MACs and savings.

Formula — which one and why

Each output scalar requires one multiply-accumulate per incoming kernel entry. A bottleneck reduces the input depth of the expensive spatial convolution.

One MAC is one multiply-accumulate pair. If counting multiplication and addition separately, the operation count is approximately twice the MAC count.

\mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K_hK_wC_{\mathrm{in}}LaTeX: \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K_hK_wC_{\mathrm{in}}

Yaad rakhne ka cue: Spatial kernel se pehle channels kam; dono stages ka cost add.

Solution — step by step

Step 1: count output positions per channel.

8(8)=64LaTeX: 8(8)=64

Step 2: count direct work per output scalar.

3(3)(16)=144LaTeX: 3(3)(16)=144

Step 3: count the full direct layer.

64(8)(144)=73728LaTeX: 64(8)(144)=73728

Step 4: count the 1 × 1 reduction.

64(4)(1)(1)(16)=4096LaTeX: 64(4)(1)(1)(16)=4096

Step 5: count the reduced-depth 3 × 3 layer.

64(8)(3)(3)(4)=18432LaTeX: 64(8)(3)(3)(4)=18432

Step 6: sum both bottleneck stages.

4096+18432=22528LaTeX: 4096+18432=22528

Step 7: subtract to obtain savings.

73728-22528=51200LaTeX: 73728-22528=51200

Final answer: Direct = 73,728 MACs; bottleneck = 22,528 MACs; savings = 51,200 MACs.