# Why one by one convolution saves work

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s2

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s2/

**Definition.** A 1 × 1 convolution combines the input channels independently at each spatial location. A bottleneck uses fewer output channels before a larger spatial convolution, reducing its parameter and multiplication counts.

![Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size.](https://deep-learning-study-room.vercel.app/assets/figures/v_bottleneck.svg)

Diagram: Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size.. Labels: Compare the two paths in the worked question | Compare the two paths in the worked question | Input 28 × 28 × 192. Output 28 × 28 × 32. Count multiplications (MACs). | DIRECT | One 5 × 5 convolution | 28 × 28 × 32 × 25 × 192 | = 120,422,400 multiplications | REDUCE | 1 × 1: 192 → 16 | 28 × 28 × 16 × 192 | = 2,408,448 | 5 × 5: 16 → 32 | 28 × 28 × 32 × 25 × 16 | = 10,035,200 | Total with reduction = 12,443,648 MACs | Reducing depth to 16 before the 5 × 5 filter saves about 89.67% of these | multiplications.

Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size. Tap the diagram to enlarge.

**Question. Input is 28 × 28 × 192. Compare a direct same-padded 5 × 5 convolution with 32 filters against a 1 × 1 reduction to 16 channels followed by the same 5 × 5 convolution. Count MACs and ignore bias additions.**

### Given

Both paths produce 28 × 28 × 32. The direct layer receives 192 channels; the bottleneck's spatial layer receives 16 channels. Stride is 1 and padding preserves spatial size.

### To find

Direct MACs, total two-stage bottleneck MACs and percentage saved.

### Formula — which one and why

Each output scalar uses one product per input-channel kernel entry. Multiply that cost by the number of output positions and channels. The two bottleneck layers require separate counts, which are added.



$$
\mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}}
$$



One MAC counts one multiply–accumulate pair. A convention counting multiplication and addition separately gives approximately twice as many FLOPs.

### Solution — step by step

**Step 1: direct layer.** Each output uses 5 × 5 × 192 = 4,800 products. There are 28 × 28 × 32 = 25,088 outputs.



$$
\mathrm{MACs}_{\mathrm{direct}}=25088(4800)=120422400
$$



**Step 2: 1 × 1 reduction.** There are 16 receiving filters, each spanning all 192 original channels.



$$
\mathrm{MACs}_{1\times1}=28(28)(16)(192)=2408448
$$



**Step 3: reduced-depth spatial layer.** Its 32 filters now span 16 channels.



$$
\mathrm{MACs}_{5\times5}=28(28)(32)(25)(16)=10035200
$$



**Step 4: add the two bottleneck costs.**



$$
\mathrm{MACs}_{\mathrm{bottleneck}}=2408448+10035200=12443648
$$



**Step 5: calculate the saving.**



$$
\mathrm{saving}=120422400-12443648=107978752
$$





$$
\mathrm{saving\ percentage}=100\frac{107978752}{120422400}\approx89.666667\%
$$



**Final answer:** direct = 120,422,400 MACs; bottleneck = 12,443,648 MACs, saving about 89.67%. Spatial size is unchanged; the saving comes from reducing the channels entering the expensive 5 × 5 layer. If the reduction width is not supplied, use a symbol for it and label any numerical choice as an assumption.

### A second bottleneck with different dimensions

**Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with a 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.**

### Given

Spatial output = 8 × 8; original channels = 16; final filters = 8; bottleneck channels = 4; stride = 1.

### To find

Direct MACs, bottleneck MACs and saving.

### Formula — which one and why

Use the same MAC formula above once for the direct path and once for each bottleneck stage. Every branch must include all of its layers.

### Solution — step by step

**Step 1: direct spatial convolution.**



$$
8(8)(8)(3)(3)(16)=73728
$$



**Step 2: 1 × 1 reduction.**



$$
8(8)(4)(1)(1)(16)=4096
$$



**Step 3: reduced-depth 3 × 3 convolution.**



$$
8(8)(8)(3)(3)(4)=18432
$$



**Step 4: add the stages and subtract from the direct cost.**



$$
4096+18432=22528
$$





$$
73728-22528=51200
$$



**Final answer:** direct = 73,728 MACs; bottleneck = 22,528 MACs; saving = 51,200 MACs, about 69.44%. Both paths produce 8 × 8 × 8, but they are different learned transformations and need not produce identical feature values.