Deep Learning · TEE · formulas

Global average pooling

Global average pooling takes one mean per complete spatial channel. The worked example compares the resulting dense classifier with a classifier after flattening.

$g_c=\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}a_{i,j,c}$ LaTeX: g_c=\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}a_{i,j,c}

Global average pooling: one average for each channel

Average each channel separately: these are the exact values used in the worked example. Tap the diagram to enlarge.

Definition. Global average pooling, or GAP, replaces every complete spatial feature map by its mean. An H × W × C tensor becomes C numbers. The channels remain separate; we do not average all channels into one number.

$g_c=\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}a_{i,j,c}$ LaTeX: g_c=\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}a_{i,j,c}

The letter c selects a channel; i and j select its row and column; a is the value at that location. Add all H × W entries in that channel and divide by H × W. GAP has no learned weights or biases. A classifier after GAP can still have trainable parameters. In GoogLeNet, 7 × 7 × 1,024 becomes 1 × 1 × 1,024 before the final 1,000-class linear classifier.

Question (practice, 3 marks). A 2 × 2 × 2 tensor has channel A = $\begin{bmatrix}1 & 3\\5 & 7\end{bmatrix}$ and channel B = $\begin{bmatrix}2 & 4\\6 & 8\end{bmatrix}$ . Apply GAP, then find the parameters of a three-output dense classifier receiving the result, including one bias per output.

Given

H = 2, W = 2, C = 2. Each of the two channels contains four values. The classifier has three outputs.

To find

The GAP result and dense-head parameter count.

Formula — which one and why

Use one mean per channel. The dense layer receives two numbers, so each of its three outputs needs two weights and one bias.

$P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}}$ LaTeX: P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}}

Solution — step by step

Step 1: add channel A. 1 + 3 + 5 + 7 = 16.

Step 2: average channel A. 16 ÷ 4 = 4.

Step 3: add channel B. 2 + 4 + 6 + 8 = 20.

Step 4: average channel B. 20 ÷ 4 = 5. Thus GAP outputs [4,5].

Step 5: count the classifier's weights. 2 inputs × 3 outputs = 6 weights.

Step 6: add three biases. 6 + 3 = 9 parameters. GAP itself contributes zero.

Final answer: GAP = [4,5]; GAP shape = 1 × 1 × 2; classifier parameters = 9. If we flattened all eight inputs instead, that head would require (8 + 1) × 3 = 27 parameters.

Yaad rakho: ek channel ka poora average; channels ko aapas mein mat milao.