Deep Learning · MT2 · section

Parameters are different from activation values

Convolutional neural networks

Definition. Parameters are learned coefficients shared across evaluations. Activations are input-dependent values computed during a forward pass. The number of activation values therefore differs from the number of trainable parameters.

Explanation and working

An activation is a number produced for a particular input. A parameter is a learned number reused across inputs. A 100 × 100 feature map can be produced by a filter with only a few weights.

Diagram: The ten learned parameters are reused at nine spatial positions. The resulting nine numbers are activations.. Labels: Learned parameters are not output activations | Learned parameters are not output activations | Example: a 5 × 5 × 1 input with one 3 × 3 filter, stride 1, no padding. | Learned filter | 3 × 3 × 1 = 9 weights | Plus one bias | Total parameters = 10 | Output feature map | (5 − 3)/1 + 1 = 3 | Shape = 3 × 3 × 1 | Total activations = 9 | The same ten parameters are reused to calculate all nine output values.

LaTeX: P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}}

The +1 is one bias per filter. If bias is explicitly disabled, remove it. Height and width of the image do not enter the parameter formula; they enter the number of output activations and the amount of computation.

One filter, then all filters

Question. A 3 × 3 convolution receives 3 input channels and has 16 filters, each with one bias. Find its trainable parameters and output-channel count.

Given

Kernel = 3 × 3; input channels = 3; filters = 16; one bias per filter.

To find

Weights, biases, total trainable parameters and output channels.

Formula — which one and why

Each filter spans every input channel and owns one bias. Spatial reuse does not create extra weights.

Kₕ,Kw are kernel dimensions; Cᵢₙ is input depth; Cₒᵤₜ is the number of filters.

LaTeX: P=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}}

Yaad rakhne ka cue: Ek filter ka count banao, phir filters se multiply.

Solution — step by step

Step 1: count weights in one filter.

LaTeX: 3(3)(3)=27

Step 2: include its one bias.

LaTeX: 27+1=28

Step 3: count all weights.

LaTeX: 27(16)=432

Step 4: count all biases.

LaTeX: 1(16)=16

Step 5: add and state channel count.

LaTeX: P=432+16=448

Each filter produces one channel, so output channels = 16.

Final answer: 432 weights + 16 biases = 448 trainable parameters; 16 output channels.

LaTeX: P_{\mathrm{pool}}=0,\quad P_{\mathrm{flatten}}=0,\quad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}}

Flatten rearranges H × W × C activation values into one vector of length HWC. It does not learn anything. A following dense layer connects to every one of those values.