# Flatten and dense parameter count

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/34

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/34/

Flattening rearranges all activation values into a list. It learns no new numbers. A dense layer then connects every value in that list to every receiving neuron; each receiving neuron also has one bias.



$$
N_{flat}=HWC
$$





$$
P_{dense}=(N_{in}+1)N_{out}
$$



## Counting parameters in a dense network

**Definition.** A dense layer connects every input component to every receiving neuron. Its trainable parameters are these connection weights and the receiving neurons' independently trainable biases.

### Deriving the parameter count

For m inputs and n receiving neurons, each receiver needs m weights. There are therefore mn weights. With one bias per receiver, there are n additional parameters:



$$
P_{\mathrm{dense}}=mn+n=(m+1)n
$$



Activation values are computed from data and parameters; they are not additional trainable parameters. The input layer supplies data and does not acquire an extra bias merely because it appears in a diagram.

![Count all input-to-neuron connections and then one bias per receiving neuron.](https://deep-learning-study-room.vercel.app/assets/figures/v_dense_count.svg)

Diagram: Count all input-to-neuron connections and then one bias per receiving neuron.. Labels: Count connections separately from biases | Count connections separately from biases | A dense layer with 3 inputs and 2 receiving neurons. | x1 | x2 | x3 | a1 | + b1 | a2 | + b2 | 3 × 2 = 6 weights | Two receiving neurons also have 2 biases. Total = 6 + 2 = 8 parameters.

Count all input-to-neuron connections and then one bias per receiving neuron. Tap the diagram to enlarge.

### Complete numerical

**Question.** A network has 10 input features, three hidden layers with 5 neurons each, and 4 outputs. Every receiving neuron has a bias. Count its trainable parameters.

### Given

Layer sizes: 10 inputs → 5 hidden → 5 hidden → 5 hidden → 4 outputs.

Every hidden/output neuron has one bias.

### To find

Count weights and biases at each receiving layer, then the total.

### Formula — which one and why

**Dense-layer parameters.** Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers.

n\_in = incoming features; n\_out = receiving neurons. The input layer itself has no learned receiving weights or biases.



$$
P=(n_{\mathrm{in}}+1)n_{\mathrm{out}}
$$



**Yaad rakhne ka cue:** Har output ke liye saare inputs plus ek bias.

### Solution — step by step

**Step 1: identify the consecutive layer widths.**



$$
10\longrightarrow5\longrightarrow5\longrightarrow5\longrightarrow4
$$



There are four trainable dense transformations. A connection between two hidden layers is counted once, as part of the receiving layer's parameters.

**Step 2: count weights and biases for each transformation.**

| Transformation | Weights | Biases | Layer total |
| --- | --- | --- | --- |
| Input → hidden 1 | 10 × 5 = 50 | 5 | 55 |
| Hidden 1 → hidden 2 | 5 × 5 = 25 | 5 | 30 |
| Hidden 2 → hidden 3 | 5 × 5 = 25 | 5 | 30 |
| Hidden 3 → output | 5 × 4 = 20 | 4 | 24 |

**Step 3: add the layer totals.**



$$
P=55+30+30+24=139
$$



Equivalently, the network has 120 weights and 19 biases, so 120 + 19 = 139.

**Final answer: 139 trainable parameters.** If a question explicitly removes biases, count only the 120 weights. Optimizer states such as Adam's moments are extra training storage, not extra parameters in this predictive network.

## One complete CNN pipeline

**Question.** Input 128 × 128 × 3 → convolution 3 × 3, 16 filters, stride 1, no padding → pooling 2 × 2, stride 2 → convolution 3 × 3, 32 filters, stride 1, no padding → pooling 2 × 2, stride 2 → dense 240 → dense 120 → dense 5 with softmax. Find every output shape and trainable parameter count. Assume one bias per receiving filter or dense neuron.

### Given

Input = 128 × 128 × 3.

Conv: 3 × 3,16 filters,stride 1,valid → pool 2,stride 2.

Conv: 3 × 3,32 filters,stride 1,valid → pool 2,stride 2.

Flatten → dense 240 → dense 120 → dense 5.

One bias per learned filter/dense neuron.

### To find

Find every output shape and layer parameter count, then the network total.

### Formula — which one and why

**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.

H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.



$$
N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1
$$





$$
P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}}
$$





$$
n_{\mathrm{flat}}=HWC
$$





$$
P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}}
$$





$$
P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0
$$



**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.

### Solution — step by step

**Step 1. Conv 1 size.** (128 − 3)/1 + 1 = 126; output 126 × 126 × 16. Parameters = (3 × 3 × 3 + 1) × 16 = 448.

![Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count.](https://deep-learning-study-room.vercel.app/assets/figures/v_cnn_pipeline.svg)

Diagram: Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count.. Labels: Follow the complete CNN question, one layer at a time | Follow the complete CNN question, one layer at a time | Start with 128 × 128 × 3. Shape and learned-parameter count are separate. | LAYER | OUTPUT SHAPE / LENGTH | PARAMETERS | 1. Convolution 1 | 126 × 126 × 16 | 448 | 2. Pooling 1 | 63 × 63 × 16 | 0 | 3. Convolution 2 | 61 × 61 × 32 | 4,640 | 4. Pooling 2 | 30 × 30 × 32 | 0 | 5. Flatten | 28,800 | 0 | 6. Dense 240 | 240 | 6,912,240 | 7. Dense 120 | 120 | 28,920 | 8. Classifier | 5 | 605 | Total learned parameters = 6,946,853 | Each output is the next layer’s input. Pooling and flattening add no learned | parameters.

Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count. Tap the diagram to enlarge.

**Step 2. Pool 1 size.**  $\left\lfloor\frac{126 - 2}{2}\right\rfloor+1$  = 63; output 63 × 63 × 16. Parameters = 0.

**Step 3. Conv 2 size.** (63 − 3)/1 + 1 = 61; output 61 × 61 × 32. Input channels are now 16, not the original 3. Parameters = (3 × 3 × 16 + 1) × 32 = 4,640.

**Step 4. Pool 2 size.**  $\left\lfloor\frac{61 - 2}{2}\right\rfloor+1$  =  $\left\lfloor 29.5\right\rfloor$  + 1 = 30; output 30 × 30 × 32. Parameters = 0.

**Step 5. Flatten.** 30 × 30 × 32 = 28,800 values. Parameters = 0.

**Step 6. Dense 240.** 28,800 weights per neuron plus one bias, multiplied by 240: (28,800 + 1) × 240 = 6,912,240.

**Step 7. Dense 120.** (240 + 1) × 120 = 28,920.

**Step 8. Classifier.** (120 + 1) × 5 = 605. Softmax itself adds no extra parameters; these 605 belong to the dense class-score layer.

**Step 9. Total.** 448 + 4,640 + 6,912,240 + 28,920 + 605 = **6,946,853**. Most parameters come from the first dense layer.

## Drill 37 Complete small CNN

## Question

**Question.** Input 10 × 10 × 1 → Conv 3 × 3,4 filters, stride 1, valid → Pool 2 stride 2 → Flatten → Dense 3. Find all shapes and parameters.

## Answer — step by step

### Given

Input = 10 × 10 × 1.

Conv: 3 × 3, 4 filters, stride 1, valid padding.

Pool: 2 × 2, stride 2; then flatten and dense layer with 3 outputs.

Include one bias per filter/dense neuron.

### To find

Find all output shapes, layer parameter counts and their sum.

### Formula — which one and why

**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.

H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.



$$
N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1
$$





$$
P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}}
$$





$$
n_{\mathrm{flat}}=HWC
$$





$$
P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}}
$$





$$
P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0
$$



**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.

### Solution — step by step

### Formulas and assumptions

“Valid” means zero padding. The convolution and dense layers each have one bias per receiving filter or neuron. Pooling and flattening have no learned parameters.



$$
N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1
$$





$$
P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}}
$$





$$
n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}}
$$



### Step 1: convolution shape



$$
N_{\mathrm{conv}}=\frac{10+0-3}{1}+1=7+1=8
$$



Four filters produce four channels: **8 × 8 × 4**.

### Step 2: convolution parameters



$$
\text{weights}=3\times3\times1\times4=36
$$





$$
\text{biases}=4,\qquad P_{\mathrm{conv}}=36+4=40
$$



### Step 3: pooling shape

Pooling now receives size 8, not the original size 10.



$$
N_{\mathrm{pool}}=\left\lfloor\frac{8-2}{2}\right\rfloor+1=3+1=4
$$



Channels remain 4: **4 × 4 × 4**. Pooling parameters = 0.

### Step 4: flatten



$$
n_{\mathrm{flat}}=4\times4\times4=64
$$



Flatten creates a vector of 64 numbers. It rearranges values, so its parameter count is 0.

### Step 5: dense layer with 3 outputs



$$
\text{weights}=64\times3=192
$$





$$
\text{biases}=3,\qquad P_{\mathrm{dense}}=192+3=195
$$



### Step 6: add learned parameters



$$
P_{\mathrm{total}}=40+0+0+195=235
$$



**Final answer:** shapes 8 × 8 × 4 → 4 × 4 × 4 → 64 → 3; total 235 parameters.

## Drill 49 Mixed CNN dimensions softmax and loss

## Question

**Question.** Input 8 × 8 × 3 → Conv 3, stride 1, padding 1, 2 filters → Pool 2, stride 2 → Flatten → Dense 3. Logits [0,1,−1], true class is the second class (one-hot target [0,1,0]). Find shapes, parameters, probabilities and CCE.

## Answer — step by step

### Given

Input = 8 × 8 × 3.

Conv: kernel 3, stride 1, padding 1, 2 filters.

Pool: kernel 2, stride 2; flatten; dense layer with 3 outputs.

Logits = [0,1,−1]; true class is the second class (one-hot target [0,1,0]).

### To find

Find shapes, parameter counts, probabilities and CCE.

### Formula — which one and why

**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.

H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.



$$
N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1
$$





$$
P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}}
$$





$$
n_{\mathrm{flat}}=HWC
$$





$$
P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}}
$$





$$
P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0
$$



**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.

**Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.

zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.



$$
M=\max_j z_j
$$





$$
p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}}
$$



**Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo.

**Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.

yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.



$$
L=-\sum_j y_j\ln p_j
$$





$$
L=-\ln p_c
$$



**Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln.

### Solution — step by step

### Formulas used

Separate shape from learned parameter count. H, W and C mean height, width and channels. K, P and S mean filter size, padding per side and stride.



$$
N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1
$$





$$
P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}}
$$





$$
n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}}
$$



The +1 counts one bias per receiving filter or neuron. Pooling and flattening add no learned parameters. Softmax converts scores to probabilities; CCE uses the probability of the correct class.



$$
p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\quad c=\max_jz_j,\quad L=-\ln p_{\mathrm{correct}}
$$



Conv size = (8+2−3)+1 = 8; output 8 × 8 × 2; parameters = (3 × 3 × 3+1) × 2 = 56. Pool 4 × 4 × 2; parameters 0. Flatten 32. Dense 3 parameters = (32+1) × 3 = 99. Total = **155**.

**Step 1 stabilize:**  subtract the largest logit 1 from every value: [-1, 0, -2]. This does not change softmax.

**Step 2 exponentiate:**  exp values = [0.367879441, 1, 0.135335283].

**Step 3 denominator:**  sum = 1.503214724.

**Step 4 divide each value by this same sum:**  probabilities = **[0.244728471, 0.665240956, 0.090030573]**. **Check:**  all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability.

Correct class probability = 0.665240956; CCE = −ln(0.665240956) = **0.407605964**.