Deep Learning · TEE · section

Counting parameters in a dense network

Neural-network foundations

Definition. A dense layer connects every input component to every receiving neuron. Its trainable parameters are these connection weights and the receiving neurons' independently trainable biases.

Deriving the parameter count

For m inputs and n receiving neurons, each receiver needs m weights. There are therefore mn weights. With one bias per receiver, there are n additional parameters:

LaTeX: P_{\mathrm{dense}}=mn+n=(m+1)n

Activation values are computed from data and parameters; they are not additional trainable parameters. The input layer supplies data and does not acquire an extra bias merely because it appears in a diagram.

Diagram: Count all input-to-neuron connections and then one bias per receiving neuron.. Labels: Count connections separately from biases | Count connections separately from biases | A dense layer with 3 inputs and 2 receiving neurons. | x1 | x2 | x3 | a1 | + b1 | a2 | + b2 | 3 × 2 = 6 weights | Two receiving neurons also have 2 biases. Total = 6 + 2 = 8 parameters.

Complete numerical

Question. A network has 10 input features, three hidden layers with 5 neurons each, and 4 outputs. Every receiving neuron has a bias. Count its trainable parameters.

Given

Layer sizes: 10 inputs → 5 hidden → 5 hidden → 5 hidden → 4 outputs.

Every hidden/output neuron has one bias.

To find

Count weights and biases at each receiving layer, then the total.

Formula — which one and why

Dense-layer parameters. Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers.

n_in = incoming features; n_out = receiving neurons. The input layer itself has no learned receiving weights or biases.

LaTeX: P=(n_{\mathrm{in}}+1)n_{\mathrm{out}}

Yaad rakhne ka cue: Har output ke liye saare inputs plus ek bias.

Solution — step by step

Step 1: identify the consecutive layer widths.

LaTeX: 10\longrightarrow5\longrightarrow5\longrightarrow5\longrightarrow4

There are four trainable dense transformations. A connection between two hidden layers is counted once, as part of the receiving layer's parameters.

Step 2: count weights and biases for each transformation.

Step 3: add the layer totals.

LaTeX: P=55+30+30+24=139

Equivalently, the network has 120 weights and 19 biases, so 120 + 19 = 139.

Final answer: 139 trainable parameters. If a question explicitly removes biases, count only the 120 weights. Optimizer states such as Adam's moments are extra training storage, not extra parameters in this predictive network.