# Forward propagation through multiple neurons

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s13

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s13/

**Definitions.** A feedforward neural network has a directed acyclic computation graph: information moves from inputs toward outputs without recurrent feedback connections. A multilayer perceptron (MLP) is a feedforward network built from fully connected layers, with one or more hidden layers and nonlinear hidden activations.

### Input, hidden and output layers

The **input layer** supplies the feature vector. A **hidden layer** learns an intermediate representation; its outputs are computed from preceding activations, not supplied as target labels. The **output layer** produces the prediction in the form needed by the task.

In a dense connection between two layers, every receiving neuron has a weight for every preceding activation. Each receiving neuron also normally has its own bias. An ANN is the broader family containing feedforward and recurrent models. A CNN can be feedforward without being an MLP, because convolution uses local shared connections.

![A feedforward network calculates each hidden representation before using it to calculate the next layer.](https://deep-learning-study-room.vercel.app/assets/figures/network.svg)

Diagram: A feedforward network calculates each hidden representation before using it to calculate the next layer.. Labels: INPUT | HIDDEN | OUTPUT | x1 | x2 | h1 | h2 | o | Each hidden neuron receives both inputs and one bias. | Forward uses activations. Backward uses OLD weights.

A feedforward network calculates each hidden representation before using it to calculate the next layer. Tap the diagram to enlarge.

### General layer equations and dimensions

Let a⁽⁰⁾ = x. For layer l:



$$
z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}
$$





$$
a^{(l)}=f^{(l)}(z^{(l)})
$$



If the preceding layer has n inputs and the current layer has m neurons, W has m rows and n columns. The vectors z, a and b have m components. Row j of W contains the incoming weights of neuron j.



$$
W^{(l)}\in\mathbb R^{m\times n},\qquad b^{(l)},z^{(l)},a^{(l)}\in\mathbb R^{m\times1}
$$



For one hidden layer, these computations compose to give:



$$
h=f^{(1)}(W^{(1)}x+b^{(1)})
$$





$$
\hat y=f^{(2)}(W^{(2)}h+b^{(2)})
$$



The hidden activation must be nonlinear to obtain nonlinear representational power. Stacking affine layers alone still gives one affine mapping. A single threshold perceptron cannot separate XOR; a hidden nonlinear representation can make the two classes separable for the output layer.

### Forward propagation versus training

Forward propagation evaluates these layer equations using the current parameters. Training additionally evaluates a loss, uses backpropagation to calculate gradients, and applies an optimizer update. The architecture specifies connections; backpropagation is a differentiation algorithm used to train that architecture.

### Complete two-layer numerical

**Question. A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ =  $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$  and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. Calculate every preactivation and the final output.**

### Given

A feedforward network receives x = [1, 0]ᵀ.

Its hidden-layer weights are W¹ =  $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$  and biases are b¹ = [0.1, 0.2]ᵀ.

The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1.

Use ReLU in both layers and one row per receiving neuron.

### To find

Find all preactivations, hidden activations and final outputs. Find the requested activation values and slopes.

### Formula — which one and why

**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.

x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.



$$
z_h=W_1x+b_1
$$





$$
h=f_1(z_h)
$$





$$
z_o=W_2h+b_2
$$





$$
o=f_2(z_o)
$$



**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.

**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.



$$
\operatorname{ReLU}(z)=\max(0,z)
$$





$$
\operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}
$$



**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

### Solution — step by step

![The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron.](https://deep-learning-study-room.vercel.app/assets/figures/given_5f379b367f.svg)

Diagram: The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron.. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.4 | 0.3 | 0.7

The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron. Tap the diagram to enlarge.

**Step 1: identify the matrix dimensions.** W¹ is 2 × 2 and x is 2 × 1, so the hidden preactivation has two entries. W² is 1 × 2, so it takes those two activations and produces one output.

**Step 2: calculate the first hidden preactivation.** Multiply the first row by x, then add the first hidden bias.



$$
z^{(1)}_1=0.2(1)+0.4(0)+0.1=0.3
$$



**Step 3: calculate the second hidden preactivation.**



$$
z^{(1)}_2=0.3(1)+0.7(0)+0.2=0.5
$$



**Step 4: activate the hidden layer.** Both values are positive, so ReLU retains them.



$$
a^{(1)}_1=\max(0,0.3)=0.3
$$





$$
a^{(1)}_2=\max(0,0.5)=0.5
$$





$$
a^{(1)}=\begin{bmatrix}0.3\\0.5\end{bmatrix}
$$



**Step 5: calculate the output preactivation.** Use the activated hidden values as this layer's inputs.



$$
z^{(2)}=0.5(0.3)+0.6(0.5)+0.1
$$





$$
z^{(2)}=0.15+0.30+0.10=0.55
$$



**Step 6: apply the output activation.**



$$
\hat y=\max(0,0.55)=0.55
$$



**Final answer:** hidden preactivations and activations are [0.3, 0.5]ᵀ; output preactivation and prediction are both 0.55. They coincide here because every ReLU input is positive. No parameter has been updated during this forward pass.