Deep Learning · MT2 · section

Forward propagation through multiple neurons

Neural-network foundations

Definitions. A feedforward neural network has a directed acyclic computation graph: information moves from inputs toward outputs without recurrent feedback connections. A multilayer perceptron (MLP) is a feedforward network built from fully connected layers, with one or more hidden layers and nonlinear hidden activations.

Input, hidden and output layers

The input layer supplies the feature vector. A hidden layer learns an intermediate representation; its outputs are computed from preceding activations, not supplied as target labels. The output layer produces the prediction in the form needed by the task.

In a dense connection between two layers, every receiving neuron has a weight for every preceding activation. Each receiving neuron also normally has its own bias. An ANN is the broader family containing feedforward and recurrent models. A CNN can be feedforward without being an MLP, because convolution uses local shared connections.

A feedforward network calculates each hidden representation before using it to calculate the next layer. Tap the diagram to enlarge.

General layer equations and dimensions

Let a⁽⁰⁾ = x. For layer l:

$z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}$ LaTeX: z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}

$a^{(l)}=f^{(l)}(z^{(l)})$ LaTeX: a^{(l)}=f^{(l)}(z^{(l)})

If the preceding layer has n inputs and the current layer has m neurons, W has m rows and n columns. The vectors z, a and b have m components. Row j of W contains the incoming weights of neuron j.

For one hidden layer, these computations compose to give:

$h=f^{(1)}(W^{(1)}x+b^{(1)})$ LaTeX: h=f^{(1)}(W^{(1)}x+b^{(1)})

$\hat y=f^{(2)}(W^{(2)}h+b^{(2)})$ LaTeX: \hat y=f^{(2)}(W^{(2)}h+b^{(2)})

The hidden activation must be nonlinear to obtain nonlinear representational power. Stacking affine layers alone still gives one affine mapping. A single threshold perceptron cannot separate XOR; a hidden nonlinear representation can make the two classes separable for the output layer.

Forward propagation versus training

Forward propagation evaluates these layer equations using the current parameters. Training additionally evaluates a loss, uses backpropagation to calculate gradients, and applies an optimizer update. The architecture specifies connections; backpropagation is a differentiation algorithm used to train that architecture.

Complete two-layer numerical

Question. A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ = $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$ and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. Calculate every preactivation and the final output.

Given

A feedforward network receives x = [1, 0]ᵀ.

Its hidden-layer weights are W¹ = $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$ and biases are b¹ = [0.1, 0.2]ᵀ.

The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1.

Use ReLU in both layers and one row per receiving neuron.

To find

Find all preactivations, hidden activations and final outputs. Find the requested activation values and slopes.

Formula — which one and why

Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.

x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.

$z_h=W_1x+b_1$ LaTeX: z_h=W_1x+b_1

$h=f_1(z_h)$ LaTeX: h=f_1(z_h)

$z_o=W_2h+b_2$ LaTeX: z_o=W_2h+b_2

$o=f_2(z_o)$ LaTeX: o=f_2(z_o)

Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.

ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.

$\operatorname{ReLU}(z)=\max(0,z)$ LaTeX: \operatorname{ReLU}(z)=\max(0,z)

$\operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}$ LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}

Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

Solution — step by step

The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron. Tap the diagram to enlarge.

Step 1: identify the matrix dimensions. W¹ is 2 × 2 and x is 2 × 1, so the hidden preactivation has two entries. W² is 1 × 2, so it takes those two activations and produces one output.

Step 2: calculate the first hidden preactivation. Multiply the first row by x, then add the first hidden bias.

Step 3: calculate the second hidden preactivation.

Step 4: activate the hidden layer. Both values are positive, so ReLU retains them.

$a^{(1)}_1=\max(0,0.3)=0.3$ LaTeX: a^{(1)}_1=\max(0,0.3)=0.3

$a^{(1)}_2=\max(0,0.5)=0.5$ LaTeX: a^{(1)}_2=\max(0,0.5)=0.5

$a^{(1)}=\begin{bmatrix}0.3\\0.5\end{bmatrix}$ LaTeX: a^{(1)}=\begin{bmatrix}0.3\\0.5\end{bmatrix}

Step 5: calculate the output preactivation. Use the activated hidden values as this layer's inputs.

Step 6: apply the output activation.

$\hat y=\max(0,0.55)=0.55$ LaTeX: \hat y=\max(0,0.55)=0.55

Final answer: hidden preactivations and activations are [0.3, 0.5]ᵀ; output preactivation and prediction are both 0.55. They coincide here because every ReLU input is positive. No parameter has been updated during this forward pass.