Deep Learning · TEE · formulas

Layer forward pass

A layer does the neuron calculation for several receiving neurons. Finish all its outputs before using them as the next layer’s inputs. The bracketed superscript is a layer label, not a power.

$z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}$ LaTeX: z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}

$a^{(l)}=f(z^{(l)})$ LaTeX: a^{(l)}=f(z^{(l)})

Forward propagation through multiple neurons

Definitions. A feedforward neural network has a directed acyclic computation graph: information moves from inputs toward outputs without recurrent feedback connections. A multilayer perceptron (MLP) is a feedforward network built from fully connected layers, with one or more hidden layers and nonlinear hidden activations.

Input, hidden and output layers

The input layer supplies the feature vector. A hidden layer learns an intermediate representation; its outputs are computed from preceding activations, not supplied as target labels. The output layer produces the prediction in the form needed by the task.

In a dense connection between two layers, every receiving neuron has a weight for every preceding activation. Each receiving neuron also normally has its own bias. An ANN is the broader family containing feedforward and recurrent models. A CNN can be feedforward without being an MLP, because convolution uses local shared connections.

A feedforward network calculates each hidden representation before using it to calculate the next layer. Tap the diagram to enlarge.

General layer equations and dimensions

Let a⁽⁰⁾ = x. For layer l:

$z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}$ LaTeX: z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}

$a^{(l)}=f^{(l)}(z^{(l)})$ LaTeX: a^{(l)}=f^{(l)}(z^{(l)})

If the preceding layer has n inputs and the current layer has m neurons, W has m rows and n columns. The vectors z, a and b have m components. Row j of W contains the incoming weights of neuron j.

For one hidden layer, these computations compose to give:

$h=f^{(1)}(W^{(1)}x+b^{(1)})$ LaTeX: h=f^{(1)}(W^{(1)}x+b^{(1)})

$\hat y=f^{(2)}(W^{(2)}h+b^{(2)})$ LaTeX: \hat y=f^{(2)}(W^{(2)}h+b^{(2)})

The hidden activation must be nonlinear to obtain nonlinear representational power. Stacking affine layers alone still gives one affine mapping. A single threshold perceptron cannot separate XOR; a hidden nonlinear representation can make the two classes separable for the output layer.

Forward propagation versus training

Forward propagation evaluates these layer equations using the current parameters. Training additionally evaluates a loss, uses backpropagation to calculate gradients, and applies an optimizer update. The architecture specifies connections; backpropagation is a differentiation algorithm used to train that architecture.

Complete two-layer numerical

Question. A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ = $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$ and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. Calculate every preactivation and the final output.

Given

A feedforward network receives x = [1, 0]ᵀ.

Its hidden-layer weights are W¹ = $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$ and biases are b¹ = [0.1, 0.2]ᵀ.

The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1.

Use ReLU in both layers and one row per receiving neuron.

To find

Find all preactivations, hidden activations and final outputs. Find the requested activation values and slopes.

Formula — which one and why

Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.

x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.

$z_h=W_1x+b_1$ LaTeX: z_h=W_1x+b_1

$h=f_1(z_h)$ LaTeX: h=f_1(z_h)

$z_o=W_2h+b_2$ LaTeX: z_o=W_2h+b_2

$o=f_2(z_o)$ LaTeX: o=f_2(z_o)

Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.

ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.

$\operatorname{ReLU}(z)=\max(0,z)$ LaTeX: \operatorname{ReLU}(z)=\max(0,z)

$\operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}$ LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}

Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

Solution — step by step

The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron. Tap the diagram to enlarge.

Step 1: identify the matrix dimensions. W¹ is 2 × 2 and x is 2 × 1, so the hidden preactivation has two entries. W² is 1 × 2, so it takes those two activations and produces one output.

Step 2: calculate the first hidden preactivation. Multiply the first row by x, then add the first hidden bias.

Step 3: calculate the second hidden preactivation.

Step 4: activate the hidden layer. Both values are positive, so ReLU retains them.

$a^{(1)}_1=\max(0,0.3)=0.3$ LaTeX: a^{(1)}_1=\max(0,0.3)=0.3

$a^{(1)}_2=\max(0,0.5)=0.5$ LaTeX: a^{(1)}_2=\max(0,0.5)=0.5

$a^{(1)}=\begin{bmatrix}0.3\\0.5\end{bmatrix}$ LaTeX: a^{(1)}=\begin{bmatrix}0.3\\0.5\end{bmatrix}

Step 5: calculate the output preactivation. Use the activated hidden values as this layer's inputs.

Step 6: apply the output activation.

$\hat y=\max(0,0.55)=0.55$ LaTeX: \hat y=\max(0,0.55)=0.55

Final answer: hidden preactivations and activations are [0.3, 0.5]ᵀ; output preactivation and prediction are both 0.55. They coincide here because every ReLU input is positive. No parameter has been updated during this forward pass.

Model answer: forward propagation, backpropagation and updating

Question

Question. Explain how a feedforward neural network learns using forward propagation, a loss function, backpropagation and gradient descent. State the order of operations.

Answer — step by step

Training repeatedly uses labeled examples to improve a network's weights and biases. The four operations have different jobs.

Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters. Tap the diagram to enlarge.

Operation	Direction or action	Result
Forward propagation	Inputs move through successive layers	Predictions and intermediate activations
Loss calculation	Compare predictions with targets	A scalar objective for the example or batch
Backpropagation	Apply the chain rule from the loss toward earlier layers	Gradients of the objective with respect to parameters
Optimizer update	Use those gradients to change parameters	A new set of weights and biases

1. Forward propagation

At each layer, calculate its weighted sums and activations. Retain these values because derivatives such as the sigmoid derivative depend on them. Complete a whole layer before calculating the next.

2. Loss

Use the loss specified by the task. For one regression output, half squared loss is one possible choice:

$L=\frac12(a-y)^2$ LaTeX: L=\frac12(a-y)^2

Here a is the prediction and y is the target. Other stated losses produce different derivatives.

3. Backpropagation

Multiply local derivatives along each dependency path. If a hidden unit influences several outputs, add the contributions from those branches. For a connection, the weight gradient is the receiving neuron's net-input derivative multiplied by the sending activation.

$\frac{\partial L}{\partial w_{ji}}=\delta_j a_i$ LaTeX: \frac{\partial L}{\partial w_{ji}}=\delta_j a_i

Here δⱼ means the derivative of loss with respect to the receiving neuron's weighted sum. It is not automatically equal to raw prediction error; the activation and loss determine it.

4. Update and repeat

$w_{\mathrm{new}}=w_{\mathrm{old}}-\eta\frac{\partial L}{\partial w}$ LaTeX: w_{\mathrm{new}}=w_{\mathrm{old}}-\eta\frac{\partial L}{\partial w}

Compute all gradients using the same pre-update weights. Then update the requested parameters. Run another forward pass to calculate the new loss. Backpropagation calculates gradients; gradient descent uses them. Those are related, separate operations.

Backpropagation with one hidden neuron

Definition. Backpropagation computes the derivatives of a scalar objective with respect to network parameters by applying the chain rule in reverse through the computational graph. An optimizer then uses these derivatives to update the parameters.

Network equations and the meaning of delta

Consider one input, one hidden neuron and one output neuron:

For the half squared loss used in this example:

$L=\frac12(\hat y-y)^2$ LaTeX: L=\frac12(\hat y-y)^2

Define a neuron's delta as the derivative of the loss with respect to its preactivation, not with respect to its output:

$\delta_o=\frac{\partial L}{\partial z_o}=(\hat y-y)g'(z_o)$ LaTeX: \delta_o=\frac{\partial L}{\partial z_o}=(\hat y-y)g'(z_o)

The output-weight derivative contains the hidden activation because zₒ = w₂h + b₂:

$\frac{\partial L}{\partial w_2}=\delta_o h,\qquad\frac{\partial L}{\partial b_2}=\delta_o$ LaTeX: \frac{\partial L}{\partial w_2}=\delta_o h,\qquad\frac{\partial L}{\partial b_2}=\delta_o

To reach the hidden neuron, continue backwards through the old output weight and the hidden activation:

$\delta_h=\delta_o w_2 f'(z_h)$ LaTeX: \delta_h=\delta_o w_2 f'(z_h)

$\frac{\partial L}{\partial w_1}=\delta_h x,\qquad\frac{\partial L}{\partial b_1}=\delta_h$ LaTeX: \frac{\partial L}{\partial w_1}=\delta_h x,\qquad\frac{\partial L}{\partial b_1}=\delta_h

The bias derivative equals the corresponding delta because the derivative of z with respect to its additive bias is 1. All these derivatives belong to the same forward pass; an updated weight must not be used partway through this backward pass.

Complete worked numerical

Question. x = 2, y = 1; w₁ = 0.5, b₁ = 0.1; w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1.