# Layer forward pass

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/7

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/7/

A layer does the neuron calculation for several receiving neurons. Finish all its outputs before using them as the next layer’s inputs. The bracketed superscript is a layer label, not a power.



$$
z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}
$$





$$
a^{(l)}=f(z^{(l)})
$$



## Forward propagation through multiple neurons

**Definitions.** A feedforward neural network has a directed acyclic computation graph: information moves from inputs toward outputs without recurrent feedback connections. A multilayer perceptron (MLP) is a feedforward network built from fully connected layers, with one or more hidden layers and nonlinear hidden activations.

### Input, hidden and output layers

The **input layer** supplies the feature vector. A **hidden layer** learns an intermediate representation; its outputs are computed from preceding activations, not supplied as target labels. The **output layer** produces the prediction in the form needed by the task.

In a dense connection between two layers, every receiving neuron has a weight for every preceding activation. Each receiving neuron also normally has its own bias. An ANN is the broader family containing feedforward and recurrent models. A CNN can be feedforward without being an MLP, because convolution uses local shared connections.

![A feedforward network calculates each hidden representation before using it to calculate the next layer.](https://deep-learning-study-room.vercel.app/assets/figures/network.svg)

Diagram: A feedforward network calculates each hidden representation before using it to calculate the next layer.. Labels: INPUT | HIDDEN | OUTPUT | x1 | x2 | h1 | h2 | o | Each hidden neuron receives both inputs and one bias. | Forward uses activations. Backward uses OLD weights.

A feedforward network calculates each hidden representation before using it to calculate the next layer. Tap the diagram to enlarge.

### General layer equations and dimensions

Let a⁽⁰⁾ = x. For layer l:



$$
z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}
$$





$$
a^{(l)}=f^{(l)}(z^{(l)})
$$



If the preceding layer has n inputs and the current layer has m neurons, W has m rows and n columns. The vectors z, a and b have m components. Row j of W contains the incoming weights of neuron j.



$$
W^{(l)}\in\mathbb R^{m\times n},\qquad b^{(l)},z^{(l)},a^{(l)}\in\mathbb R^{m\times1}
$$



For one hidden layer, these computations compose to give:



$$
h=f^{(1)}(W^{(1)}x+b^{(1)})
$$





$$
\hat y=f^{(2)}(W^{(2)}h+b^{(2)})
$$



The hidden activation must be nonlinear to obtain nonlinear representational power. Stacking affine layers alone still gives one affine mapping. A single threshold perceptron cannot separate XOR; a hidden nonlinear representation can make the two classes separable for the output layer.

### Forward propagation versus training

Forward propagation evaluates these layer equations using the current parameters. Training additionally evaluates a loss, uses backpropagation to calculate gradients, and applies an optimizer update. The architecture specifies connections; backpropagation is a differentiation algorithm used to train that architecture.

### Complete two-layer numerical

**Question. A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ =  $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$  and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. Calculate every preactivation and the final output.**

### Given

A feedforward network receives x = [1, 0]ᵀ.

Its hidden-layer weights are W¹ =  $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$  and biases are b¹ = [0.1, 0.2]ᵀ.

The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1.

Use ReLU in both layers and one row per receiving neuron.

### To find

Find all preactivations, hidden activations and final outputs. Find the requested activation values and slopes.

### Formula — which one and why

**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.

x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.



$$
z_h=W_1x+b_1
$$





$$
h=f_1(z_h)
$$





$$
z_o=W_2h+b_2
$$





$$
o=f_2(z_o)
$$



**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.

**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.



$$
\operatorname{ReLU}(z)=\max(0,z)
$$





$$
\operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}
$$



**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

### Solution — step by step

![The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron.](https://deep-learning-study-room.vercel.app/assets/figures/given_5f379b367f.svg)

Diagram: The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron.. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.4 | 0.3 | 0.7

The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron. Tap the diagram to enlarge.

**Step 1: identify the matrix dimensions.** W¹ is 2 × 2 and x is 2 × 1, so the hidden preactivation has two entries. W² is 1 × 2, so it takes those two activations and produces one output.

**Step 2: calculate the first hidden preactivation.** Multiply the first row by x, then add the first hidden bias.



$$
z^{(1)}_1=0.2(1)+0.4(0)+0.1=0.3
$$



**Step 3: calculate the second hidden preactivation.**



$$
z^{(1)}_2=0.3(1)+0.7(0)+0.2=0.5
$$



**Step 4: activate the hidden layer.** Both values are positive, so ReLU retains them.



$$
a^{(1)}_1=\max(0,0.3)=0.3
$$





$$
a^{(1)}_2=\max(0,0.5)=0.5
$$





$$
a^{(1)}=\begin{bmatrix}0.3\\0.5\end{bmatrix}
$$



**Step 5: calculate the output preactivation.** Use the activated hidden values as this layer's inputs.



$$
z^{(2)}=0.5(0.3)+0.6(0.5)+0.1
$$





$$
z^{(2)}=0.15+0.30+0.10=0.55
$$



**Step 6: apply the output activation.**



$$
\hat y=\max(0,0.55)=0.55
$$



**Final answer:** hidden preactivations and activations are [0.3, 0.5]ᵀ; output preactivation and prediction are both 0.55. They coincide here because every ReLU input is positive. No parameter has been updated during this forward pass.

## Model answer: forward propagation, backpropagation and updating

## Question

**Question.** Explain how a feedforward neural network learns using forward propagation, a loss function, backpropagation and gradient descent. State the order of operations.

## Answer — step by step

Training repeatedly uses labeled examples to improve a network's weights and biases. The four operations have different jobs.

![Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters.](https://deep-learning-study-room.vercel.app/assets/figures/v_training_loop.svg)

Diagram: Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters.. Labels: The four operations in one training step | The four operations in one training step | Repeat this sequence using the current parameters. | Predict | Forward pass | Measure error | Compute loss | Find | gradients | Backpropagation | Adjust | Optimizer | update | Calculate all gradients before replacing the old weights.

Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters. Tap the diagram to enlarge.

| Operation | Direction or action | Result |
| --- | --- | --- |
| Forward propagation | Inputs move through successive layers | Predictions and intermediate activations |
| Loss calculation | Compare predictions with targets | A scalar objective for the example or batch |
| Backpropagation | Apply the chain rule from the loss toward earlier layers | Gradients of the objective with respect to parameters |
| Optimizer update | Use those gradients to change parameters | A new set of weights and biases |

### 1. Forward propagation

At each layer, calculate its weighted sums and activations. Retain these values because derivatives such as the sigmoid derivative depend on them. Complete a whole layer before calculating the next.



$$
z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)},\qquad a^{(l)}=f(z^{(l)})
$$



### 2. Loss

Use the loss specified by the task. For one regression output, half squared loss is one possible choice:



$$
L=\frac12(a-y)^2
$$



Here a is the prediction and y is the target. Other stated losses produce different derivatives.

### 3. Backpropagation

Multiply local derivatives along each dependency path. If a hidden unit influences several outputs, add the contributions from those branches. For a connection, the weight gradient is the receiving neuron's net-input derivative multiplied by the sending activation.



$$
\frac{\partial L}{\partial w_{ji}}=\delta_j a_i
$$



Here δⱼ means the derivative of loss with respect to the receiving neuron's weighted sum. It is not automatically equal to raw prediction error; the activation and loss determine it.

### 4. Update and repeat



$$
w_{\mathrm{new}}=w_{\mathrm{old}}-\eta\frac{\partial L}{\partial w}
$$



Compute all gradients using the same pre-update weights. Then update the requested parameters. Run another forward pass to calculate the new loss. Backpropagation calculates gradients; gradient descent uses them. Those are related, separate operations.

## Backpropagation with one hidden neuron

**Definition.** Backpropagation computes the derivatives of a scalar objective with respect to network parameters by applying the chain rule in reverse through the computational graph. An optimizer then uses these derivatives to update the parameters.

### Network equations and the meaning of delta

Consider one input, one hidden neuron and one output neuron:



$$
z_h=w_1x+b_1,\qquad h=f(z_h)
$$





$$
z_o=w_2h+b_2,\qquad\hat y=g(z_o)
$$



For the half squared loss used in this example:



$$
L=\frac12(\hat y-y)^2
$$



Define a neuron's delta as the derivative of the loss with respect to its **preactivation**, not with respect to its output:



$$
\delta_o=\frac{\partial L}{\partial z_o}=(\hat y-y)g'(z_o)
$$



The output-weight derivative contains the hidden activation because zₒ = w₂h + b₂:



$$
\frac{\partial L}{\partial w_2}=\delta_o h,\qquad\frac{\partial L}{\partial b_2}=\delta_o
$$



To reach the hidden neuron, continue backwards through the old output weight and the hidden activation:



$$
\delta_h=\delta_o w_2 f'(z_h)
$$





$$
\frac{\partial L}{\partial w_1}=\delta_h x,\qquad\frac{\partial L}{\partial b_1}=\delta_h
$$



The bias derivative equals the corresponding delta because the derivative of z with respect to its additive bias is 1. All these derivatives belong to the same forward pass; an updated weight must not be used partway through this backward pass.

### Complete worked numerical

**Question.** x = 2, y = 1; w₁ = 0.5, b₁ = 0.1; w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1.

### Given

x = 2, y = 1.

w₁ = 0.5, b₁ = 0.1.

w₂ = 0.8, b₂ = 0.2.

Both neurons use ReLU.

Use half squared loss.

Find the output and all four parameter gradients, then update all four parameters using η = 0.1.

### To find

Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters.

### Formula — which one and why

**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.

x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.



$$
z_h=W_1x+b_1
$$





$$
h=f_1(z_h)
$$





$$
z_o=W_2h+b_2
$$





$$
o=f_2(z_o)
$$



**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.

**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.

oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.



$$
L=\frac12\sum_j(o_j-y_j)^2
$$





$$
\frac{\partial L}{\partial o_j}=o_j-y_j
$$



**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.

**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.

δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.



$$
\delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})
$$





$$
\delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}
$$





$$
\frac{\partial L}{\partial W_{ji}}=\delta_ja_i
$$





$$
\frac{\partial L}{\partial b_j}=\delta_j
$$





$$
W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}
$$



**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.

### Solution — step by step

![The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.](https://deep-learning-study-room.vercel.app/assets/figures/v_single_backprop.svg)

Diagram: The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.. Labels: Follow this one-hidden-neuron example in both directions | Follow this one-hidden-neuron example in both directions | x = 2, target = 1. Both activations are ReLU; both raw sums are positive. | 2 | INPUT | 1.1 | HIDDEN | 1.08 | OUTPUT | weight 0.5 | weight 0.8 | 0.5 × 2 + 0.1 = 1.1 | 0.8 × 1.1 + 0.2 = 1.08 | Output delta = (1.08 − 1) × 1 = 0.08 | Back through old weight 0.8 | Hidden delta = 0.08 × 0.8 × 1 = 0.064 | For a weight gradient, multiply its receiving delta by the value entering that | connection.

The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights. Tap the diagram to enlarge.

**Step 1: forward propagation through the hidden neuron.**



$$
z_h=0.5(2)+0.1=1.1
$$





$$
h=\operatorname{ReLU}(1.1)=1.1
$$



**Step 2: forward propagation through the output neuron.**



$$
z_o=0.8(1.1)+0.2=1.08
$$





$$
\hat y=\operatorname{ReLU}(1.08)=1.08
$$



ReLU is not bounded above by 1, so this output is valid. It is not being interpreted as a probability here.

**Step 3: calculate the loss and output delta.**



$$
L=\frac12(1.08-1)^2=0.0032
$$



Both preactivations are positive, so both ReLU derivatives are 1.



$$
\delta_o=(1.08-1)(1)=0.08
$$



**Step 4: calculate the output-layer parameter gradients.**



$$
\frac{\partial L}{\partial w_2}=0.08(1.1)=0.088
$$





$$
\frac{\partial L}{\partial b_2}=0.08
$$



**Step 5: propagate the derivative to the hidden neuron.**

Use w₂ = 0.8, its value during the forward pass:



$$
\delta_h=0.08(0.8)(1)=0.064
$$





$$
\frac{\partial L}{\partial w_1}=0.064(2)=0.128
$$





$$
\frac{\partial L}{\partial b_1}=0.064
$$



**Step 6: update each parameter using its own gradient.**



$$
w_{1,\mathrm{new}}=0.5-0.1(0.128)=0.4872
$$





$$
b_{1,\mathrm{new}}=0.1-0.1(0.064)=0.0936
$$





$$
w_{2,\mathrm{new}}=0.8-0.1(0.088)=0.7912
$$





$$
b_{2,\mathrm{new}}=0.2-0.1(0.08)=0.192
$$



| Parameter | Gradient from the original forward pass | Updated value |
| --- | --- | --- |
| w₁ | 0.128 | 0.4872 |
| b₁ | 0.064 | 0.0936 |
| w₂ | 0.088 | 0.7912 |
| b₂ | 0.08 | 0.192 |

**Final answer:** the original prediction is 1.08 and its loss is 0.0032. The table contains all requested gradients and updated parameters. A new forward pass would use all four new parameters together.

## Drill 11 Full ReLU backpropagation

## Question

**Question.** Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases.

![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_0601bfe757.svg)

Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.1 | 0.3 | −0.1 | W₂ | 0.4 | 0.5

Given matrices for this question Tap the diagram to enlarge.

![Worked example: all incoming weights and biases](https://deep-learning-study-room.vercel.app/assets/figures/net_drill13.svg)

Diagram: Worked example: all incoming weights and biases. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=2 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.2 + x2 × 0.1; bias 0.1 | h2 incoming: x1 × 0.3 + x2 × -0.1; bias 0.2 | o1 incoming: h1 × 0.4 + h2 × 0.5; bias 0.1 | Use these OLD connections for all gradients in the same pass.

Worked example: all incoming weights and biases Tap the diagram to enlarge.

## Answer — step by step

### Given

Input x = [1, 2].

W₁ is printed with the question.

b₁ = [0.1, 0.2].

W₂ is printed with the question.

b₂ = [0.1].

Hidden and output activations are ReLU.

Target y = [1].

Learning rate η = 0.1.

Use half squared loss and update all weights and biases.

### To find

Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters.

### Formula — which one and why

**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.

x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.



$$
z_h=W_1x+b_1
$$





$$
h=f_1(z_h)
$$





$$
z_o=W_2h+b_2
$$





$$
o=f_2(z_o)
$$



**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.

**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.

oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.



$$
L=\frac12\sum_j(o_j-y_j)^2
$$





$$
\frac{\partial L}{\partial o_j}=o_j-y_j
$$



**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.

**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.

δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.



$$
\delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})
$$





$$
\delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}
$$





$$
\frac{\partial L}{\partial W_{ji}}=\delta_ja_i
$$





$$
\frac{\partial L}{\partial b_j}=\delta_j
$$





$$
W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}
$$



**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.

### Solution — step by step

### Formulas used before substituting numbers

For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons.



$$
z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2
$$



Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path.



$$
\delta_{o_k}=(o_k-y_k)f'(z_{o_k})
$$





$$
\delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j})
$$



The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1.



$$
\frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j
$$





$$
w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j
$$



For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated.

**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.

**Hidden neuron 1:**



$$
z_{h_1} = (0.2 \times  1) + (0.1 \times  2) + (0.1) = 0.5
$$





$$
h_{1} = \max(0,0.5) = 0.5
$$



**Hidden neuron 2:**



$$
z_{h_2} = (0.3 \times  1) + (-0.1 \times  2) + (0.2) = 0.3
$$





$$
h_{2} = \max(0,0.3) = 0.3
$$



**Output neuron 1:**



$$
z_{o_1} = (0.4 \times  0.5) + (0.5 \times  0.3) + (0.1) = 0.45
$$





$$
o_{1} = \max(0,0.45) = 0.45
$$



**Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.

Prediction minus target = [-0.55]. Squared errors = [0.3025]. Half-sum squared loss = 0.15125; mean squared error = 0.3025. These are different unless the number of output components makes the scale coincide.

**Output delta 1:**



$$
\delta_{o_1} = 1 \times  (0.45-1) \times  1 = -0.55
$$



The ReLU derivative is 1 here because this preactivation is positive.

**Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta.

**h1 → o1:**

The calculation on h1 → o1

Sending activation**0.5**h1

×

Receiving delta**−0.55**δ at o1

=

Connection gradient**−0.275**h1 → o1

Now update this weight

Old weight**0.4**

→

New weight**0.4275**

Subtract η × gradient = **−0.0275**. This weight **increases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.



$$
g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1
$$





$$
g_{1,1}^{(2)}=(-0.55)\times(0.5)\approx -0.275
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(2)}_{1,1,\mathrm{new}}=0.4-0.1(-0.275)\approx 0.4275
$$



**h2 → o1:**

The calculation on h2 → o1

Sending activation**0.3**h2

×

Receiving delta**−0.55**δ at o1

=

Connection gradient**−0.165**h2 → o1

Now update this weight

Old weight**0.5**

→

New weight**0.5165**

Subtract η × gradient = **−0.0165**. This weight **increases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.



$$
g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2
$$





$$
g_{1,2}^{(2)}=(-0.55)\times(0.3)\approx -0.165
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(2)}_{1,2,\mathrm{new}}=0.5-0.1(-0.165)\approx 0.5165
$$



**Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.

**Hidden 1 downstream contribution:**

Calculate each outgoing path using the **old** outgoing weight and that output's delta.

Several output paths meet at this hidden neuron

Output 1

Output delta**−0.55**

×

Old outgoing weight**0.4**

=

Path contribution**−0.22**

↓

Add every path contribution**−0.22**Then multiply by the hidden activation derivative.

Multiply within each path. Add the resulting contributions at the shared hidden neuron.

**Contribution from output 1.**



$$
C_1=(0.4)\times(-0.55)\approx -0.22
$$



**Accumulate the contributions, then apply the hidden activation derivative below.**



$$
\text{Sum}=C_1\approx -0.22
$$





$$
\delta_{h_1} = -0.22 \times  1 = -0.22
$$



**Hidden 2 downstream contribution:**

Calculate each outgoing path using the **old** outgoing weight and that output's delta.

Several output paths meet at this hidden neuron

Output 1

Output delta**−0.55**

×

Old outgoing weight**0.5**

=

Path contribution**−0.275**

↓

Add every path contribution**−0.275**Then multiply by the hidden activation derivative.

Multiply within each path. Add the resulting contributions at the shared hidden neuron.

**Contribution from output 1.**



$$
C_1=(0.5)\times(-0.55)\approx -0.275
$$



**Accumulate the contributions, then apply the hidden activation derivative below.**



$$
\text{Sum}=C_1\approx -0.275
$$





$$
\delta_{h_2} = -0.275 \times  1 = -0.275
$$



**Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output.

**x1 → h1:**

The calculation on x1 → h1

Sending input**1**x1

×

Receiving delta**−0.22**δ at h1

=

Connection gradient**−0.22**x1 → h1

Now update this weight

Old weight**0.2**

→

New weight**0.222**

Subtract η × gradient = **−0.022**. This weight **increases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.



$$
g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1
$$





$$
g_{1,1}^{(1)}=(-0.22)\times(1)\approx -0.22
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(1)}_{1,1,\mathrm{new}}=0.2-0.1(-0.22)\approx 0.222
$$



**x2 → h1:**

The calculation on x2 → h1

Sending input**2**x2

×

Receiving delta**−0.22**δ at h1

=

Connection gradient**−0.44**x2 → h1

Now update this weight

Old weight**0.1**

→

New weight**0.144**

Subtract η × gradient = **−0.044**. This weight **increases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.



$$
g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2
$$





$$
g_{1,2}^{(1)}=(-0.22)\times(2)\approx -0.44
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(1)}_{1,2,\mathrm{new}}=0.1-0.1(-0.44)\approx 0.144
$$



**x1 → h2:**

The calculation on x1 → h2

Sending input**1**x1

×

Receiving delta**−0.275**δ at h2

=

Connection gradient**−0.275**x1 → h2

Now update this weight

Old weight**0.3**

→

New weight**0.3275**

Subtract η × gradient = **−0.0275**. This weight **increases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.



$$
g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1
$$





$$
g_{2,1}^{(1)}=(-0.275)\times(1)\approx -0.275
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(1)}_{2,1,\mathrm{new}}=0.3-0.1(-0.275)\approx 0.3275
$$



**x2 → h2:**

The calculation on x2 → h2

Sending input**2**x2

×

Receiving delta**−0.275**δ at h2

=

Connection gradient**−0.55**x2 → h2

Now update this weight

Old weight**−0.1**

→

New weight**−0.045**

Subtract η × gradient = **−0.055**. This weight **increases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.



$$
g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2
$$





$$
g_{2,2}^{(1)}=(-0.275)\times(2)\approx -0.55
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(1)}_{2,2,\mathrm{new}}=-0.1-0.1(-0.55)\approx -0.045
$$



**Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.



$$
b_{h_1,\mathrm{new}}=0.1-0.1(-0.220000000)=0.122000000
$$





$$
b_{h_2,\mathrm{new}}=0.2-0.1(-0.275000000)=0.227500000
$$





$$
b_{o_1,\mathrm{new}}=0.1-0.1(-0.550000000)=0.155000000
$$



These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.

## Drill 12 Two output sigmoid backpropagation

## Question

**Question.** Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters.

![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_907c03b5dc.svg)

Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.1 | 0.2 | −0.2 | 0.3 | W₂ | 0.2 | −0.1 | 0.3 | 0.4

Given matrices for this question Tap the diagram to enlarge.

![Worked example: both output paths must contribute to hidden gradients](https://deep-learning-study-room.vercel.app/assets/figures/net_drill14.svg)

Diagram: Worked example: both output paths must contribute to hidden gradients. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=0.2 | x2=0.4 | h1 | h2 | o1 | target 1 | o2 | target 0 | h1 incoming: x1 × 0.1 + x2 × 0.2; bias 0.1 | h2 incoming: x1 × -0.2 + x2 × 0.3; bias 0.1 | o1 incoming: h1 × 0.2 + h2 × -0.1; bias 0 | o2 incoming: h1 × 0.3 + h2 × 0.4; bias 0.1 | Use these OLD connections for all gradients in the same pass.

Worked example: both output paths must contribute to hidden gradients Tap the diagram to enlarge.

## Answer — step by step

### Given

Input x = [0.2, 0.4].

W₁ is printed with the question.

b₁ = [0.1, 0.1].

W₂ is printed with the question.

b₂ = [0, 0.1].

Hidden and output activations are sigmoid.

Target y = [1, 0].

Learning rate η = 0.2.

Use half summed squared loss and update all parameters.

### To find

Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.

### Formula — which one and why

**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.

x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.



$$
z_h=W_1x+b_1
$$





$$
h=f_1(z_h)
$$





$$
z_o=W_2h+b_2
$$





$$
o=f_2(z_o)
$$



**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.

**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.

oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.



$$
L=\frac12\sum_j(o_j-y_j)^2
$$





$$
\frac{\partial L}{\partial o_j}=o_j-y_j
$$



**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.

**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.



$$
p=\frac{1}{1+e^{-z}}
$$





$$
\frac{dp}{dz}=p(1-p)
$$



**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).

**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.

δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.



$$
\delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})
$$





$$
\delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}
$$





$$
\frac{\partial L}{\partial W_{ji}}=\delta_ja_i
$$





$$
\frac{\partial L}{\partial b_j}=\delta_j
$$





$$
W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}
$$



**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.

### Solution — step by step

### Formulas used before substituting numbers

For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons.



$$
z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2
$$



Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path.



$$
\delta_{o_k}=(o_k-y_k)f'(z_{o_k})
$$





$$
\delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j})
$$



The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1.



$$
\frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j
$$





$$
w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j
$$



For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated.

**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.

**Hidden neuron 1:**



$$
z_{h_1} = (0.1 \times  0.2) + (0.2 \times  0.4) + (0.1) = 0.2
$$





$$
h_{1}=\frac{1}{1+e^{-(0.2)}}=0.549833997
$$



**Hidden neuron 2:**



$$
z_{h_2} = (-0.2 \times  0.2) + (0.3 \times  0.4) + (0.1) = 0.18
$$





$$
h_{2}=\frac{1}{1+e^{-(0.18)}}=0.544878892
$$



**Output neuron 1:**



$$
z_{o_1} = (0.2 \times  0.549833997) + (-0.1 \times  0.544878892) + (0) = 0.05547891
$$





$$
o_{1}=\frac{1}{1+e^{-(0.05547891)}}=0.513866171
$$



**Output neuron 2:**



$$
z_{o_2} = (0.3 \times  0.549833997) + (0.4 \times  0.544878892) + (0.1) = 0.482901756
$$





$$
o_{2}=\frac{1}{1+e^{-(0.482901756)}}=0.618432848
$$



**Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.

Prediction minus target = [-0.486133829, 0.618432848]. Squared errors = [0.2363261, 0.382459187]. Half-sum squared loss = 0.309392643; mean squared error = 0.309392643. These are different unless the number of output components makes the scale coincide.

**Output delta 1:**



$$
\delta_{o_1} = 1 \times  (0.513866171-1) \times  0.249807729 = -0.121439988
$$



The derivative is o(1−o).

**Output delta 2:**



$$
\delta_{o_2} = 1 \times  (0.618432848-0) \times  0.235973661 = 0.145933863
$$



The derivative is o(1−o).

**Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta.

**h1 → o1:**

The calculation on h1 → o1

Sending activation**0.549833997**h1

×

Receiving delta**−0.121439988**δ at o1

=

Connection gradient**−0.066771834**h1 → o1

Now update this weight

Old weight**0.2**

→

New weight**0.213354367**

Subtract η × gradient = **−0.013354367**. This weight **increases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.



$$
g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1
$$





$$
g_{1,1}^{(2)}=(-0.121439988)\times(0.549833997)\approx -0.066771834
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(2)}_{1,1,\mathrm{new}}=0.2-0.2(-0.066771834)\approx 0.213354367
$$



**h2 → o1:**

The calculation on h2 → o1

Sending activation**0.544878892**h2

×

Receiving delta**−0.121439988**δ at o1

=

Connection gradient**−0.066170086**h2 → o1

Now update this weight

Old weight**−0.1**

→

New weight**−0.086765983**

Subtract η × gradient = **−0.013234017**. This weight **increases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.



$$
g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2
$$





$$
g_{1,2}^{(2)}=(-0.121439988)\times(0.544878892)\approx -0.066170086
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(2)}_{1,2,\mathrm{new}}=-0.1-0.2(-0.066170086)\approx -0.086765983
$$



**h1 → o2:**

The calculation on h1 → o2

Sending activation**0.549833997**h1

×

Receiving delta**0.145933863**δ at o2

=

Connection gradient**0.080239399**h1 → o2

Now update this weight

Old weight**0.3**

→

New weight**0.28395212**

Subtract η × gradient = **0.01604788**. This weight **decreases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.



$$
g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1
$$





$$
g_{2,1}^{(2)}=(0.145933863)\times(0.549833997)\approx 0.080239399
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(2)}_{2,1,\mathrm{new}}=0.3-0.2(0.080239399)\approx 0.28395212
$$



**h2 → o2:**

The calculation on h2 → o2

Sending activation**0.544878892**h2

×

Receiving delta**0.145933863**δ at o2

=

Connection gradient**0.079516282**h2 → o2

Now update this weight

Old weight**0.4**

→

New weight**0.384096744**

Subtract η × gradient = **0.015903256**. This weight **decreases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.



$$
g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2
$$





$$
g_{2,2}^{(2)}=(0.145933863)\times(0.544878892)\approx 0.079516282
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(2)}_{2,2,\mathrm{new}}=0.4-0.2(0.079516282)\approx 0.384096744
$$



**Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.

**Hidden 1 downstream contribution:**

Calculate each outgoing path using the **old** outgoing weight and that output's delta.

Several output paths meet at this hidden neuron

Output 1

Output delta**−0.121439988**

×

Old outgoing weight**0.2**

=

Path contribution**−0.024287998**

Output 2

Output delta**0.145933863**

×

Old outgoing weight**0.3**

=

Path contribution**0.043780159**

↓

Add every path contribution**0.019492161**Then multiply by the hidden activation derivative.

Multiply within each path. Add the resulting contributions at the shared hidden neuron.

**Contribution from output 1.**



$$
C_1=(0.2)\times(-0.121439988)\approx -0.024287998
$$



**Contribution from output 2.**



$$
C_2=(0.3)\times(0.145933863)\approx 0.043780159
$$



**Accumulate the contributions, then apply the hidden activation derivative below.**



$$
\text{Sum}=C_1+C_2\approx 0.019492161
$$





$$
\delta_{h_1} = 0.019492161 \times  0.247516573 = 0.004824633
$$



**Hidden 2 downstream contribution:**

Calculate each outgoing path using the **old** outgoing weight and that output's delta.

Several output paths meet at this hidden neuron

Output 1

Output delta**−0.121439988**

×

Old outgoing weight**−0.1**

=

Path contribution**0.012143999**

Output 2

Output delta**0.145933863**

×

Old outgoing weight**0.4**

=

Path contribution**0.058373545**

↓

Add every path contribution**0.070517544**Then multiply by the hidden activation derivative.

Multiply within each path. Add the resulting contributions at the shared hidden neuron.

**Contribution from output 1.**



$$
C_1=(-0.1)\times(-0.121439988)\approx 0.012143999
$$



**Contribution from output 2.**



$$
C_2=(0.4)\times(0.145933863)\approx 0.058373545
$$



**Accumulate the contributions, then apply the hidden activation derivative below.**



$$
\text{Sum}=C_1+C_2\approx 0.070517544
$$





$$
\delta_{h_2} = 0.070517544 \times  0.247985885 = 0.017487356
$$



**Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output.

**x1 → h1:**

The calculation on x1 → h1

Sending input**0.2**x1

×

Receiving delta**0.004824633**δ at h1

=

Connection gradient**0.000964927**x1 → h1

Now update this weight

Old weight**0.1**

→

New weight**0.099807015**

Subtract η × gradient = **0.000192985**. This weight **decreases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.



$$
g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1
$$





$$
g_{1,1}^{(1)}=(0.004824633)\times(0.2)\approx 0.000964927
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(1)}_{1,1,\mathrm{new}}=0.1-0.2(0.000964927)\approx 0.099807015
$$



**x2 → h1:**

The calculation on x2 → h1

Sending input**0.4**x2

×

Receiving delta**0.004824633**δ at h1

=

Connection gradient**0.001929853**x2 → h1

Now update this weight

Old weight**0.2**

→

New weight**0.199614029**

Subtract η × gradient = **0.000385971**. This weight **decreases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.



$$
g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2
$$





$$
g_{1,2}^{(1)}=(0.004824633)\times(0.4)\approx 0.001929853
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(1)}_{1,2,\mathrm{new}}=0.2-0.2(0.001929853)\approx 0.199614029
$$



**x1 → h2:**

The calculation on x1 → h2

Sending input**0.2**x1

×

Receiving delta**0.017487356**δ at h2

=

Connection gradient**0.003497471**x1 → h2

Now update this weight

Old weight**−0.2**

→

New weight**−0.200699494**

Subtract η × gradient = **0.000699494**. This weight **decreases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.



$$
g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1
$$





$$
g_{2,1}^{(1)}=(0.017487356)\times(0.2)\approx 0.003497471
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(1)}_{2,1,\mathrm{new}}=-0.2-0.2(0.003497471)\approx -0.200699494
$$



**x2 → h2:**

The calculation on x2 → h2

Sending input**0.4**x2

×

Receiving delta**0.017487356**δ at h2

=

Connection gradient**0.006994942**x2 → h2

Now update this weight

Old weight**0.3**

→

New weight**0.298601012**

Subtract η × gradient = **0.001398988**. This weight **decreases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.



$$
g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2
$$





$$
g_{2,2}^{(1)}=(0.017487356)\times(0.4)\approx 0.006994942
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(1)}_{2,2,\mathrm{new}}=0.3-0.2(0.006994942)\approx 0.298601012
$$



**Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.



$$
b_{h_1,\mathrm{new}}=0.1-0.2(0.004824633)=0.099035073
$$





$$
b_{h_2,\mathrm{new}}=0.1-0.2(0.017487356)=0.096502529
$$





$$
b_{o_1,\mathrm{new}}=0-0.2(-0.121439988)=0.024287998
$$





$$
b_{o_2,\mathrm{new}}=0.1-0.2(0.145933863)=0.070813227
$$



These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.