# F23 Q3a

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/13/tee-c13-s6

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s6/

## Question

**Question (10 marks).** For x = [1,0,1], target 1, η = 0.9, use sigmoid at hidden and output layers. Weights w14 = .2, w15 = −.3, w24 = .4, w25 = .1, w34 = −.5, w35 = .2, w46 = −.3, w56 = −.2, biases at 4,5,6 are−.4,.2,.1. Update w46 and w56 using backpropagation.

![F23 Q3a connections with the supplied biases](https://deep-learning-study-room.vercel.app/assets/figures/net_f23.svg)

Diagram: F23 Q3a connections with the supplied biases. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=0 | x3=1 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.2 + x2 × 0.4 + x3 × -0.5; bias -0.4 | h2 incoming: x1 × -0.3 + x2 × 0.1 + x3 × 0.2; bias 0.2 | o1 incoming: h1 × -0.3 + h2 × -0.2; bias 0.1 | Use these OLD connections for all gradients in the same pass.

F23 Q3a connections with the supplied biases Tap the diagram to enlarge.

## Answer — step by step

### Given

For x = [1,0,1], target 1, η = 0.9, use sigmoid at hidden and output layers.

Weights w14 = .2, w15 = −.3, w24 = .4, w25 = .1, w34 = −.5, w35 = .2, w46 = −.3, w56 = −.2, biases at 4,5,6 are−.4,.2,.1.

Update w46 and w56 using backpropagation.

### To find

Find all preactivations, hidden activations and final outputs. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.

### Formula — which one and why

**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.

x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.



$$
z_h=W_1x+b_1
$$





$$
h=f_1(z_h)
$$





$$
z_o=W_2h+b_2
$$





$$
o=f_2(z_o)
$$



**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.

**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.



$$
p=\frac{1}{1+e^{-z}}
$$





$$
\frac{dp}{dz}=p(1-p)
$$



**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).

**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.

δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.



$$
\delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})
$$





$$
\delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}
$$





$$
\frac{\partial L}{\partial W_{ji}}=\delta_ja_i
$$





$$
\frac{\partial L}{\partial b_j}=\delta_j
$$





$$
W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}
$$



**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.

### Solution — step by step

**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.

**Hidden neuron 1:**



$$
z_{h_1} = (0.2 \times  1) + (0.4 \times  0) + (-0.5 \times  1) + (-0.4) = -0.7
$$





$$
h_{1}=\frac{1}{1+e^{-(-0.7)}}=0.331812228
$$



**Hidden neuron 2:**



$$
z_{h_2} = (-0.3 \times  1) + (0.1 \times  0) + (0.2 \times  1) + (0.2) = 0.1
$$





$$
h_{2}=\frac{1}{1+e^{-(0.1)}}=0.524979187
$$



**Output neuron 1:**



$$
z_{o_1} = (-0.3 \times  0.331812228) + (-0.2 \times  0.524979187) + (0.1) = -0.104539506
$$





$$
o_{1}=\frac{1}{1+e^{-(-0.104539506)}}=0.473888899
$$



**Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.

Prediction minus target = [-0.526111101]. Squared errors = [0.276792891]. Half-sum squared loss = 0.138396445; mean squared error = 0.276792891. These are different unless the number of output components makes the scale coincide.

**Output delta 1:**



$$
\delta_{o_1} = 1 \times  (0.473888899-1) \times  0.24931821 = -0.131169078
$$



The derivative is o(1−o).

**Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta.

**h1 → o1:**

The calculation on h1 → o1

Sending activation**0.331812228**h1

×

Receiving delta**−0.131169078**δ at o1

=

Connection gradient**−0.043523504**h1 → o1

Now update this weight

Old weight**−0.3**

→

New weight**−0.260828846**

Subtract η × gradient = **−0.039171154**. This weight **increases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.



$$
g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1
$$





$$
g_{1,1}^{(2)}=(-0.131169078)\times(0.331812228)\approx -0.043523504
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(2)}_{1,1,\mathrm{new}}=-0.3-0.9(-0.043523504)\approx -0.260828846
$$



**h2 → o1:**

The calculation on h2 → o1

Sending activation**0.524979187**h2

×

Receiving delta**−0.131169078**δ at o1

=

Connection gradient**−0.068861036**h2 → o1

Now update this weight

Old weight**−0.2**

→

New weight**−0.138025068**

Subtract η × gradient = **−0.061974932**. This weight **increases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.



$$
g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2
$$





$$
g_{1,2}^{(2)}=(-0.131169078)\times(0.524979187)\approx -0.068861036
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(2)}_{1,2,\mathrm{new}}=-0.2-0.9(-0.068861036)\approx -0.138025068
$$



**Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.

**Hidden 1 downstream contribution:**

Calculate each outgoing path using the **old** outgoing weight and that output's delta.

Several output paths meet at this hidden neuron

Output 1

Output delta**−0.131169078**

×

Old outgoing weight**−0.3**

=

Path contribution**0.039350723**

↓

Add every path contribution**0.039350723**Then multiply by the hidden activation derivative.

Multiply within each path. Add the resulting contributions at the shared hidden neuron.

**Contribution from output 1.**



$$
C_1=(-0.3)\times(-0.131169078)\approx 0.039350723
$$



**Accumulate the contributions, then apply the hidden activation derivative below.**



$$
\text{Sum}=C_1\approx 0.039350723
$$





$$
\delta_{h_1} = 0.039350723 \times  0.221712873 = 0.008724562
$$



**Hidden 2 downstream contribution:**

Calculate each outgoing path using the **old** outgoing weight and that output's delta.

Several output paths meet at this hidden neuron

Output 1

Output delta**−0.131169078**

×

Old outgoing weight**−0.2**

=

Path contribution**0.026233816**

↓

Add every path contribution**0.026233816**Then multiply by the hidden activation derivative.

Multiply within each path. Add the resulting contributions at the shared hidden neuron.

**Contribution from output 1.**



$$
C_1=(-0.2)\times(-0.131169078)\approx 0.026233816
$$



**Accumulate the contributions, then apply the hidden activation derivative below.**



$$
\text{Sum}=C_1\approx 0.026233816
$$





$$
\delta_{h_2} = 0.026233816 \times  0.24937604 = 0.006542085
$$



**Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output.

**x1 → h1:**

The calculation on x1 → h1

Sending input**1**x1

×

Receiving delta**0.008724562**δ at h1

=

Connection gradient**0.008724562**x1 → h1

Now update this weight

Old weight**0.2**

→

New weight**0.192147894**

Subtract η × gradient = **0.007852106**. This weight **decreases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.



$$
g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1
$$





$$
g_{1,1}^{(1)}=(0.008724562)\times(1)\approx 0.008724562
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(1)}_{1,1,\mathrm{new}}=0.2-0.9(0.008724562)\approx 0.192147894
$$



**x2 → h1:**

The calculation on x2 → h1

Sending input**0**x2

×

Receiving delta**0.008724562**δ at h1

=

Connection gradient**0**x2 → h1

Now update this weight

Old weight**0.4**

→

New weight**0.4**

Subtract η × gradient = **0**. This weight **stays unchanged**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.



$$
g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2
$$





$$
g_{1,2}^{(1)}=(0.008724562)\times(0)\approx 0
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(1)}_{1,2,\mathrm{new}}=0.4-0.9(0)\approx 0.4
$$



**x3 → h1:**

The calculation on x3 → h1

Sending input**1**x3

×

Receiving delta**0.008724562**δ at h1

=

Connection gradient**0.008724562**x3 → h1

Now update this weight

Old weight**−0.5**

→

New weight**−0.507852106**

Subtract η × gradient = **0.007852106**. This weight **decreases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.



$$
g_{1,3}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,3}}=\delta_{h_1}x_3
$$





$$
g_{1,3}^{(1)}=(0.008724562)\times(1)\approx 0.008724562
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(1)}_{1,3,\mathrm{new}}=-0.5-0.9(0.008724562)\approx -0.507852106
$$



**x1 → h2:**

The calculation on x1 → h2

Sending input**1**x1

×

Receiving delta**0.006542085**δ at h2

=

Connection gradient**0.006542085**x1 → h2

Now update this weight

Old weight**−0.3**

→

New weight**−0.305887877**

Subtract η × gradient = **0.005887876**. This weight **decreases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.



$$
g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1
$$





$$
g_{2,1}^{(1)}=(0.006542085)\times(1)\approx 0.006542085
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(1)}_{2,1,\mathrm{new}}=-0.3-0.9(0.006542085)\approx -0.305887877
$$



**x2 → h2:**

The calculation on x2 → h2

Sending input**0**x2

×

Receiving delta**0.006542085**δ at h2

=

Connection gradient**0**x2 → h2

Now update this weight

Old weight**0.1**

→

New weight**0.1**

Subtract η × gradient = **0**. This weight **stays unchanged**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.



$$
g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2
$$





$$
g_{2,2}^{(1)}=(0.006542085)\times(0)\approx 0
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(1)}_{2,2,\mathrm{new}}=0.1-0.9(0)\approx 0.1
$$



**x3 → h2:**

The calculation on x3 → h2

Sending input**1**x3

×

Receiving delta**0.006542085**δ at h2

=

Connection gradient**0.006542085**x3 → h2

Now update this weight

Old weight**0.2**

→

New weight**0.194112123**

Subtract η × gradient = **0.005887876**. This weight **decreases**.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.



$$
g_{2,3}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,3}}=\delta_{h_2}x_3
$$





$$
g_{2,3}^{(1)}=(0.006542085)\times(1)\approx 0.006542085
$$



Apply the learning rate to this gradient and subtract the result from the old weight.



$$
W^{(1)}_{2,3,\mathrm{new}}=0.2-0.9(0.006542085)\approx 0.194112123
$$



**Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.



$$
b_{h_1,\mathrm{new}}=-0.4-0.9(0.008724562)=-0.407852106
$$





$$
b_{h_2,\mathrm{new}}=0.2-0.9(0.006542085)=0.194112124
$$





$$
b_{o_1,\mathrm{new}}=0.1-0.9(-0.131169078)=0.218052170
$$



These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.

**Required:**  w46 = **−0.260828846**, w56 = **−0.138025068** under half squared loss. If a one-output unhalved MSE is explicitly required, double the shown gradients before updating; the paper itself does not state that scaling.