Deep Learning · TEE · section

F23 Q3a

Final examination December 2023 fully solved

Question

Question (10 marks). For x = [1,0,1], target 1, η = 0.9, use sigmoid at hidden and output layers. Weights w14 = .2, w15 = −.3, w24 = .4, w25 = .1, w34 = −.5, w35 = .2, w46 = −.3, w56 = −.2, biases at 4,5,6 are−.4,.2,.1. Update w46 and w56 using backpropagation.

F23 Q3a connections with the supplied biases Tap the diagram to enlarge.

Answer — step by step

Given

For x = [1,0,1], target 1, η = 0.9, use sigmoid at hidden and output layers.

Weights w14 = .2, w15 = −.3, w24 = .4, w25 = .1, w34 = −.5, w35 = .2, w46 = −.3, w56 = −.2, biases at 4,5,6 are−.4,.2,.1.

Update w46 and w56 using backpropagation.

To find

Find all preactivations, hidden activations and final outputs. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.

Formula — which one and why

Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.

x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.

$z_h=W_1x+b_1$ LaTeX: z_h=W_1x+b_1

$h=f_1(z_h)$ LaTeX: h=f_1(z_h)

$z_o=W_2h+b_2$ LaTeX: z_o=W_2h+b_2

$o=f_2(z_o)$ LaTeX: o=f_2(z_o)

Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.

Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.

$p=\frac{1}{1+e^{-z}}$ LaTeX: p=\frac{1}{1+e^{-z}}

$\frac{dp}{dz}=p(1-p)$ LaTeX: \frac{dp}{dz}=p(1-p)

Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).

Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.

δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.

$\delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})$ LaTeX: \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})

$\delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}$ LaTeX: \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}

$\frac{\partial L}{\partial W_{ji}}=\delta_ja_i$ LaTeX: \frac{\partial L}{\partial W_{ji}}=\delta_ja_i

$\frac{\partial L}{\partial b_j}=\delta_j$ LaTeX: \frac{\partial L}{\partial b_j}=\delta_j

$W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}$ LaTeX: W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}

Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.

Solution — step by step

Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.

Hidden neuron 1:

Hidden neuron 2:

Output neuron 1:

Step 2 loss convention. This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.

Prediction minus target = [-0.526111101]. Squared errors = [0.276792891]. Half-sum squared loss = 0.138396445; mean squared error = 0.276792891. These are different unless the number of output components makes the scale coincide.

Output delta 1:

The derivative is o(1−o).

Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.

h1 → o1:

The calculation on h1 → o1

Sending activation0.331812228h1

Receiving delta−0.131169078δ at o1

Connection gradient−0.043523504h1 → o1

Now update this weight

Old weight−0.3

New weight−0.260828846

Subtract η × gradient = −0.039171154. This weight increases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.

$g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1$ LaTeX: g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1

Apply the learning rate to this gradient and subtract the result from the old weight.

h2 → o1:

The calculation on h2 → o1

Sending activation0.524979187h2

Receiving delta−0.131169078δ at o1

Connection gradient−0.068861036h2 → o1

Now update this weight

Old weight−0.2

New weight−0.138025068

Subtract η × gradient = −0.061974932. This weight increases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.

$g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2$ LaTeX: g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2

Apply the learning rate to this gradient and subtract the result from the old weight.

Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.

Hidden 1 downstream contribution:

Calculate each outgoing path using the old outgoing weight and that output's delta.

Several output paths meet at this hidden neuron

Output 1

Output delta−0.131169078

Old outgoing weight−0.3

Path contribution0.039350723

Add every path contribution0.039350723Then multiply by the hidden activation derivative.

Multiply within each path. Add the resulting contributions at the shared hidden neuron.

Contribution from output 1.

Accumulate the contributions, then apply the hidden activation derivative below.

$\text{Sum}=C_1\approx 0.039350723$ LaTeX: \text{Sum}=C_1\approx 0.039350723

Hidden 2 downstream contribution:

Calculate each outgoing path using the old outgoing weight and that output's delta.

Several output paths meet at this hidden neuron

Output 1

Output delta−0.131169078

Old outgoing weight−0.2

Path contribution0.026233816

Add every path contribution0.026233816Then multiply by the hidden activation derivative.

Multiply within each path. Add the resulting contributions at the shared hidden neuron.

Contribution from output 1.

Accumulate the contributions, then apply the hidden activation derivative below.

$\text{Sum}=C_1\approx 0.026233816$ LaTeX: \text{Sum}=C_1\approx 0.026233816

Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.

x1 → h1:

The calculation on x1 → h1

Sending input1x1

Receiving delta0.008724562δ at h1

Connection gradient0.008724562x1 → h1

Now update this weight

Old weight0.2

New weight0.192147894

Subtract η × gradient = 0.007852106. This weight decreases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.

$g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1$ LaTeX: g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1

Apply the learning rate to this gradient and subtract the result from the old weight.

x2 → h1:

The calculation on x2 → h1

Sending input0x2

Receiving delta0.008724562δ at h1

Connection gradient0x2 → h1

Now update this weight

Old weight0.4

New weight0.4

Subtract η × gradient = 0. This weight stays unchanged.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.

$g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2$ LaTeX: g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2

Apply the learning rate to this gradient and subtract the result from the old weight.

x3 → h1:

The calculation on x3 → h1

Sending input1x3

Receiving delta0.008724562δ at h1

Connection gradient0.008724562x3 → h1

Now update this weight

Old weight−0.5

New weight−0.507852106

Subtract η × gradient = 0.007852106. This weight decreases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.

$g_{1,3}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,3}}=\delta_{h_1}x_3$ LaTeX: g_{1,3}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,3}}=\delta_{h_1}x_3

Apply the learning rate to this gradient and subtract the result from the old weight.

x1 → h2:

The calculation on x1 → h2

Sending input1x1

Receiving delta0.006542085δ at h2

Connection gradient0.006542085x1 → h2

Now update this weight

Old weight−0.3

New weight−0.305887877

Subtract η × gradient = 0.005887876. This weight decreases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.

$g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1$ LaTeX: g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1

Apply the learning rate to this gradient and subtract the result from the old weight.

x2 → h2:

The calculation on x2 → h2

Sending input0x2

Receiving delta0.006542085δ at h2

Connection gradient0x2 → h2

Now update this weight

Old weight0.1

New weight0.1

Subtract η × gradient = 0. This weight stays unchanged.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.

$g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2$ LaTeX: g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2

Apply the learning rate to this gradient and subtract the result from the old weight.

x3 → h2:

The calculation on x3 → h2

Sending input1x3

Receiving delta0.006542085δ at h2

Connection gradient0.006542085x3 → h2

Now update this weight

Old weight0.2

New weight0.194112123

Subtract η × gradient = 0.005887876. This weight decreases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.

$g_{2,3}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,3}}=\delta_{h_2}x_3$ LaTeX: g_{2,3}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,3}}=\delta_{h_2}x_3

Apply the learning rate to this gradient and subtract the result from the old weight.

Step 6 bias updates. A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.

These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.

Required: w46 = −0.260828846, w56 = −0.138025068 under half squared loss. If a one-output unhalved MSE is explicitly required, double the shown gradients before updating; the paper itself does not state that scaling.