Deep Learning · MT2 · section

MT1A Q4

Prerequisite repair and teacher backprop examples

Question

Question (3 marks). Input x = [2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1]; W₂ is shown below; b₂ = [0.2]. Hidden and output activations are ReLU. Target y = [1]. Use L = ½(o−y)². Calculate the gradients of the output weight and output bias.

Answer — step by step

Given

Input x = [2].

W₁ is printed with the question.

b₁ = [0.1].

W₂ is printed with the question.

b₂ = [0.2].

Hidden and output activations are ReLU.

Target y = [1].

To find

Find all preactivations, hidden activations and final outputs. Find the requested deltas and parameter gradients, then update the specified parameters.

Formula — which one and why

Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.

x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.

$z_h=W_1x+b_1$ LaTeX: z_h=W_1x+b_1

$h=f_1(z_h)$ LaTeX: h=f_1(z_h)

$z_o=W_2h+b_2$ LaTeX: z_o=W_2h+b_2

$o=f_2(z_o)$ LaTeX: o=f_2(z_o)

Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.

Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.

δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.

$\delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})$ LaTeX: \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})

$\delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}$ LaTeX: \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}

$\frac{\partial L}{\partial W_{ji}}=\delta_ja_i$ LaTeX: \frac{\partial L}{\partial W_{ji}}=\delta_ja_i

$\frac{\partial L}{\partial b_j}=\delta_j$ LaTeX: \frac{\partial L}{\partial b_j}=\delta_j

$W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}$ LaTeX: W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}

Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.

Solution — step by step

Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.

Hidden neuron 1:

$h_{1} = \max(0,1.1) = 1.1$ LaTeX: h_{1} = \max(0,1.1) = 1.1

Output neuron 1:

$o_{1} = \max(0,1.08) = 1.08$ LaTeX: o_{1} = \max(0,1.08) = 1.08

Step 2 loss convention. The question specifies half squared loss. No learning rate is supplied, so report gradients, not invented weight updates.

Prediction minus target = [0.08]. Squared errors = [0.0064]. Half-sum squared loss = 0.0032; mean squared error = 0.0064. These are different unless the number of output components makes the scale coincide.

Output delta 1:

The ReLU derivative is 1 here because this preactivation is positive.

Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.

h1 → o1:

The calculation on h1 → o1

Sending activation1.1h1

Receiving delta0.08δ at o1

Connection gradient0.088h1 → o1

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.

$g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1$ LaTeX: g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1

Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.

Hidden 1 downstream contribution:

Calculate each outgoing path using the old outgoing weight and that output's delta.

Several output paths meet at this hidden neuron

Output 1

Output delta0.08

Old outgoing weight0.8

Path contribution0.064

Add every path contribution0.064Then multiply by the hidden activation derivative.

Multiply within each path. Add the resulting contributions at the shared hidden neuron.

Contribution from output 1.

Accumulate the contributions, then apply the hidden activation derivative below.

$\text{Sum}=C_1\approx 0.064$ LaTeX: \text{Sum}=C_1\approx 0.064

$\delta_{h_1} = 0.064 \times 1 = 0.064$ LaTeX: \delta_{h_1} = 0.064 \times 1 = 0.064

Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.

x1 → h1:

The calculation on x1 → h1

Sending input2x1

Receiving delta0.064δ at h1

Connection gradient0.128x1 → h1

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.

$g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1$ LaTeX: g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1

Requested answers: dL/dw₂ = 0.088 and dL/db₂ = 0.08. Hidden gradients above explain the dependency and are additional practice.