# MT1A Q4

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/read/10/mt2-c10-s2

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c10-s2/

## Question

**Question (3 marks).** Input x = [2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1]; W₂ is shown below; b₂ = [0.2]. Hidden and output activations are ReLU. Target y = [1]. Use L = ½(o−y)². Calculate the gradients of the output weight and output bias.

![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_bbb5f79ed7.svg)

Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.5 | W₂ | 0.8

Given matrices for this question Tap the diagram to enlarge.

## Answer — step by step

### Given

Input x = [2].

W₁ is printed with the question.

b₁ = [0.1].

W₂ is printed with the question.

b₂ = [0.2].

Hidden and output activations are ReLU.

Target y = [1].

### To find

Find all preactivations, hidden activations and final outputs. Find the requested deltas and parameter gradients, then update the specified parameters.

### Formula — which one and why

**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.

x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.



$$
z_h=W_1x+b_1
$$





$$
h=f_1(z_h)
$$





$$
z_o=W_2h+b_2
$$





$$
o=f_2(z_o)
$$



**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.

**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.

δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.



$$
\delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})
$$





$$
\delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}
$$





$$
\frac{\partial L}{\partial W_{ji}}=\delta_ja_i
$$





$$
\frac{\partial L}{\partial b_j}=\delta_j
$$





$$
W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}
$$



**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.

### Solution — step by step

**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.

**Hidden neuron 1:**



$$
z_{h_1} = (0.5 \times  2) + (0.1) = 1.1
$$





$$
h_{1} = \max(0,1.1) = 1.1
$$



**Output neuron 1:**



$$
z_{o_1} = (0.8 \times  1.1) + (0.2) = 1.08
$$





$$
o_{1} = \max(0,1.08) = 1.08
$$



**Step 2 loss convention.** The question specifies half squared loss. No learning rate is supplied, so report gradients, not invented weight updates.

Prediction minus target = [0.08]. Squared errors = [0.0064]. Half-sum squared loss = 0.0032; mean squared error = 0.0064. These are different unless the number of output components makes the scale coincide.

**Output delta 1:**



$$
\delta_{o_1} = 1 \times  (1.08-1) \times  1 = 0.08
$$



The ReLU derivative is 1 here because this preactivation is positive.

**Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta.

**h1 → o1:**

The calculation on h1 → o1

Sending activation**1.1**h1

×

Receiving delta**0.08**δ at o1

=

Connection gradient**0.088**h1 → o1

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.



$$
g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1
$$





$$
g_{1,1}^{(2)}=(0.08)\times(1.1)\approx 0.088
$$



**Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.

**Hidden 1 downstream contribution:**

Calculate each outgoing path using the **old** outgoing weight and that output's delta.

Several output paths meet at this hidden neuron

Output 1

Output delta**0.08**

×

Old outgoing weight**0.8**

=

Path contribution**0.064**

↓

Add every path contribution**0.064**Then multiply by the hidden activation derivative.

Multiply within each path. Add the resulting contributions at the shared hidden neuron.

**Contribution from output 1.**



$$
C_1=(0.8)\times(0.08)\approx 0.064
$$



**Accumulate the contributions, then apply the hidden activation derivative below.**



$$
\text{Sum}=C_1\approx 0.064
$$





$$
\delta_{h_1} = 0.064 \times  1 = 0.064
$$



**Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output.

**x1 → h1:**

The calculation on x1 → h1

Sending input**2**x1

×

Receiving delta**0.064**δ at h1

=

Connection gradient**0.128**x1 → h1

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.



$$
g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1
$$





$$
g_{1,1}^{(1)}=(0.064)\times(2)\approx 0.128
$$



**Requested answers:**  dL/dw₂ = **0.088** and dL/db₂ = **0.08**. Hidden gradients above explain the dependency and are additional practice.