# Chain rule

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/4

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/4/

The chain rule differentiates a composite function. Multiply local derivatives along a dependency path and add the contributions of different paths to the same variable.



$$
\frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw}
$$



## The chain rule is multiplying linked effects

**Definition.** The chain rule finds a rate of change when one quantity depends on another through intermediate calculations. Multiply the local rates along one path. If several paths connect the same starting quantity to the final answer, add their complete contributions.

### First understand why the rates multiply

Suppose w controls u and u controls v. If each one-unit change in w makes u change by 3 units, and each one-unit change in u makes v change by 2 units, the combined rate is 3 × 2 = 6. Adding 3+2 would describe neither dependency.

The notation du/dw reads “the rate at which u changes when w changes”. A partial derivative ∂L/∂w means the same kind of rate while other independent parameters are held fixed. It is not ordinary division of the values L and w.

**Yaad rakhne ka cue:** ek raaste par slopes multiply; alag raaston ke full answers add.

### Chain-rule example 1: two linear links

**Question. Let u = 3w+1 and v = 2u. At w = 1, find u, v and dv/dw.**

### Given

w = 1; u = 3w+1; v = 2u.

### To find

The two function values and the rate dv/dw.

### Formula — which one and why

v depends on w through u. Both links must be included, so multiply the two local derivatives.

du/dw is the slope of 3w+1. dv/du is the slope of 2u.



$$
\frac{dv}{dw}=\frac{dv}{du}\frac{du}{dw}
$$



**Yaad rakhne ka cue:** Andar ka slope × bahar ka slope.

### Solution — step by step

**Step 1: calculate the intermediate value.**



$$
u=3(1)+1=4
$$



**Step 2: calculate the final value.**



$$
v=2(4)=8
$$



**Step 3: differentiate the first link.**



$$
\frac{du}{dw}=3
$$



**Step 4: differentiate the second link.**



$$
\frac{dv}{du}=2
$$



**Step 5: multiply along the path.**



$$
\frac{dv}{dw}=2(3)=6
$$



**Final answer:** u = 4, v = 8 and dv/dw = 6. The function value 8 and its rate 6 are different.

### 2. One output: calculate both parameter gradients

**Question.** A linear neuron has z = wx + b and half squared loss L = ½(z − y)². Given x = 2, y = 4, w = 1 and b = 0, obtain both parameter gradients and perform one gradient-descent update with η = 0.1.

### Given

Input x = 2; target y = 4.

Old weight w = 1; old bias b = 0.

Linear prediction z = wx+b; loss L = ½(z−y)².

Learning rate η = 0.1; update both parameters once.

### To find

Find the loss gradient for w and for b, then w\_new and b\_new. Finally evaluate the changed prediction and loss.

### Formula — which one and why

**Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x.

z = prediction; y = target; e = z−y; g\_w and g\_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate.



$$
z=wx+b
$$





$$
L=\frac12(z-y)^2
$$





$$
g_w=\frac{\partial L}{\partial w}=(z-y)x
$$





$$
g_b=\frac{\partial L}{\partial b}=z-y
$$





$$
w_{\mathrm{new}}=w-\eta g_w
$$





$$
b_{\mathrm{new}}=b-\eta g_b
$$



**Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add.

### Solution — step by step

![The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient.](https://deep-learning-study-room.vercel.app/assets/figures/chain_linear_revision.svg)

Diagram: The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient.. Labels: One output: values and derivatives | One output: values and derivatives | Given: x = 2, y = 4, w = 1, b = 0 | FORWARD VALUES | z = wx + b | z = 2 | e = z − y | e = −2 | L = ½e² | L = 2 | BACKWARD DERIVATIVES | Loss derivative reaches the neuron | ∂L/∂z = (∂L/∂e)(∂e/∂z) = (−2)(1) = −2 | Weight: multiply by x = 2 | ∂L/∂w = (−2)(2) = −4 | Bias: multiply by 1 | ∂L/∂b = (−2)(1) = −2 | Then update: w → 1.4 and b → 0.2 (η = 0.1).

The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient. Tap the diagram to enlarge.

**Step 1 — evaluate the original model.** Use the given parameters before applying any update.



$$
z=wx+b=1(2)+0=2
$$





$$
e=z-y=2-4=-2
$$





$$
L=\frac12e^2=\frac12(-2)^2=2
$$



Forward pass: three distinct quantities

Prediction z**2**1 × 2 + 0

→

Error e**−2**Prediction 2 − target 4

→

Half squared loss**2**½ × (−2)²

The prediction is 2, the signed error is −2 and the loss is 2. These are three different quantities, even when some numerical values happen to coincide.

**Step 2 — differentiate the loss, keeping the target constant.** First differentiate the square, then the error inside it.



$$
\frac{\partial L}{\partial e}=\frac12(2e)=e
$$





$$
\frac{\partial e}{\partial z}=\frac{\partial(z-y)}{\partial z}=1
$$





$$
\delta_z=\frac{\partial L}{\partial z}=e(1)=-2
$$



The symbol δ (delta) is a convenient name for the derivative with respect to a neuron's preactivation. Delta is not automatically the weight gradient; the weight gradient also depends on the input to that connection.

**Step 3 — differentiate the neuron's affine expression.** When differentiating with respect to w, x is a constant coefficient. When differentiating with respect to b, the coefficient of b is 1.



$$
\frac{\partial z}{\partial w}=x=2
$$





$$
\frac{\partial z}{\partial b}=1
$$



**Step 4 — multiply along each parameter's path.**



$$
g_w=\frac{\partial L}{\partial w}=\delta_z\frac{\partial z}{\partial w}=(-2)(2)=-4
$$





$$
g_b=\frac{\partial L}{\partial b}=\delta_z\frac{\partial z}{\partial b}=(-2)(1)=-2
$$



Backward pass: a separate derivative for each parameter

Shared loss derivative δz**−2**Both parameter paths start here.

Weight path: multiply by x = 2**−4**∂L/∂w = (−2) × 2

Bias path: multiply by 1**−2**∂L/∂b = (−2) × 1

Both derivatives use the original forward-pass values. The weight and bias are different parameters, so keep their gradients separate.

**Step 5 — update both parameters simultaneously.** Gradient descent subtracts the learning rate times the corresponding gradient.



$$
w_{\mathrm{new}}=w-\eta g_w=1-0.1(-4)=1.4
$$





$$
b_{\mathrm{new}}=b-\eta g_b=0-0.1(-2)=0.2
$$



One simultaneous parameter update

Weight**1 → 1.4**Subtract 0.1 × (−4)

Bias**0 → 0.2**Subtract 0.1 × (−2)

Both updates use gradients from the same original forward pass. Re-evaluate the model afterward.

Subtracting a negative gradient increases the parameter. Do not recompute the bias gradient using the already changed weight; that would be a different update procedure.

**Step 6 — perform a new forward pass to evaluate the update.**



$$
z_{\mathrm{new}}=1.4(2)+0.2=3
$$





$$
L_{\mathrm{new}}=\frac12(3-4)^2=0.5
$$



**Final answer:** weight gradient −4, bias gradient −2; updated weight 1.4 and bias 0.2; new prediction 3 and loss 0.5. If a different question fixes the bias at zero, update only the weight; its new prediction would be 2.8, not 3.

### 3. Two outputs: why backward contributions are added

A shared intermediate value can influence more than one output. The derivative with respect to that shared value must include every output path. Adding paths happens before passing the accumulated derivative further backward.

**Question.** A shared linear hidden value is h = wx + b. Two linear outputs are o₁ = 2h and o₂ = −h; their targets are y₁ = 1 and y₂ = 0. The total loss is L = ½(o₁ − y₁)² + ½(o₂ − y₂)². At x = 2, w = 0.5 and b = 0.1, calculate the loss and the gradients with respect to h, w and b. Treat the output multipliers 2 and −1 as fixed.

### Given

h = wx+b; outputs o₁ = 2h and o₂ = −h.

x = 2; w = 0.5; b = 0.1; targets = [1,0].

Loss = half the sum of the two squared output errors.

Output multipliers 2 and −1 stay fixed.

### To find

Find total loss and its derivatives with respect to h, w and b.

### Formula — which one and why

**Chain rule with more than one path.** The shared hidden value affects each output. Multiply derivatives within each path, then add the complete path contributions before continuing to w and b.

oⱼ = output j; yⱼ = its target; h = shared hidden value; ∂oⱼ/∂h is the multiplier on that output path.



$$
L=\frac12\sum_j(o_j-y_j)^2
$$





$$
\frac{\partial L}{\partial h}=\sum_j(o_j-y_j)\frac{\partial o_j}{\partial h}
$$





$$
\frac{\partial L}{\partial w}=\frac{\partial L}{\partial h}x
$$





$$
\frac{\partial L}{\partial b}=\frac{\partial L}{\partial h}
$$



**Yaad rakhne ka cue:** Har raaste ka effect pehle multiply, junction par sab add.

### Solution — step by step

![The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias.](https://deep-learning-study-room.vercel.app/assets/figures/chain_branch_revision.svg)

Diagram: The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias.. Labels: Two outputs: add complete path contributions | Two outputs: add complete path contributions | h = wx + b = 0.5(2) + 0.1 = 1.1 | Shared hidden value | h = 1.1 | o₁ = 2h; target y₁ = 1 | o₁ = 2.2 | o₂ = −h; target y₂ = 0 | o₂ = −1.1 | × 2 | × (−1) | Output derivative: 2.2 − 1 = 1.2 | Output derivative: −1.1 − 0 = −1.1 | BACK TO h | Path 1: (1.2)(2) = 2.4 | Path 2: (−1.1)(−1) = 1.1 | Add at the shared value, then continue backward | ∂L/∂h = 2.4 + 1.1 = 3.5 | ∂L/∂w = 3.5 × 2 = 7 | ∂L/∂b = 3.5 × 1 = 3.5

The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias. Tap the diagram to enlarge.

**Step 1 — forward values.**



$$
h=0.5(2)+0.1=1.1
$$





$$
o_1=2(1.1)=2.2,\qquad o_2=-(1.1)=-1.1
$$





$$
L_1=\frac12(2.2-1)^2=0.72
$$





$$
L_2=\frac12(-1.1-0)^2=0.605
$$





$$
L=L_1+L_2=0.72+0.605=1.325
$$



**Step 2 — output derivatives.** Each half squared loss differentiates to prediction minus target. The other loss does not depend on this output.



$$
\frac{\partial L}{\partial o_1}=o_1-y_1=2.2-1=1.2
$$





$$
\frac{\partial L}{\partial o_2}=o_2-y_2=-1.1-0=-1.1
$$



**Step 3 — calculate the contribution of each complete path.** The derivative of 2h with respect to h is 2; the derivative of −h is −1.



$$
\frac{\partial L_1}{\partial h}=(1.2)(2)=2.4
$$





$$
\frac{\partial L_2}{\partial h}=(-1.1)(-1)=1.1
$$



**Step 4 — add the two contributions at the shared hidden value.**



$$
\frac{\partial L}{\partial h}=\frac{\partial L_1}{\partial h}+\frac{\partial L_2}{\partial h}
$$





$$
\frac{\partial L}{\partial h}=2.4+1.1=3.5
$$



Adding the raw output errors, 1.2 + (−1.1), would omit the output multipliers. Each error must first be multiplied by the local derivative on its own path.

**Step 5 — continue to the hidden weight and bias.** Since h = wx + b, its derivatives with respect to w and b are x and 1.



$$
\frac{\partial L}{\partial w}=\frac{\partial L}{\partial h}x=3.5(2)=7
$$





$$
\frac{\partial L}{\partial b}=\frac{\partial L}{\partial h}(1)=3.5
$$



**Final answer:** L = 1.325, ∂L/∂h = 3.5, ∂L/∂w = 7 and ∂L/∂b = 3.5. If the hidden value instead includes a nonlinear activation, multiply the accumulated downstream derivative by that activation's derivative before multiplying by x. This is the hidden-delta calculation in multilayer backpropagation.

### Chain-rule example 4: a square after a linear link

**Question. Let z = 3w+1 and L = z². At w = 1, find z, L and dL/dw.**

### Given

w = 1; z = 3w+1; L = z².

### To find

z, L and the derivative of L with respect to w.

### Formula — which one and why

The square acts on z, not directly on w. Differentiate the square first and include the change of z with w.

dL/dz = slope of the outer square; dz/dw = slope of the inner linear expression.



$$
\frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw}
$$





$$
\frac{d(z^2)}{dz}=2z
$$



**Yaad rakhne ka cue:** Square ka 2z; andar ka 3; dono multiply.

### Solution — step by step

**Step 1: evaluate the inner expression.**



$$
z=3(1)+1=4
$$



**Step 2: evaluate the loss expression.**



$$
L=4^2=16
$$



**Step 3: differentiate the square and use z = 4.**



$$
\frac{dL}{dz}=2z=2(4)=8
$$



**Step 4: differentiate the inner expression.**



$$
\frac{dz}{dw}=3
$$



**Step 5: multiply, because the changes occur in sequence.**



$$
\frac{dL}{dw}=8(3)=24
$$



**Final answer:** z = 4, L = 16 and dL/dw = 24. The combined operation is multiplication, not 8+3.

## Drill 6 Chain rule on a tiny loss

## Question

**Question.** Let o = wx+b, L = ½(o−y)², x = 3, w = 0.2, b = 0.1, y = 1. Find dL/dw, dL/db and update at η = 0.1.

## Answer — step by step

### Given

Model: o = wx+b; loss L = ½(o−y)².

x = 3; y = 1; w = 0.2; b = 0.1.

Learning rate η = 0.1.

### To find

Find ∂L/∂w, ∂L/∂b and the updated weight and bias.

### Formula — which one and why

**Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x.

z = prediction; y = target; e = z−y; g\_w and g\_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate.



$$
z=wx+b
$$





$$
L=\frac12(z-y)^2
$$





$$
g_w=\frac{\partial L}{\partial w}=(z-y)x
$$





$$
g_b=\frac{\partial L}{\partial b}=z-y
$$





$$
w_{\mathrm{new}}=w-\eta g_w
$$





$$
b_{\mathrm{new}}=b-\eta g_b
$$



**Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add.

### Solution — step by step

### Meaning and formulas

The output o is a linear prediction: no sigmoid or ReLU is specified. Changing w changes o, which changes L. The chain rule multiplies those linked effects.



$$
o=wx+b,\qquad L=\frac12(o-y)^2
$$





$$
\frac{\partial L}{\partial o}=o-y,\quad \frac{\partial o}{\partial w}=x,\quad \frac{\partial o}{\partial b}=1
$$





$$
\frac{\partial L}{\partial w}=(o-y)x,\qquad \frac{\partial L}{\partial b}=o-y
$$



The learning rate η says how much of the gradient to subtract.



$$
w_{\mathrm{new}}=w-\eta\frac{\partial L}{\partial w},\qquad b_{\mathrm{new}}=b-\eta\frac{\partial L}{\partial b}
$$



### Step 1: obtain the current prediction and error



$$
o=(0.2\times3)+0.1=0.6+0.1=0.7
$$





$$
o-y=0.7-1=-0.3
$$





$$
L=\frac12(-0.3)^2=\frac12(0.09)=0.045
$$



### Step 2: calculate both gradients from this same prediction



$$
\frac{\partial L}{\partial w}=(-0.3)\times3=-0.9
$$





$$
\frac{\partial L}{\partial b}=(-0.3)\times1=-0.3
$$



### Step 3: update the weight



$$
w_{\mathrm{new}}=0.2-0.1(-0.9)=0.2-(-0.09)=0.29
$$



### Step 4: update the bias



$$
b_{\mathrm{new}}=0.1-0.1(-0.3)=0.1-(-0.03)=0.13
$$



**Final answer:** weight gradient −0.9; bias gradient −0.3; new weight 0.29; new bias 0.13. Both updates use the old forward pass.