Deep Learning · MT2 · section

Drill 6 Chain rule on a tiny loss

MT2 numerical practice from direct to mixed

Question

Question. Let o = wx+b, L = ½(o−y)², x = 3, w = 0.2, b = 0.1, y = 1. Find dL/dw, dL/db and update at η = 0.1.

Answer — step by step

Given

Model: o = wx+b; loss L = ½(o−y)².

x = 3; y = 1; w = 0.2; b = 0.1.

Learning rate η = 0.1.

To find

Find ∂L/∂w, ∂L/∂b and the updated weight and bias.

Formula — which one and why

Chain rule for a linear neuron. The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x.

z = prediction; y = target; e = z−y; g_w and g_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate.

z=wx+bLaTeX: z=wx+b
L=\frac12(z-y)^2LaTeX: L=\frac12(z-y)^2
g_w=\frac{\partial L}{\partial w}=(z-y)xLaTeX: g_w=\frac{\partial L}{\partial w}=(z-y)x
g_b=\frac{\partial L}{\partial b}=z-yLaTeX: g_b=\frac{\partial L}{\partial b}=z-y
w_{\mathrm{new}}=w-\eta g_wLaTeX: w_{\mathrm{new}}=w-\eta g_w
b_{\mathrm{new}}=b-\eta g_bLaTeX: b_{\mathrm{new}}=b-\eta g_b

Yaad rakhne ka cue: Ek path mein multiply; alag paths mil rahe hon toh add.

Solution — step by step

Meaning and formulas

The output o is a linear prediction: no sigmoid or ReLU is specified. Changing w changes o, which changes L. The chain rule multiplies those linked effects.

o=wx+b,\qquad L=\frac12(o-y)^2LaTeX: o=wx+b,\qquad L=\frac12(o-y)^2
\frac{\partial L}{\partial o}=o-y,\quad \frac{\partial o}{\partial w}=x,\quad \frac{\partial o}{\partial b}=1LaTeX: \frac{\partial L}{\partial o}=o-y,\quad \frac{\partial o}{\partial w}=x,\quad \frac{\partial o}{\partial b}=1
\frac{\partial L}{\partial w}=(o-y)x,\qquad \frac{\partial L}{\partial b}=o-yLaTeX: \frac{\partial L}{\partial w}=(o-y)x,\qquad \frac{\partial L}{\partial b}=o-y

The learning rate η says how much of the gradient to subtract.

w_{\mathrm{new}}=w-\eta\frac{\partial L}{\partial w},\qquad b_{\mathrm{new}}=b-\eta\frac{\partial L}{\partial b}LaTeX: w_{\mathrm{new}}=w-\eta\frac{\partial L}{\partial w},\qquad b_{\mathrm{new}}=b-\eta\frac{\partial L}{\partial b}

Step 1: obtain the current prediction and error

o=(0.2\times3)+0.1=0.6+0.1=0.7LaTeX: o=(0.2\times3)+0.1=0.6+0.1=0.7
o-y=0.7-1=-0.3LaTeX: o-y=0.7-1=-0.3
L=\frac12(-0.3)^2=\frac12(0.09)=0.045LaTeX: L=\frac12(-0.3)^2=\frac12(0.09)=0.045

Step 2: calculate both gradients from this same prediction

\frac{\partial L}{\partial w}=(-0.3)\times3=-0.9LaTeX: \frac{\partial L}{\partial w}=(-0.3)\times3=-0.9
\frac{\partial L}{\partial b}=(-0.3)\times1=-0.3LaTeX: \frac{\partial L}{\partial b}=(-0.3)\times1=-0.3

Step 3: update the weight

w_{\mathrm{new}}=0.2-0.1(-0.9)=0.2-(-0.09)=0.29LaTeX: w_{\mathrm{new}}=0.2-0.1(-0.9)=0.2-(-0.09)=0.29

Step 4: update the bias

b_{\mathrm{new}}=0.1-0.1(-0.3)=0.1-(-0.03)=0.13LaTeX: b_{\mathrm{new}}=0.1-0.1(-0.3)=0.1-(-0.03)=0.13

Final answer: weight gradient −0.9; bias gradient −0.3; new weight 0.29; new bias 0.13. Both updates use the old forward pass.