Deep Learning · MT2 · section

Partial derivatives hold other things still

The maths behind a network

For z = wx + b, the derivative with respect to w is x, because x is held fixed. The derivative with respect to b is 1. The derivative with respect to x is w. This is the entire local maths of a neuron’s weighted sum.

Diagram: Choose one changing parameter. Everything else is held fixed while you find its effect.. Labels: Change one parameter. Hold the others fixed. | Change one parameter. Hold the others fixed. | For z = wx + b, choose which parameter you are changing. | Change w; keep x and b fixed | ∂z/∂w = x | If x = 3, the slope is 3. | Change b; keep x and w fixed | ∂z/∂b = 1 | Adding 1 to b adds 1 to z. | “Partial” identifies one chosen variable while the other variables stay fixed.

LaTeX: \frac{\partial z}{\partial w}=x,\qquad \frac{\partial z}{\partial b}=1,\qquad \frac{\partial z}{\partial x}=w

Question: z = 3w + b, w = 2, b = 1. Find z, ∂z/∂w and ∂z/∂b.

Given

z = 3w+b; w = 2; b = 1.

To find

Find z, ∂z/∂w and ∂z/∂b.

Formula — which one and why

Partial derivatives. Differentiate only with respect to the named variable; all other independent inputs and parameters remain fixed.

∂z/∂w reads “rate of change of z when w changes”. For z = wx+b, x is the coefficient of w, while b has coefficient 1.

LaTeX: \frac{\partial z}{\partial w}=x

LaTeX: \frac{\partial z}{\partial b}=1

Yaad rakhne ka cue: Neeche jis letter ka naam hai, sirf usko change samjho.

Solution — step by step

Step 1: calculate the function value.

LaTeX: z=3(2)+1=6+1=7

Step 2: differentiate with respect to w while b is fixed.

LaTeX: \frac{\partial z}{\partial w}=3+0=3

Step 3: differentiate with respect to b while w is fixed.

LaTeX: \frac{\partial z}{\partial b}=0+1=1

Final answer: z = 7; weight derivative = 3; bias derivative = 1. A function value and its rate of change are different quantities.