Deep Learning · TEE · section

Chain rule: the complete calculation

The maths behind a network

When one value depends on another, a change passes through both calculations. The total derivative multiplies the two local derivatives. It does not add them. Here w is the number being changed, z is an intermediate value, and L is the final value.

Diagram: Follow w to z to loss. The local slopes on that dependency path multiply.. Labels: Linked changes multiply along a path | Linked changes multiply along a path | Example: z = 3w + 1 and L = z², evaluated at w = 1. | w = 1 | dz/dw = 3 | z = 4 | dL/dz = 2z = 8 | L = 16 | dL/dw = 8 × 3 = 24 | w affects z; z affects L. Multiply the two local slopes.

Let z = 3w + 1 and L = z². We will calculate the derivative of L with respect to w at w = 1.

Step 1: find the intermediate value. Replace w by 1: z = 3 × 1 + 1 = 4.

Step 2: differentiate the last calculation. For L = z², the power rule gives dL/dz = 2z. At z = 4, this is 2 × 4 = 8. This measures how L responds to z.

Step 3: differentiate the first calculation. For z = 3w + 1, the derivative of 3w is 3 and the derivative of the constant 1 is 0. Therefore dz/dw = 3. This measures how z responds to w.

Step 4: connect the two effects. Multiply 8 by 3. The final derivative is 24.

LaTeX: \frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw}=(2z)(3)=8\times3=24

The function value is L = 4² = 16. The derivative is 24. These have different meanings: 16 is the current value of L; 24 is its local rate of change per unit change in w.