Deep Learning · TEE · section

Derivatives: from an ordinary change to a local slope

The maths behind a network

1. Function value and change are different numbers

Let L(w) = w². At w = 3, loss is 9. If we increase w to 4, loss becomes 16. The loss change is 16 − 9 = 7 and the weight change is 4 − 3 = 1. Their ratio is 7. This is an average slope over a whole interval.

Diagram: Start at weight 3 each time. As the increase becomes smaller, the slope estimate approaches 6.. Labels: Shrink the interval to see the local slope | Shrink the interval to see the local slope | Use the same starting point every time: w = 3, L = 9. | Average slope = change in loss ÷ change in weight | Weight increase | New loss | Slope estimate | 1 | 16 | (16 − 9) ÷ 1 = 7 | 0.1 | 9.61 | 0.61 ÷ 0.1 = 6.1 | 0.01 | 9.0601 | 0.0601 ÷ 0.01 = 6.01 | 7 → 6.1 → 6.01 → … → 6. The derivative at w = 3 is 6, not the loss value 9.

LaTeX: \text{average slope}=\frac{\text{new loss}-\text{old loss}}{\text{new weight}-\text{old weight}}

2. Shrink the interval

For the same starting weight 3, use an increase of 0.1. The new weight is 3.1 and its square is 9.61. The slope estimate is (9.61 − 9)/0.1 = 6.1. With an increase of 0.01, it is (9.0601 − 9)/0.01 = 6.01. Smaller intervals approach 6.

The derivative is the limiting local slope as that interval approaches zero. We do not substitute a literal zero into a division by the interval.

3. Why the square has derivative 2w

Call a small nonzero change Δw, read “delta w”. Expanding the new square gives:

LaTeX: (w+\Delta w)^2=w^2+2w\Delta w+(\Delta w)^2

Subtract the old square w² and divide by the change Δw:

LaTeX: \frac{(w+\Delta w)^2-w^2}{\Delta w}=2w+\Delta w

As Δw approaches zero, this approaches 2w. That is the origin of the rule d(w²)/dw = 2w. At w = 3 it gives 6, matching the table's trend.

4. Reading derivative notation

In dL/dw, the top names the quantity whose response we study, and the bottom names what we vary. Read it as “the derivative of L with respect to w”. The notation is not asking you to divide the current loss 9 by the current weight 3; that would give 3 and answer a different question.

For a function of several variables, ∂L/∂w is a partial derivative: change w while holding the other independent variables fixed. The gradient groups those partial derivatives for all parameters.

5. Why subtract the gradient?

If the derivative is positive, increasing w locally increases loss, so we try a small decrease. If it is negative, increasing w locally decreases loss, so we try a small increase. The subtraction in gradient descent automatically handles both signs.

LaTeX: w_{\mathrm{new}}=w-\eta g

This is local information. A very large step can cross a minimum and make loss worse. The learning rate η keeps control of the step; the derivative does not promise that any step size is safe.