Deep Learning · TEE · section

A graph and a slope

The maths behind a network

A graph shows how one value changes with another. Put the weight w on the horizontal axis and the loss L on the vertical axis. A slope is change in height divided by change in horizontal position. If L rises by 6 when w rises by 2, the average slope is 3.

A derivative is the slope for an extremely small change at one location. A positive derivative says increasing the weight initially increases loss. A negative derivative says increasing the weight initially decreases loss. Gradient descent moves against that local slope. Gradient batata hai loss kis taraf badhega; hum ulte taraf chhota step lete hain.

LaTeX: L(w)=w^2,\qquad \frac{dL}{dw}=2w

At w = 3, slope = 6. With learning rate 0.1, the update is 3 − 0.1 × 6 = 2.4. Loss falls from 9 to 5.76. At w = −3, slope = −6, so the update is −3 − 0.1 × (−6) = −2.4. Both moves approach zero.

Diagram: A slope tells you the local direction of increasing loss. Labels: w | Loss | Negative slope | increase w | Positive slope | decrease w | New weight = old weight − learning rate × gradient