Deep Learning · TEE · section
Learning rate and common training problems
Optimization algorithms
Definition. The learning rate is a hyperparameter controlling the scale of a gradient-based update. It affects convergence speed and stability; it is not the gradient and is not the magnitude of the loss.
Explanation and working
A very small rate can make progress slow. A very large rate can overshoot, oscillate or diverge. A suitable rate permits progress without uncontrolled jumps. Feature scaling can reduce long zig-zag paths. A noisy curve alone is not proof of failure; individual mini-batch losses fluctuate.
| Observation | Possible reason | Sensible first action |
|---|---|---|
| Loss barely changes | Very small rate, saturated/dead units, wrong implementation | Check gradients and data; then tune rate or activation |
| Loss jumps wildly | Rate too high or exploding gradients | Reduce rate; inspect gradient size; clip if appropriate |
| Noisy but decreasing loss | Small-batch sampling noise | Compare a smoothed trend; consider larger batches or momentum |
| Training improves but validation worsens | Overfitting | Use regularization, augmentation or early stopping |
| All identical hidden neurons learn alike | Symmetric initialization | Use suitable nonidentical initial weights |
Convergence means updates or the objective are settling under a criterion. A zero gradient can also occur at a saddle point or in a saturated region; it is not proof of a global optimum.