# Learning rate and common training problems

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s7

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s7/

**Definition.** The learning rate is a hyperparameter controlling the scale of a gradient-based update. It affects convergence speed and stability; it is not the gradient and is not the magnitude of the loss.

### Explanation and working

A very small rate can make progress slow. A very large rate can overshoot, oscillate or diverge. A suitable rate permits progress without uncontrolled jumps. Feature scaling can reduce long zig-zag paths. A noisy curve alone is not proof of failure; individual mini-batch losses fluctuate.

![Illustrative loss traces show why smoothness and progress are different](https://deep-learning-study-room.vercel.app/assets/figures/optimizers.svg)

Diagram: Illustrative loss traces show why smoothness and progress are different. Labels: Loss | Updates | Full batch trend | SGD fluctuations | Illustration only: smoothness alone does not prove better learning.

Illustrative loss traces show why smoothness and progress are different Tap the diagram to enlarge.

| Observation | Possible reason | Sensible first action |
| --- | --- | --- |
| Loss barely changes | Very small rate, saturated/dead units, wrong implementation | Check gradients and data; then tune rate or activation |
| Loss jumps wildly | Rate too high or exploding gradients | Reduce rate; inspect gradient size; clip if appropriate |
| Noisy but decreasing loss | Small-batch sampling noise | Compare a smoothed trend; consider larger batches or momentum |
| Training improves but validation worsens | Overfitting | Use regularization, augmentation or early stopping |
| All identical hidden neurons learn alike | Symmetric initialization | Use suitable nonidentical initial weights |

Convergence means updates or the objective are settling under a criterion. A zero gradient can also occur at a saddle point or in a saturated region; it is not proof of a global optimum.