# Recognizing overfitting and underfitting

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/6/tee-c6-s0

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c6-s0/

**Definitions.** Generalization is a model's ability to perform on unseen examples from the relevant task distribution. Underfitting occurs when the model fails to capture the task's important relationships. Overfitting occurs when fitting the training data produces substantially poorer generalization, often by modeling noise or incidental details.

### Interpreting learning curves

Underfitting commonly gives high training and validation errors. Overfitting commonly gives low training error with a substantially higher validation error, or validation loss that rises while training loss continues to fall. These patterns are diagnostic clues; distribution shift, leakage and an unrepresentative split can also affect the curves.

![Compare training and validation behavior when selecting model capacity or a stopping point.](https://deep-learning-study-room.vercel.app/assets/figures/overfitting.svg)

Diagram: Compare training and validation behavior when selecting model capacity or a stopping point.. Labels: Loss | Epochs | Validation | Training | Best validation region | Save the best validation checkpoint; do not choose by test loss.

Compare training and validation behavior when selecting model capacity or a stopping point. Tap the diagram to enlarge.

| Observation | Plausible explanation | Appropriate response |
| --- | --- | --- |
| Both errors remain high | Insufficient capacity, weak features or inadequate optimization | Improve representation, train appropriately, reconsider architecture or reduce excessive regularization |
| Training error low; validation error high | Overfitting or a distribution/split problem | Inspect the split; use suitable regularization, more representative data or early stopping |
| Both losses fluctuate or diverge | Optimization instability | Examine learning rate, data scaling and gradient magnitude |

### Bias–variance perspective

Insufficient model flexibility can create high approximation bias. A highly flexible model can be sensitive to the particular training sample, increasing variance. Regularization trades some training fit for constraints that can improve generalization. A more complex network is not automatically more accurate on unseen data.

The validation set is used to choose hyperparameters and checkpoints. The test set should remain outside these choices. Repeatedly selecting models from test results turns the test data into part of the selection process and weakens its role as an unbiased final evaluation.