Deep Learning · TEE · section
Recognizing overfitting and underfitting
Regularization and generalization
Definitions. Generalization is a model's ability to perform on unseen examples from the relevant task distribution. Underfitting occurs when the model fails to capture the task's important relationships. Overfitting occurs when fitting the training data produces substantially poorer generalization, often by modeling noise or incidental details.
Interpreting learning curves
Underfitting commonly gives high training and validation errors. Overfitting commonly gives low training error with a substantially higher validation error, or validation loss that rises while training loss continues to fall. These patterns are diagnostic clues; distribution shift, leakage and an unrepresentative split can also affect the curves.
| Observation | Plausible explanation | Appropriate response |
|---|---|---|
| Both errors remain high | Insufficient capacity, weak features or inadequate optimization | Improve representation, train appropriately, reconsider architecture or reduce excessive regularization |
| Training error low; validation error high | Overfitting or a distribution/split problem | Inspect the split; use suitable regularization, more representative data or early stopping |
| Both losses fluctuate or diverge | Optimization instability | Examine learning rate, data scaling and gradient magnitude |
Bias–variance perspective
Insufficient model flexibility can create high approximation bias. A highly flexible model can be sensitive to the particular training sample, increasing variance. Regularization trades some training fit for constraints that can improve generalization. A more complex network is not automatically more accurate on unseen data.
The validation set is used to choose hyperparameters and checkpoints. The test set should remain outside these choices. Repeatedly selecting models from test results turns the test data into part of the selection process and weakens its role as an unbiased final evaluation.