Deep Learning · MT2 · section
Early stopping uses the best validation point
Regularization and generalization
Definition. Early stopping is a regularization and model-selection procedure that terminates training when a monitored validation criterion no longer improves according to a specified rule, typically retaining the best checkpoint.
Explanation and working
Monitor validation loss after epochs. When it stops improving for a chosen patience interval, stop training and restore the best validation checkpoint. Training loss can fluctuate; it does not always decrease monotonically. Never select the stopping epoch by repeatedly consulting the test set.
Question. Validation losses after epochs 1–5 are [0.8,0.6,0.5,0.53,0.55]. Patience is 2 with no required minimum improvement. When do you stop and what do you restore?
Given
Validation losses at epochs 1–5 = [0.8,0.6,0.5,0.53,0.55].
Patience = 2; no required minimum improvement.
To find
Find the stop epoch and the best checkpoint to restore.
Formula — which one and why
Early stopping rule. Compare each validation loss with the best one so far. A new best resets the waiting count; a non-improvement increases it. Stop when the count reaches patience and restore the best checkpoint.
Patience = allowed consecutive epochs without improvement. With minimum improvement zero, a strictly smaller validation loss counts as improvement.
Yaad rakhne ka cue: Best validation save; patience khatam toh stop aur best restore.
Solution — step by step
Step 1: process epoch 1.
Loss 0.8 establishes the first best checkpoint; wait = 0.
Step 2: process epoch 2.
0.6 < 0.8: save epoch 2; wait = 0.
Step 3: process epoch 3.
0.5 < 0.6: save epoch 3; wait = 0.
Step 4: process epoch 4.
0.53 > 0.5: no improvement; wait = 1.
Step 5: process epoch 5.
0.55 > 0.5: no improvement; wait = 2, equal to patience. Stop.
Step 6: restore the best checkpoint.
The smallest observed validation loss is 0.5 at epoch 3. Restore its weights.
Final answer: stop after epoch 5; restore epoch 3. These are different epochs.
Early stopping with [0.8, 0.7, 0.72, 0.68, 0.69, 0.7]
Question. Validation losses are [0.8, 0.7, 0.72, 0.68, 0.69, 0.7]. With patience 2 and strict improvement, identify the stopping epoch and checkpoint to restore.
Given
Validation losses in epoch order = [0.8, 0.7, 0.72, 0.68, 0.69, 0.7].
Patience = 2; smaller loss is better; ties are not improvements.
To find
The stopping epoch and best checkpoint.
Formula — which one and why
Save an epoch when its validation loss is strictly lower than the best so far. Stop after the stated number of consecutive non-improving epochs.
Patience counts consecutive non-improvements. A new best resets the count.
Yaad rakhne ka cue: Naya best mila toh save aur wait zero; patience khatam toh best restore.
Solution — step by step
Step 1: process epoch 1.
Validation loss = 0.8; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 1, loss 0.8.
Step 2: process epoch 2.
Validation loss = 0.7; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 2, loss 0.7.
Step 3: process epoch 3.
Validation loss = 0.72; no strict improvement: wait count becomes 1. Best checkpoint is epoch 2, loss 0.7.
Step 4: process epoch 4.
Validation loss = 0.68; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 4, loss 0.68.
Step 5: process epoch 5.
Validation loss = 0.69; no strict improvement: wait count becomes 1. Best checkpoint is epoch 4, loss 0.68.
Step 6: process epoch 6.
Validation loss = 0.7; no strict improvement: wait count becomes 2. Best checkpoint is epoch 4, loss 0.68.
Final answer: Stop after epoch 6; restore epoch 4, with validation loss 0.68.
Early stopping with [0.6, 0.6, 0.61, 0.62]
Question. Validation losses are [0.6, 0.6, 0.61, 0.62]. With patience 2 and strict improvement, identify the stopping epoch and checkpoint to restore.
Given
Validation losses in epoch order = [0.6, 0.6, 0.61, 0.62].
Patience = 2; smaller loss is better; ties are not improvements.
To find
The stopping epoch and best checkpoint.
Formula — which one and why
Save an epoch when its validation loss is strictly lower than the best so far. Stop after the stated number of consecutive non-improving epochs.
Patience counts consecutive non-improvements. A new best resets the count.
Yaad rakhne ka cue: Naya best mila toh save aur wait zero; patience khatam toh best restore.
Solution — step by step
Step 1: process epoch 1.
Validation loss = 0.6; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 1, loss 0.6.
Step 2: process epoch 2.
Validation loss = 0.6; no strict improvement: wait count becomes 1. Best checkpoint is epoch 1, loss 0.6.
Step 3: process epoch 3.
Validation loss = 0.61; no strict improvement: wait count becomes 2. Best checkpoint is epoch 1, loss 0.6.
Final answer: Stop after epoch 3; restore epoch 1, with validation loss 0.6.