# Early stopping uses the best validation point Subject: Deep Learning Exam: TEE Reader: https://deep-learning-study-room.vercel.app/#tee/read/6/tee-c6-s4 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c6-s4/ **Definition.** Early stopping is a regularization and model-selection procedure that terminates training when a monitored validation criterion no longer improves according to a specified rule, typically retaining the best checkpoint. ### Explanation and working Monitor validation loss after epochs. When it stops improving for a chosen patience interval, stop training and restore the best validation checkpoint. Training loss can fluctuate; it does not always decrease monotonically. Never select the stopping epoch by repeatedly consulting the test set. ![Training loss keeps falling while validation loss starts rising. Keep the checkpoint from the validation minimum.](https://deep-learning-study-room.vercel.app/assets/figures/v_early_stop.svg) Diagram: Training loss keeps falling while validation loss starts rising. Keep the checkpoint from the validation minimum.. Labels: Keep the model from the best validation point | Keep the model from the best validation point | Illustrative curves: lower loss is better. | Epoch | Loss | Validation | Training | Save this best checkpoint | Patience may wait for later epochs, but restore the model with the best | validation score. Training loss keeps falling while validation loss starts rising. Keep the checkpoint from the validation minimum. Tap the diagram to enlarge. **Question.** Validation losses after epochs 1–5 are [0.8,0.6,0.5,0.53,0.55]. Patience is 2 with no required minimum improvement. When do you stop and what do you restore? ### Given Validation losses at epochs 1–5 = [0.8,0.6,0.5,0.53,0.55]. Patience = 2; no required minimum improvement. ### To find Find the stop epoch and the best checkpoint to restore. ### Formula — which one and why **Early stopping rule.** Compare each validation loss with the best one so far. A new best resets the waiting count; a non-improvement increases it. Stop when the count reaches patience and restore the best checkpoint. Patience = allowed consecutive epochs without improvement. With minimum improvement zero, a strictly smaller validation loss counts as improvement. **Yaad rakhne ka cue:** Best validation save; patience khatam toh stop aur best restore. ### Solution — step by step **Step 1: process epoch 1.** Loss 0.8 establishes the first best checkpoint; wait = 0. **Step 2: process epoch 2.** 0.6 < 0.8: save epoch 2; wait = 0. **Step 3: process epoch 3.** 0.5 < 0.6: save epoch 3; wait = 0. **Step 4: process epoch 4.** 0.53 > 0.5: no improvement; wait = 1. **Step 5: process epoch 5.** 0.55 > 0.5: no improvement; wait = 2, equal to patience. Stop. **Step 6: restore the best checkpoint.** The smallest observed validation loss is 0.5 at epoch 3. Restore its weights. **Final answer:** stop after epoch 5; restore epoch 3. These are different epochs. ### Early stopping with [0.8, 0.7, 0.72, 0.68, 0.69, 0.7] **Question. Validation losses are [0.8, 0.7, 0.72, 0.68, 0.69, 0.7]. With patience 2 and strict improvement, identify the stopping epoch and checkpoint to restore.** ### Given Validation losses in epoch order = [0.8, 0.7, 0.72, 0.68, 0.69, 0.7]. Patience = 2; smaller loss is better; ties are not improvements. ### To find The stopping epoch and best checkpoint. ### Formula — which one and why Save an epoch when its validation loss is strictly lower than the best so far. Stop after the stated number of consecutive non-improving epochs. Patience counts consecutive non-improvements. A new best resets the count. **Yaad rakhne ka cue:** Naya best mila toh save aur wait zero; patience khatam toh best restore. ### Solution — step by step **Step 1: process epoch 1.** Validation loss = 0.8; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 1, loss 0.8. **Step 2: process epoch 2.** Validation loss = 0.7; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 2, loss 0.7. **Step 3: process epoch 3.** Validation loss = 0.72; no strict improvement: wait count becomes 1. Best checkpoint is epoch 2, loss 0.7. **Step 4: process epoch 4.** Validation loss = 0.68; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 4, loss 0.68. **Step 5: process epoch 5.** Validation loss = 0.69; no strict improvement: wait count becomes 1. Best checkpoint is epoch 4, loss 0.68. **Step 6: process epoch 6.** Validation loss = 0.7; no strict improvement: wait count becomes 2. Best checkpoint is epoch 4, loss 0.68. **Final answer:** Stop after epoch 6; restore epoch 4, with validation loss 0.68. ### Early stopping with [0.6, 0.6, 0.61, 0.62] **Question. Validation losses are [0.6, 0.6, 0.61, 0.62]. With patience 2 and strict improvement, identify the stopping epoch and checkpoint to restore.** ### Given Validation losses in epoch order = [0.6, 0.6, 0.61, 0.62]. Patience = 2; smaller loss is better; ties are not improvements. ### To find The stopping epoch and best checkpoint. ### Formula — which one and why Save an epoch when its validation loss is strictly lower than the best so far. Stop after the stated number of consecutive non-improving epochs. Patience counts consecutive non-improvements. A new best resets the count. **Yaad rakhne ka cue:** Naya best mila toh save aur wait zero; patience khatam toh best restore. ### Solution — step by step **Step 1: process epoch 1.** Validation loss = 0.6; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 1, loss 0.6. **Step 2: process epoch 2.** Validation loss = 0.6; no strict improvement: wait count becomes 1. Best checkpoint is epoch 1, loss 0.6. **Step 3: process epoch 3.** Validation loss = 0.61; no strict improvement: wait count becomes 2. Best checkpoint is epoch 1, loss 0.6. **Final answer:** Stop after epoch 3; restore epoch 1, with validation loss 0.6.