Deep Learning · TEE · chapter

Regularization and generalization

Overfitting, L1/L2 penalties, dropout and early stopping: mechanisms, equations, comparisons and numerical applications.

1. Recognizing overfitting and underfitting
2. L1 and L2 add a cost for large weights
3. Drill 30 Regularized objective
4. Drill 31 L1 update on positive and negative weights
5. Drill 32 L2 update
6. Drill 47 Mixed backpropagation and L2
7. Dropout removes activations during training
8. Early stopping uses the best validation point
9. Choosing a remedy
10. Inverted dropout during training and inference
11. Drill 33 Inverted dropout
12. Drill 48 Mixed dropout and a momentum update
13. F25 Q1c
14. R26 Q1c
15. F23 Q7b
16. R24 Q3b
17. F22 Q3a
18. F22 Q6b
19. R23 Q2b