Deep Learning · TEE · chapter

Optimization algorithms

Gradient descent, batch variants, momentum, Nesterov, AdaGrad, RMSProp and Adam. Definitions and derivations lead into complete worked updates.

1. What an optimizer is trying to do
2. The ordinary gradient descent update
3. Drill 18 GD variation 1
4. Drill 19 GD variation 2
5. Batch stochastic and mini batch gradient descent
6. Drill 17 Incomplete mini batch
7. A tiny dataset shows why the methods differ
8. Drill 16 Batch and sequential SGD
9. Learning rate and common training problems
10. A moving average remembers the recent past
11. Momentum remembers movement
12. Drill 20 Momentum variation 1
13. Drill 21 Momentum variation 2
14. Nesterov looks before correcting
15. Drill 22 Nesterov variation 1
16. Drill 23 Nesterov variation 2
17. AdaGrad adds every squared gradient
18. Drill 24 AdaGrad variation 1
19. Drill 25 AdaGrad variation 2
20. RMSProp remembers recent squared gradients
21. Drill 26 RMSProp variation 1
22. Drill 27 RMSProp variation 2
23. Adam remembers both direction and squared size
24. Adam bias correction: why the denominator is one minus beta to a power
25. Drill 28 Adam variation 1
26. Drill 29 Adam variation 2
27. One comparison to rebuild all optimizer formulas
28. AdaGrad and RMSProp with a negative gradient
29. F25 Q4a
30. F25 Q5b
31. R26 Q4a
32. R26 Q5b
33. F23 Q1b
34. F23 Q3b
35. R24 Q1c
36. R24 Q2b
37. F22 Q1b
38. F22 Q4b
39. F22 Q5b
40. R23 Q1d
41. R23 Q6a
42. R23 Q7b