Deep Learning · MT2 · chapter

Optimization algorithms

Gradient descent, batch variants, momentum, Nesterov, AdaGrad, RMSProp and Adam. Definitions and derivations lead into complete worked updates.

1. What an optimizer is trying to do
2. The ordinary gradient descent update
3. Drill 16 GD variation 1
4. Drill 17 GD variation 2
5. Batch stochastic and mini batch gradient descent
6. Drill 15 Incomplete mini batch
7. A tiny dataset shows why the methods differ
8. Drill 14 Batch and sequential SGD
9. Learning rate and common training problems
10. A moving average remembers the recent past
11. Momentum remembers movement
12. Drill 18 Momentum variation 1
13. Drill 19 Momentum variation 2
14. Nesterov looks before correcting
15. Drill 20 Nesterov variation 1
16. Drill 21 Nesterov variation 2
17. AdaGrad adds every squared gradient
18. Drill 22 AdaGrad variation 1
19. Drill 23 AdaGrad variation 2
20. RMSProp remembers recent squared gradients
21. Drill 24 RMSProp variation 1
22. Drill 25 RMSProp variation 2
23. Adam remembers both direction and squared size
24. Adam bias correction: why the denominator is one minus beta to a power
25. Drill 26 Adam variation 1
26. Drill 27 Adam variation 2
27. One comparison to rebuild all optimizer formulas
28. AdaGrad and RMSProp with a negative gradient
29. F25 Q4a
30. F25 Q5b
31. R26 Q4a
32. R26 Q5b
33. F23 Q1b
34. F23 Q3b
35. R24 Q1c
36. R24 Q2b
37. F22 Q1b
38. F22 Q4b
39. F22 Q5b
40. R23 Q1d
41. R23 Q6a
42. R23 Q7b