# Optimization algorithms

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/chapter/5/

Gradient descent, batch variants, momentum, Nesterov, AdaGrad, RMSProp and Adam. Definitions and derivations lead into complete worked updates.

1. [What an optimizer is trying to do](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s2/)
2. [The ordinary gradient descent update](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s3/)
3. [Drill 16 GD variation 1](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s16/)
4. [Drill 17 GD variation 2](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s17/)
5. [Batch stochastic and mini batch gradient descent](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s4/)
6. [Drill 15 Incomplete mini batch](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s15/)
7. [A tiny dataset shows why the methods differ](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s6/)
8. [Drill 14 Batch and sequential SGD](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s14/)
9. [Learning rate and common training problems](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s7/)
10. [A moving average remembers the recent past](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s8/)
11. [Momentum remembers movement](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s9/)
12. [Drill 18 Momentum variation 1](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s18/)
13. [Drill 19 Momentum variation 2](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s19/)
14. [Nesterov looks before correcting](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s10/)
15. [Drill 20 Nesterov variation 1](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s20/)
16. [Drill 21 Nesterov variation 2](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s21/)
17. [AdaGrad adds every squared gradient](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s12/)
18. [Drill 22 AdaGrad variation 1](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s22/)
19. [Drill 23 AdaGrad variation 2](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s23/)
20. [RMSProp remembers recent squared gradients](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s13/)
21. [Drill 24 RMSProp variation 1](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s24/)
22. [Drill 25 RMSProp variation 2](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s25/)
23. [Adam remembers both direction and squared size](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s14/)
24. [Adam bias correction: why the denominator is one minus beta to a power](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s15/)
25. [Drill 26 Adam variation 1](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s26/)
26. [Drill 27 Adam variation 2](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s27/)
27. [One comparison to rebuild all optimizer formulas](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s17/)
28. [AdaGrad and RMSProp with a negative gradient](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s18/)
29. [F25 Q4a](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c11-s4/)
30. [F25 Q5b](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c11-s5/)
31. [R26 Q4a](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c12-s4/)
32. [R26 Q5b](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c12-s6/)
33. [F23 Q1b](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c13-s0/)
34. [F23 Q3b](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c13-s2/)
35. [R24 Q1c](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c14-s0/)
36. [R24 Q2b](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c14-s2/)
37. [F22 Q1b](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c15-s1/)
38. [F22 Q4b](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c15-s6/)
39. [F22 Q5b](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c15-s7/)
40. [R23 Q1d](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c16-s2/)
41. [R23 Q6a](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c16-s7/)
42. [R23 Q7b](https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c16-s9/)