# Optimization algorithms

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/5

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/chapter/5/

Gradient descent, batch variants, momentum, Nesterov, AdaGrad, RMSProp and Adam. Definitions and derivations lead into complete worked updates.

1. [What an optimizer is trying to do](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s2/)
2. [The ordinary gradient descent update](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s3/)
3. [Drill 18 GD variation 1](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s18/)
4. [Drill 19 GD variation 2](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s19/)
5. [Batch stochastic and mini batch gradient descent](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s4/)
6. [Drill 17 Incomplete mini batch](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s17/)
7. [A tiny dataset shows why the methods differ](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s6/)
8. [Drill 16 Batch and sequential SGD](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s16/)
9. [Learning rate and common training problems](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s7/)
10. [A moving average remembers the recent past](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s8/)
11. [Momentum remembers movement](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s9/)
12. [Drill 20 Momentum variation 1](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s20/)
13. [Drill 21 Momentum variation 2](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s21/)
14. [Nesterov looks before correcting](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s10/)
15. [Drill 22 Nesterov variation 1](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s22/)
16. [Drill 23 Nesterov variation 2](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s23/)
17. [AdaGrad adds every squared gradient](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s12/)
18. [Drill 24 AdaGrad variation 1](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s24/)
19. [Drill 25 AdaGrad variation 2](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s25/)
20. [RMSProp remembers recent squared gradients](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s13/)
21. [Drill 26 RMSProp variation 1](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s26/)
22. [Drill 27 RMSProp variation 2](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s27/)
23. [Adam remembers both direction and squared size](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s14/)
24. [Adam bias correction: why the denominator is one minus beta to a power](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s15/)
25. [Drill 28 Adam variation 1](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s28/)
26. [Drill 29 Adam variation 2](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s29/)
27. [One comparison to rebuild all optimizer formulas](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s17/)
28. [AdaGrad and RMSProp with a negative gradient](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s18/)
29. [F25 Q4a](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s8/)
30. [F25 Q5b](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s11/)
31. [R26 Q4a](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s8/)
32. [R26 Q5b](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s11/)
33. [F23 Q1b](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s1/)
34. [F23 Q3b](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s7/)
35. [R24 Q1c](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s2/)
36. [R24 Q2b](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s5/)
37. [F22 Q1b](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s1/)
38. [F22 Q4b](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s10/)
39. [F22 Q5b](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s12/)
40. [R23 Q1d](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s3/)
41. [R23 Q6a](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s13/)
42. [R23 Q7b](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s16/)