# Optimization algorithms

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/guide/5

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/guide/5/

1. [Reading an optimizer formula without getting lost](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s0/)
2. [Optimizer notation: old value, memory, new value](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s1/)
3. [What an optimizer is trying to do](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s2/)
4. [The ordinary gradient descent update](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s3/)
5. [Batch stochastic and mini batch gradient descent](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s4/)
6. [Model answer: batch, stochastic and mini-batch training](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s5/)
7. [A tiny dataset shows why the methods differ](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s6/)
8. [Learning rate and common training problems](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s7/)
9. [A moving average remembers the recent past](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s8/)
10. [Momentum remembers movement](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s9/)
11. [Nesterov looks before correcting](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s10/)
12. [Model answer: momentum and Nesterov](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s11/)
13. [AdaGrad adds every squared gradient](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s12/)
14. [RMSProp remembers recent squared gradients](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s13/)
15. [Adam remembers both direction and squared size](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s14/)
16. [Adam bias correction: why the denominator is one minus beta to a power](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s15/)
17. [Model answer: AdaGrad, RMSProp and Adam](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s16/)
18. [One comparison to rebuild all optimizer formulas](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s17/)
19. [AdaGrad and RMSProp with a negative gradient](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s18/)