Deep Learning · MT2 · guide

Optimization algorithms

  1. Reading an optimizer formula without getting lost
  2. Optimizer notation: old value, memory, new value
  3. What an optimizer is trying to do
  4. The ordinary gradient descent update
  5. Batch stochastic and mini batch gradient descent
  6. Model answer: batch, stochastic and mini-batch training
  7. A tiny dataset shows why the methods differ
  8. Learning rate and common training problems
  9. A moving average remembers the recent past
  10. Momentum remembers movement
  11. Nesterov looks before correcting
  12. Model answer: momentum and Nesterov
  13. AdaGrad adds every squared gradient
  14. RMSProp remembers recent squared gradients
  15. Adam remembers both direction and squared size
  16. Adam bias correction: why the denominator is one minus beta to a power
  17. Model answer: AdaGrad, RMSProp and Adam
  18. One comparison to rebuild all optimizer formulas
  19. AdaGrad and RMSProp with a negative gradient