Deep Learning · MT2 · section
One comparison to rebuild all optimizer formulas
Optimization algorithms
| Method | What is remembered | What changes the step |
|---|---|---|
| GD or SGD | No optimizer memory in the basic rule | Current gradient × η |
| Momentum | Previous movement | Old velocity plus current correction |
| Nesterov | Previous movement | Gradient evaluated at lookahead |
| AdaGrad | Sum of all squared gradients | Divide current gradient by accumulated root |
| RMSProp | Moving average of squared gradients | Divide current gradient by recent root |
| Adam | Moving averages of gradients and squares | Divide corrected mean by corrected root |
Working order. Start each method on a separate line with its states initialized. The solved examples use the same supplied gradients to show how their stored states differ. They do not have to give the same parameter value. The prescribed gradient sequence is practice data; if a question instead gives J(θ), recalculate its gradient at the appropriate current or lookahead parameter.