# One comparison to rebuild all optimizer formulas

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s17

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s17/

| Method | What is remembered | What changes the step |
| --- | --- | --- |
| GD or SGD | No optimizer memory in the basic rule | Current gradient × η |
| Momentum | Previous movement | Old velocity plus current correction |
| Nesterov | Previous movement | Gradient evaluated at lookahead |
| AdaGrad | Sum of all squared gradients | Divide current gradient by accumulated root |
| RMSProp | Moving average of squared gradients | Divide current gradient by recent root |
| Adam | Moving averages of gradients and squares | Divide corrected mean by corrected root |

**Working order.** Start each method on a separate line with its states initialized. The solved examples use the same supplied gradients to show how their stored states differ. They do not have to give the same parameter value. The prescribed gradient sequence is practice data; if a question instead gives J(θ), recalculate its gradient at the appropriate current or lookahead parameter.

![Compare the memory rules first. Then use the matching memory in the denominator of the update.](https://deep-learning-study-room.vercel.app/assets/figures/v_adagrad_rms.svg)

Diagram: Compare the memory rules first. Then use the matching memory in the denominator of the update.. Labels: What happens to an old squared gradient? | What happens to an old squared gradient? | Start with old memory 4. The current gradient is −3, so its square is 9. | AdaGrad: keep adding | Old sum G + new g² | 4 + 9 = 13 | An old contribution stays. | RMSProp: decay and add | ρs + (1 − ρ)g² | 0.9 × 4 + 0.1 × 9 = 4.5 | Old contributions fade. | Both divide the signed current gradient by a square-root scale plus epsilon.

Compare the memory rules first. Then use the matching memory in the denominator of the update. Tap the diagram to enlarge.