# AdaGrad and RMSProp with a negative gradient

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s18

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s18/

The gradient has a sign, but its square is nonnegative. Both methods store squared gradients to control update size. The signed gradient is still used separately in the update numerator to preserve direction.

![The gradient sign remains in the numerator. Its square contributes a nonnegative value to the memory.](https://deep-learning-study-room.vercel.app/assets/figures/v_adagrad_rms.svg)

Diagram: The gradient sign remains in the numerator. Its square contributes a nonnegative value to the memory.. Labels: What happens to an old squared gradient? | What happens to an old squared gradient? | Start with old memory 4. The current gradient is −3, so its square is 9. | AdaGrad: keep adding | Old sum G + new g² | 4 + 9 = 13 | An old contribution stays. | RMSProp: decay and add | ρs + (1 − ρ)g² | 0.9 × 4 + 0.1 × 9 = 4.5 | Old contributions fade. | Both divide the signed current gradient by a square-root scale plus epsilon.

The gradient sign remains in the numerator. Its square contributes a nonnegative value to the memory. Tap the diagram to enlarge.

For an old stored state of 4 and a new gradient of −3, first square the gradient: (−3) × (−3) = 9.

**AdaGrad:** add the new square to the entire old sum: 4 + 9 = 13.

**RMSProp with decay 0.9:** keep 0.9 × 4 = 3.6 of the old state; add (1 − 0.9) × 9 = 0.1 × 9 = 0.9 of the new square. The new state is 3.6 + 0.9 = 4.5.

AdaGrad retains all earlier squares in a growing sum. RMSProp gradually discounts earlier squares. This difference explains why their effective step sizes change differently during training.