Deep Learning · TEE · section

AdaGrad and RMSProp with a negative gradient

Optimization algorithms

The gradient has a sign, but its square is nonnegative. Both methods store squared gradients to control update size. The signed gradient is still used separately in the update numerator to preserve direction.

Diagram: The gradient sign remains in the numerator. Its square contributes a nonnegative value to the memory.. Labels: What happens to an old squared gradient? | What happens to an old squared gradient? | Start with old memory 4. The current gradient is −3, so its square is 9. | AdaGrad: keep adding | Old sum G + new g² | 4 + 9 = 13 | An old contribution stays. | RMSProp: decay and add | ρs + (1 − ρ)g² | 0.9 × 4 + 0.1 × 9 = 4.5 | Old contributions fade. | Both divide the signed current gradient by a square-root scale plus epsilon.

For an old stored state of 4 and a new gradient of −3, first square the gradient: (−3) × (−3) = 9.

AdaGrad: add the new square to the entire old sum: 4 + 9 = 13.

RMSProp with decay 0.9: keep 0.9 × 4 = 3.6 of the old state; add (1 − 0.9) × 9 = 0.1 × 9 = 0.9 of the new square. The new state is 3.6 + 0.9 = 4.5.

AdaGrad retains all earlier squares in a growing sum. RMSProp gradually discounts earlier squares. This difference explains why their effective step sizes change differently during training.