# RMSProp remembers recent squared gradients

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s13

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s13/

**Definition.** RMSProp is an adaptive optimizer that normalizes the current gradient by the root of an exponentially weighted moving average of squared gradients. Its decaying history addresses AdaGrad's permanently growing accumulator.

### Recurrence and interpretation

Using decay β and zero initial state:



$$
s_t=\beta s_{t-1}+(1-\beta)g_t^2
$$





$$
\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon}
$$



β is the fraction of the previous squared-gradient average retained, not a signed velocity. Some papers use ρ for this same decay. The square is applied to the gradient before averaging; the square root is applied to the resulting average in the update.

If recent squared gradients are smaller than the old average, s can decrease. Thus older large gradients do not remain in the denominator with their original weight forever.

![Retain part of the old squared-gradient average and add the weighted square of the current gradient.](https://deep-learning-study-room.vercel.app/assets/figures/v_rmsprop.svg)

Diagram: Retain part of the old squared-gradient average and add the weighted square of the current gradient.. Labels: RMSProp discounts older squared gradients | RMSProp discounts older squared gradients | Use the worked question: initial s = 0; gradients 2, then 1; decay 0.9. | Step 1 memory | 0.9 × 0 + 0.1 × 4 = | 0.4 | Step 2 memory | 0.9 × 0.4 + 0.1 × 1 = | 0.46 | Scale each step | Divide g by (√s + ε) | Keep 90% of the old memory, then add 10% of the current squared gradient.

Retain part of the old squared-gradient average and add the weighted square of the current gradient. Tap the diagram to enlarge.

### Complete numerical: two iterations

**Question.** θ₀ = 1, s₀ = 0, η = 0.1, β = 0.9; supplied gradients are 2 then 1. Perform two RMSProp updates, with ε = 0 for this hand calculation.

### Given

θ₀ = 1; initial state(s) = 0; η = 0.1.

Supplied gradients g₁ = 2, g₂ = 1.

ε = 0 for this hand calculation.

β = 0.9.

### To find

Calculate all required states and the parameter after each of two updates.

### Formula — which one and why

**RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence.

s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer.



$$
s_t=\rho s_{t-1}+(1-\rho)g_t^2
$$





$$
\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon}
$$



**Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average.

### Solution — step by step

**Iteration 1 — step 1: calculate the squared-gradient average.**



$$
s_1=0.9(0)+0.1(2^2)=0.4
$$



**Step 2: calculate the denominator and signed correction.**



$$
\sqrt{s_1}=\sqrt{0.4}\approx0.632455532
$$





$$
\frac{\eta g_1}{\sqrt{s_1}}=\frac{0.1(2)}{\sqrt{0.4}}\approx0.316227766
$$



**Step 3: update the parameter.**



$$
\theta_1=1-0.316227766\approx0.683772234
$$



**Iteration 2 — step 1: use s₁, not the original zero state.**



$$
s_2=0.9(0.4)+0.1(1^2)=0.46
$$



**Step 2: calculate the denominator and signed correction.**



$$
\sqrt{s_2}=\sqrt{0.46}\approx0.678232998
$$





$$
\frac{\eta g_2}{\sqrt{s_2}}=\frac{0.1(1)}{\sqrt{0.46}}\approx0.147441956
$$



**Step 3: update from θ₁, retaining calculator precision.**



$$
\theta_2\approx0.683772234-0.147441956\approx0.536330278
$$



**Final answer:** θ₁ ≈ 0.683772 and θ₂ ≈ 0.536330; s₁ = 0.4 and s₂ = 0.46.

### Comparison with AdaGrad and Adam

AdaGrad stores an unweighted cumulative sum; RMSProp stores a decaying average. Basic RMSProp has no separate first-moment average and no Adam-style bias correction. With zero initialization, the early RMSProp state can therefore be much smaller than the first squared gradient, which explains the relatively large first correction in this example. The method still requires appropriate learning-rate and decay choices; adaptive scaling does not guarantee convergence to a global optimum.