Deep Learning · MT2 · section

RMSProp remembers recent squared gradients

Optimization algorithms

Definition. RMSProp is an adaptive optimizer that normalizes the current gradient by the root of an exponentially weighted moving average of squared gradients. Its decaying history addresses AdaGrad's permanently growing accumulator.

Recurrence and interpretation

Using decay β and zero initial state:

LaTeX: s_t=\beta s_{t-1}+(1-\beta)g_t^2

LaTeX: \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon}

β is the fraction of the previous squared-gradient average retained, not a signed velocity. Some papers use ρ for this same decay. The square is applied to the gradient before averaging; the square root is applied to the resulting average in the update.

If recent squared gradients are smaller than the old average, s can decrease. Thus older large gradients do not remain in the denominator with their original weight forever.

Diagram: Retain part of the old squared-gradient average and add the weighted square of the current gradient.. Labels: RMSProp discounts older squared gradients | RMSProp discounts older squared gradients | Use the worked question: initial s = 0; gradients 2, then 1; decay 0.9. | Step 1 memory | 0.9 × 0 + 0.1 × 4 = | 0.4 | Step 2 memory | 0.9 × 0.4 + 0.1 × 1 = | 0.46 | Scale each step | Divide g by (√s + ε) | Keep 90% of the old memory, then add 10% of the current squared gradient.

Complete numerical: two iterations

Question. θ₀ = 1, s₀ = 0, η = 0.1, β = 0.9; supplied gradients are 2 then 1. Perform two RMSProp updates, with ε = 0 for this hand calculation.

Given

θ₀ = 1; initial state(s) = 0; η = 0.1.

Supplied gradients g₁ = 2, g₂ = 1.

ε = 0 for this hand calculation.

β = 0.9.

To find

Calculate all required states and the parameter after each of two updates.

Formula — which one and why

RMSProp. Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence.

s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer.

LaTeX: s_t=\rho s_{t-1}+(1-\rho)g_t^2

LaTeX: \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon}

Yaad rakhne ka cue: Saare purane squares nahi—unki fading average.

Solution — step by step

Iteration 1 — step 1: calculate the squared-gradient average.

LaTeX: s_1=0.9(0)+0.1(2^2)=0.4

Step 2: calculate the denominator and signed correction.

LaTeX: \sqrt{s_1}=\sqrt{0.4}\approx0.632455532

LaTeX: \frac{\eta g_1}{\sqrt{s_1}}=\frac{0.1(2)}{\sqrt{0.4}}\approx0.316227766

Step 3: update the parameter.

LaTeX: \theta_1=1-0.316227766\approx0.683772234

Iteration 2 — step 1: use s₁, not the original zero state.

LaTeX: s_2=0.9(0.4)+0.1(1^2)=0.46

Step 2: calculate the denominator and signed correction.

LaTeX: \sqrt{s_2}=\sqrt{0.46}\approx0.678232998

LaTeX: \frac{\eta g_2}{\sqrt{s_2}}=\frac{0.1(1)}{\sqrt{0.46}}\approx0.147441956

Step 3: update from θ₁, retaining calculator precision.

LaTeX: \theta_2\approx0.683772234-0.147441956\approx0.536330278

Final answer: θ₁ ≈ 0.683772 and θ₂ ≈ 0.536330; s₁ = 0.4 and s₂ = 0.46.

Comparison with AdaGrad and Adam

AdaGrad stores an unweighted cumulative sum; RMSProp stores a decaying average. Basic RMSProp has no separate first-moment average and no Adam-style bias correction. With zero initialization, the early RMSProp state can therefore be much smaller than the first squared gradient, which explains the relatively large first correction in this example. The method still requires appropriate learning-rate and decay choices; adaptive scaling does not guarantee convergence to a global optimum.