# R25 Q6b · AdaGrad stagnation versus RMSProp

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s13

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s13/

## Question

**Question. A healthcare model trained with AdaGrad becomes stagnant after many epochs. Discuss AdaGrad and RMSProp in detail and their fundamental differences.** (10 marks)

## Answer — step by step

**Definition.** AdaGrad is an adaptive first-order optimizer that scales each parameter's update using the inverse square root of the sum of its previous and current squared gradients. Each parameter maintains its own accumulator.

### State, effective learning rate and update

For one parameter θ, let gₜ be its current gradient and s₀ = 0:



$$
s_t=s_{t-1}+g_t^2
$$





$$
\eta_{\mathrm{effective},t}=\frac{\eta}{\sqrt{s_t}+\varepsilon}
$$





$$
\theta_t=\theta_{t-1}-\eta_{\mathrm{effective},t}g_t
$$



s stores squared-gradient history. Both positive and negative gradients increase this nonnegative sum after squaring. The effective learning rate is the coefficient multiplying the signed gradient; the actual displacement also depends on that gradient's magnitude and sign.

![The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients.](https://deep-learning-study-room.vercel.app/assets/figures/v_adagrad.svg)

Diagram: The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients.. Labels: AdaGrad accumulates all past squared gradients | AdaGrad accumulates all past squared gradients | Use the worked question: initial s = 0; gradients 2, then 1. | First gradient | s₁ = 0 + 2² = 4 | Second gradient | s₂ = 4 + 1² = 5 | Scale each step | Divide g by (√s + ε) | The accumulator cannot decrease. Use the current accumulator in each parameter | update.

The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients. Tap the diagram to enlarge.

### Complete numerical: supplied gradients

**Question.** Start θ₀ = 1, s₀ = 0, η = 0.1. Given gradients g₁ = 2 and g₂ = 1, perform two AdaGrad updates. Ignore epsilon only for this hand calculation because neither denominator is zero.

### Given

θ₀ = 1; initial state(s) = 0; η = 0.1.

Supplied gradients g₁ = 2, g₂ = 1.

ε = 0 for this hand calculation.

### To find

Calculate all required states and the parameter after each of two updates.

### Formula — which one and why

**AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude.

G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so.



$$
G_t=G_{t-1}+g_t^2
$$





$$
\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon}
$$



**Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo.

### Solution — step by step

The gradients are supplied by the question. They are not computed from an assumed objective, and g₂ is not replaced by the updated parameter.

**Iteration 1 — step 1: update the accumulator.**



$$
s_1=0+2^2=4
$$



**Step 2: calculate the effective learning rate.**



$$
\eta_{\mathrm{effective},1}=\frac{0.1}{\sqrt4}=0.05
$$



**Step 3: apply the signed gradient.**



$$
\theta_1=1-0.05(2)=0.9
$$



**Iteration 2 — step 1: retain s₁ and add the new squared gradient.**



$$
s_2=4+1^2=5
$$



**Step 2: calculate the new coefficient.**



$$
\eta_{\mathrm{effective},2}=\frac{0.1}{\sqrt5}\approx0.044721360
$$



**Step 3: update from θ₁.**



$$
\theta_2=0.9-\frac{0.1(1)}{\sqrt5}\approx0.855278640
$$



**Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.855279, with accumulators s₁ = 4 and s₂ = 5.

### Advantage, limitation and epsilon convention

Coordinate-specific scaling can be useful when some features or gradients occur infrequently. Because s never decreases, its effective learning rate is nonincreasing when η and ε are fixed, and can become too small during prolonged training. The actual displacement magnitude need not decrease monotonically because gₜ also changes.

ε is normally a small positive stabilizer. These notes place it outside the square root. If a question places it inside, use that specified denominator consistently; the two placements are not numerically identical.

**Definition.** RMSProp is an adaptive optimizer that normalizes the current gradient by the root of an exponentially weighted moving average of squared gradients. Its decaying history addresses AdaGrad's permanently growing accumulator.

### Recurrence and interpretation

Using decay β and zero initial state:



$$
s_t=\beta s_{t-1}+(1-\beta)g_t^2
$$





$$
\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon}
$$



β is the fraction of the previous squared-gradient average retained, not a signed velocity. Some papers use ρ for this same decay. The square is applied to the gradient before averaging; the square root is applied to the resulting average in the update.

If recent squared gradients are smaller than the old average, s can decrease. Thus older large gradients do not remain in the denominator with their original weight forever.

![Retain part of the old squared-gradient average and add the weighted square of the current gradient.](https://deep-learning-study-room.vercel.app/assets/figures/v_rmsprop.svg)

Diagram: Retain part of the old squared-gradient average and add the weighted square of the current gradient.. Labels: RMSProp discounts older squared gradients | RMSProp discounts older squared gradients | Use the worked question: initial s = 0; gradients 2, then 1; decay 0.9. | Step 1 memory | 0.9 × 0 + 0.1 × 4 = | 0.4 | Step 2 memory | 0.9 × 0.4 + 0.1 × 1 = | 0.46 | Scale each step | Divide g by (√s + ε) | Keep 90% of the old memory, then add 10% of the current squared gradient.

Retain part of the old squared-gradient average and add the weighted square of the current gradient. Tap the diagram to enlarge.

### Complete numerical: two iterations

**Question.** θ₀ = 1, s₀ = 0, η = 0.1, β = 0.9; supplied gradients are 2 then 1. Perform two RMSProp updates, with ε = 0 for this hand calculation.

### Given

θ₀ = 1; initial state(s) = 0; η = 0.1.

Supplied gradients g₁ = 2, g₂ = 1.

ε = 0 for this hand calculation.

β = 0.9.

### To find

Calculate all required states and the parameter after each of two updates.

### Formula — which one and why

**RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence.

s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer.



$$
s_t=\rho s_{t-1}+(1-\rho)g_t^2
$$





$$
\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon}
$$



**Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average.

### Solution — step by step

**Iteration 1 — step 1: calculate the squared-gradient average.**



$$
s_1=0.9(0)+0.1(2^2)=0.4
$$



**Step 2: calculate the denominator and signed correction.**



$$
\sqrt{s_1}=\sqrt{0.4}\approx0.632455532
$$





$$
\frac{\eta g_1}{\sqrt{s_1}}=\frac{0.1(2)}{\sqrt{0.4}}\approx0.316227766
$$



**Step 3: update the parameter.**



$$
\theta_1=1-0.316227766\approx0.683772234
$$



**Iteration 2 — step 1: use s₁, not the original zero state.**



$$
s_2=0.9(0.4)+0.1(1^2)=0.46
$$



**Step 2: calculate the denominator and signed correction.**



$$
\sqrt{s_2}=\sqrt{0.46}\approx0.678232998
$$





$$
\frac{\eta g_2}{\sqrt{s_2}}=\frac{0.1(1)}{\sqrt{0.46}}\approx0.147441956
$$



**Step 3: update from θ₁, retaining calculator precision.**



$$
\theta_2\approx0.683772234-0.147441956\approx0.536330278
$$



**Final answer:** θ₁ ≈ 0.683772 and θ₂ ≈ 0.536330; s₁ = 0.4 and s₂ = 0.46.

### Comparison with AdaGrad and Adam

AdaGrad stores an unweighted cumulative sum; RMSProp stores a decaying average. Basic RMSProp has no separate first-moment average and no Adam-style bias correction. With zero initialization, the early RMSProp state can therefore be much smaller than the first squared gradient, which explains the relatively large first correction in this example. The method still requires appropriate learning-rate and decay choices; adaptive scaling does not guarantee convergence to a global optimum.

AdaGrad’s nondecreasing squared-gradient sum can shrink its effective step excessively. RMSProp’s exponential average discounts older squared gradients, so it avoids that permanent accumulation. Both adapt rates coordinatewise and use ε for numerical stability. Switching is a plausible remedy, not a guarantee: inspect learning rate, gradient scale, data and convergence before attributing every plateau to the optimizer.