Deep Learning · TEE · section

R26 Q5b

Re examination February 2026 fully solved

Question

Question (10 marks). Explain RMSProp, its update rule and accumulated squared gradients. Give advantages and disadvantages relative to SGD.

Answer — step by step

RMSProp keeps an exponentially weighted average of squared gradients, instead of AdaGrad's ever-growing sum. For each parameter, initialize s₀ = 0 and use:

s_t=\rho s_{t-1}+(1-\rho)g_t^2,\quad\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon}LaTeX: s_t=\rho s_{t-1}+(1-\rho)g_t^2,\quad\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon}

ρ controls how strongly earlier squared gradients are retained. A large average magnitude produces a larger denominator and a smaller step. A small average magnitude produces a relatively larger step. Squaring happens before averaging; averaging signed gradients instead would allow cancellation and would not be RMSProp.

Example: θ₀ = 1, η = 0.1, ρ = 0.9, s₀ = 0, g₁ = 2. Then s₁ = 0.9 × 0+0.1 × 4 = 0.4. θ₁ = 1−0.2/√0.4 = 0.683772234 when ε is negligible. If g₂ = 1, s₂ = 0.9 × 0.4+0.1 × 1 = 0.46 and θ₂ = 0.683772234−0.1/√0.46 = 0.536330278.

Benefits include adapting to different coordinate scales and avoiding AdaGrad's permanent accumulation of all past squared gradients. Costs include extra state memory and tuning η, ρ and ε; noisy or badly chosen settings can still produce poor training. Unlike standard SGD it rescales each coordinate; unlike Adam it does not include a separate first-moment estimate in this basic form.