Deep Learning · TEE · section

Model answer: AdaGrad, RMSProp and Adam

Optimization algorithms

Question

Question. Compare AdaGrad, RMSProp and Adam. Explain what each method stores and how that memory changes a parameter update.

Answer — step by step

AdaGrad, RMSProp and Adam are adaptive gradient-based optimizers. Each stores separate state for each trainable parameter and uses that state to scale its update. Let gₜ be the current gradient for parameter θ, η the base learning rate and ε a small positive stabilizer. The equations below put ε outside the square root.

AdaGrad: cumulative squared-gradient history

$s_t=s_{t-1}+g_t^2$ LaTeX: s_t=s_{t-1}+g_t^2

$\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon}$ LaTeX: \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon}

Starting at s₀ = 0, every squared gradient contributes permanently. Parameters with frequent large gradients acquire smaller effective learning rates. This can be useful for sparse features, but the growing denominator can make later updates too small. The base learning rate still has to be chosen.

RMSProp: a decaying squared-gradient average

$s_t=\beta s_{t-1}+(1-\beta)g_t^2$ LaTeX: s_t=\beta s_{t-1}+(1-\beta)g_t^2

$\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon}$ LaTeX: \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon}

Old contributions decay by repeated multiplication by β. The state can decrease when recent squared gradients are small, avoiding AdaGrad's permanent accumulation. Basic RMSProp has one squared-gradient state and no separate signed-gradient average.

Adam: first and second moments with bias correction

Initialize m₀ = s₀ = 0 and count the first update as t = 1:

$m_t=\beta_1m_{t-1}+(1-\beta_1)g_t$ LaTeX: m_t=\beta_1m_{t-1}+(1-\beta_1)g_t

$s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2$ LaTeX: s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2

$\theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\varepsilon}$ LaTeX: \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\varepsilon}

The first moment stores signed directional information; the second stores squared-gradient scale. Correction compensates for their zero initialization. The corrected first moment is already the numerator's gradient estimate, so the update is not multiplied by gₜ again. Raw moments are carried into the next iteration.

Aspect	AdaGrad	RMSProp	Adam
Squared-gradient history	Cumulative sum	Decaying average	Decaying average
Separate signed-gradient average	No	No in the basic method	Yes
Standard initialization correction	No	No in the basic method	Corrects both moments
State values per parameter	One	One	Two
Main concern	Effective rate can become very small	Learning rate and decay need tuning	More state; hyperparameters still matter

All three adapt updates coordinate by coordinate. None removes the need for validation or guarantees a global minimum for a general neural-network objective.