# Model answer: momentum and Nesterov

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s11

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s11/

## Question

**Question.** Explain momentum and Nesterov accelerated gradient. How does the point at which the gradient is measured differ?

## Answer — step by step

Momentum retains a fraction of the previous movement and combines it with the current gradient correction. Movement in a persistent direction can build up, while alternating directions can partially cancel. This can reduce zig-zagging and improve progress through poorly scaled regions.

![Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum.](https://deep-learning-study-room.vercel.app/assets/figures/v_nesterov.svg)

Diagram: Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum.. Labels: Nesterov checks the slope at the look-ahead point | Nesterov checks the slope at the look-ahead point | Use the signed-velocity convention shown in the chapter. | Current w | Look-ahead | Updated w | Move by βv\_old | Correct using gradient here | Look-ahead = w + βv\_old | Nesterov measures the gradient after the look-ahead, before choosing its final | correction.

Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum. Tap the diagram to enlarge.

### Momentum formulas

This answer uses signed displacement velocity v: a negative velocity moves the parameter left. η is learning rate and β is the fraction of previous velocity retained.



$$
g_t=J'(\theta_{t-1})
$$





$$
v_t=\beta v_{t-1}-\eta g_t
$$





$$
\theta_t=\theta_{t-1}+v_t
$$



### Nesterov's change

Nesterov first forms a lookahead position using the old momentum. It measures the gradient there, then computes a new velocity and updates the actual old parameter.



$$
\theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1}
$$





$$
g_t=J'(\theta_{\mathrm{look}})
$$



The velocity and parameter update formulas remain as displayed above. Add the new velocity to θ at the previous iteration, not to the lookahead point.

| Basis | Momentum | Nesterov |
| --- | --- | --- |
| Gradient evaluated at | Current parameter | Lookahead parameter |
| Information retained | Previous velocity | Previous velocity |
| Intuition | Continue useful movement | Inspect the slope where old momentum points |
| Initial zero velocity | No movement contribution yet | First lookahead equals current parameter |

Both require suitable learning rate and momentum settings. Neither guarantees the global minimum of a general neural-network objective. Different books use different definitions of stored velocity, so the definition and update must be used together consistently.