# R24 Q2b

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/14/tee-c14-s5

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s5/

## Question

**Question.** Compare standard momentum and Nesterov accelerated gradient using their gradient-evaluation points and update equations. Explain what may happen when momentum is extremely close to 1, for example 0.9999.

**Syllabus-aligned extract:** source R24 Q2b. The current-syllabus optimizer comparison and high-momentum discussion are retained; marks for this extract are not separately stated.

## Answer — step by step

Use signed velocity. Standard momentum evaluates g at current θ: vnew = βv−ηg(θ), θ new = θ+vnew. Nesterov evaluates at a look-ahead point: θ look = θ+βv; vnew = βv−ηg(θ look); θ new = θ+vnew. Thus the gradient can correct a direction after anticipating part of the momentum move. Nesterov often reduces overshoot in suitable settings, but it is not universally faster for every neural network.

With β = 0.9999, old velocity decays very slowly. The method can respond sluggishly to direction changes, overshoot or oscillate, particularly with a large learning rate. An approximate memory scale 1/(1−β) = 10,000 steps illustrates the long persistence; it is not an exact convergence time.