Deep Learning · MT2 · section

R24 Q2b

R24 selected MT2 practice

Question

Question. Compare standard momentum and Nesterov accelerated gradient using their gradient-evaluation points and update equations. Explain what may happen when momentum is extremely close to 1, for example 0.9999.

Syllabus-aligned extract: source R24 Q2b. The current-syllabus optimizer comparison and high-momentum discussion are retained; marks for this extract are not separately stated.

Answer — step by step

Use signed velocity. Standard momentum evaluates g at current θ: vnew = βv−ηg(θ), θ new = θ+vnew. Nesterov evaluates at a look-ahead point: θ look = θ+βv; vnew = βv−ηg(θ look); θ new = θ+vnew. Thus the gradient can correct a direction after anticipating part of the momentum move. Nesterov often reduces overshoot in suitable settings, but it is not universally faster for every neural network.

With β = 0.9999, old velocity decays very slowly. The method can respond sluggishly to direction changes, overshoot or oscillate, particularly with a large learning rate. An approximate memory scale 1/(1−β) = 10,000 steps illustrates the long persistence; it is not an exact convergence time.