Deep Learning · MT2 · formulas

Momentum: signed velocity

Momentum carries some of the previous movement into the next update. Here v is signed movement, so the new parameter is the old parameter plus v. A negative v moves the parameter down. Use the same convention throughout the calculation.

$v_t=\beta v_{t-1}-\eta g_t$ LaTeX: v_t=\beta v_{t-1}-\eta g_t

$\theta_t=\theta_{t-1}+v_t$ LaTeX: \theta_t=\theta_{t-1}+v_t

Model answer: momentum and Nesterov

Question

Question. Explain momentum and Nesterov accelerated gradient. How does the point at which the gradient is measured differ?

Answer — step by step

Momentum retains a fraction of the previous movement and combines it with the current gradient correction. Movement in a persistent direction can build up, while alternating directions can partially cancel. This can reduce zig-zagging and improve progress through poorly scaled regions.

Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum. Tap the diagram to enlarge.

Momentum formulas

This answer uses signed displacement velocity v: a negative velocity moves the parameter left. η is learning rate and β is the fraction of previous velocity retained.

$g_t=J'(\theta_{t-1})$ LaTeX: g_t=J'(\theta_{t-1})

$v_t=\beta v_{t-1}-\eta g_t$ LaTeX: v_t=\beta v_{t-1}-\eta g_t

$\theta_t=\theta_{t-1}+v_t$ LaTeX: \theta_t=\theta_{t-1}+v_t

Nesterov's change

Nesterov first forms a lookahead position using the old momentum. It measures the gradient there, then computes a new velocity and updates the actual old parameter.

$\theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1}$ LaTeX: \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1}

$g_t=J'(\theta_{\mathrm{look}})$ LaTeX: g_t=J'(\theta_{\mathrm{look}})

The velocity and parameter update formulas remain as displayed above. Add the new velocity to θ at the previous iteration, not to the lookahead point.

Basis	Momentum	Nesterov
Gradient evaluated at	Current parameter	Lookahead parameter
Information retained	Previous velocity	Previous velocity
Intuition	Continue useful movement	Inspect the slope where old momentum points
Initial zero velocity	No movement contribution yet	First lookahead equals current parameter

Both require suitable learning rate and momentum settings. Neither guarantees the global minimum of a general neural-network objective. Different books use different definitions of stored velocity, so the definition and update must be used together consistently.

Momentum remembers movement

Definition. Momentum is a gradient-based optimization method that combines the current gradient correction with a fraction of the previous parameter displacement. It can accumulate movement along persistent directions and damp some oscillations across alternating directions.

Update equations and convention

This course uses signed displacement velocity v. It already includes the learning rate, and the parameter update adds it to the old parameter:

$g_t=J'(\theta_{t-1})$ LaTeX: g_t=J'(\theta_{t-1})

$v_t=\beta v_{t-1}-\eta g_t$ LaTeX: v_t=\beta v_{t-1}-\eta g_t

$\theta_t=\theta_{t-1}+v_t$ LaTeX: \theta_t=\theta_{t-1}+v_t

θ is the parameter, η is the learning rate and β controls how much previous displacement is retained, normally with 0 ≤ β < 1. A negative v moves θ in the negative direction. Every parameter has its own velocity; this optimizer state is distinct from the network's trainable parameters.

Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter. Tap the diagram to enlarge.

Complete numerical: two iterations

Question. J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1, β = 0.9. Find two iterations using signed displacement velocity.

Given

J(θ) = ½θ²; θ₀ = 1; v₀ = 0.

η = 0.1; β = 0.9; signed-displacement velocity.

To find

Find the gradient, velocity and actual parameter after each of two iterations.

Formula — which one and why

Momentum with signed velocity. Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.

v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.

$v_t=\beta v_{t-1}-\eta g_t$ LaTeX: v_t=\beta v_{t-1}-\eta g_t

$\theta_t=\theta_{t-1}+v_t$ LaTeX: \theta_t=\theta_{t-1}+v_t

Yaad rakhne ka cue: Pehle velocity banao; phir old parameter mein velocity jodo.

Solution — step by step

Derivative used: differentiating J gives J′(θ) = θ. The gradient must therefore be reevaluated at the current θ on each iteration.

Step 1: Iteration 1: gradient, velocity, parameter.

$g_1=J'(1)=1$ LaTeX: g_1=J'(1)=1

$\theta_1=1+(-0.1)=0.9$ LaTeX: \theta_1=1+(-0.1)=0.9

Step 2: Iteration 2: use θ₁ and v₁.

$g_2=J'(0.9)=0.9$ LaTeX: g_2=J'(0.9)=0.9

$\theta_2=0.9+(-0.18)=0.72$ LaTeX: \theta_2=0.9+(-0.18)=0.72

Iteration	Parameter where gradient is evaluated	Gradient	New velocity	New parameter
1	1	1	−0.1	0.9
2	0.9	0.9	−0.18	0.72

Final answer: θ₁ = 0.9 and θ₂ = 0.72. At iteration 2, the retained displacement contributes −0.09 and the new gradient correction contributes another −0.09. Their sum explains the larger displacement despite the smaller current gradient.

Properties and alternative notation

Momentum can improve progress through long, poorly scaled valleys, but excessive learning rate or momentum can overshoot. It does not guarantee a global minimum for a general neural-network objective.

Some texts store a positive gradient accumulator u instead:

With constant η and consistent initialization, this corresponds to vₜ = −ηuₜ. A normalized moving average containing (1−β)gₜ is a different scaling convention. Use the exact recurrence stated in the question instead of combining pieces of different conventions.

Drill 18 Momentum variation 1

Question

Question. J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity.