Deep Learning · TEE · formulas

Nesterov: look-ahead gradient

Nesterov first uses the old movement to choose a temporary look-ahead position. It measures the gradient there, calculates the new movement, then adds that movement to the original parameter. The temporary look-ahead position does not replace the original parameter in the final addition.

$\theta_{look}=\theta+\beta v$ LaTeX: \theta_{look}=\theta+\beta v

$v_{new}=\beta v-\eta g(\theta_{look})$ LaTeX: v_{new}=\beta v-\eta g(\theta_{look})

$\theta_{new}=\theta+v_{new}$ LaTeX: \theta_{new}=\theta+v_{new}

Nesterov looks before correcting

Definition. Nesterov accelerated gradient is a momentum-based method that evaluates the gradient at a lookahead parameter, formed using the previous velocity, before calculating the new velocity and actual parameter update.

What changes relative to momentum

Ordinary momentum evaluates J′ at θₜ₋₁. Nesterov evaluates it after the anticipated displacement βvₜ₋₁. This changes the point where the objective's slope is measured:

$\theta_{\mathrm{look},t}=\theta_{t-1}+\beta v_{t-1}$ LaTeX: \theta_{\mathrm{look},t}=\theta_{t-1}+\beta v_{t-1}

$g_t=J'(\theta_{\mathrm{look},t})$ LaTeX: g_t=J'(\theta_{\mathrm{look},t})

$v_t=\beta v_{t-1}-\eta g_t$ LaTeX: v_t=\beta v_{t-1}-\eta g_t

$\theta_t=\theta_{t-1}+v_t$ LaTeX: \theta_t=\theta_{t-1}+v_t

The lookahead point is an auxiliary evaluation point, not the new actual parameter. In this signed-displacement convention, the new velocity is added to θₜ₋₁.

Locate the lookahead point using the old velocity; measure the gradient there; use that gradient to calculate the actual update. Tap the diagram to enlarge.

Complete numerical: two iterations

Question. J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1 and β = 0.9. Perform two Nesterov updates using signed displacement velocity.

Given

J(θ) = ½θ²; θ₀ = 1; v₀ = 0.

η = 0.1; β = 0.9; signed-displacement velocity.

To find

Find the gradient, velocity and actual parameter after each of two iterations.

Formula — which one and why

Nesterov lookahead. First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter.

θ_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate.

$\theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1}$ LaTeX: \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1}

$g_t=J\prime(\theta_{\mathrm{look}})$ LaTeX: g_t=J\prime(\theta_{\mathrm{look}})

$v_t=\beta v_{t-1}-\eta g_t$ LaTeX: v_t=\beta v_{t-1}-\eta g_t

$\theta_t=\theta_{t-1}+v_t$ LaTeX: \theta_t=\theta_{t-1}+v_t

Yaad rakhne ka cue: Pehle look ahead; gradient wahan; update original position se.

Solution — step by step

The derivative is J′(θ) = θ. Therefore the gradient equals the lookahead value in this particular objective.

Iteration 1 — step 1: locate the lookahead point.

$\theta_{\mathrm{look},1}=1+0.9(0)=1$ LaTeX: \theta_{\mathrm{look},1}=1+0.9(0)=1

Step 2: evaluate the gradient there.

$g_1=J'(1)=1$ LaTeX: g_1=J'(1)=1

Step 3: calculate the new velocity and actual parameter.

$\theta_1=1+(-0.1)=0.9$ LaTeX: \theta_1=1+(-0.1)=0.9

Iteration 2 — step 1: use the previous actual parameter and velocity.

Step 2: evaluate the gradient at 0.81, not at 0.9.

$g_2=J'(0.81)=0.81$ LaTeX: g_2=J'(0.81)=0.81

Step 3: calculate the new velocity.

Step 4: update the old actual parameter.

Iteration	Old actual θ	Lookahead θ	Gradient	New v	New actual θ
1	1	1	1	−0.1	0.9
2	0.9	0.81	0.81	−0.171	0.729

Final answer: θ₁ = 0.9 and θ₂ = 0.729. The first update matches ordinary momentum because v₀ = 0. At the second update, ordinary momentum would use gradient 0.9 and produce θ₂ = 0.72. Adding v₂ to 0.81 instead of 0.9 would count the old momentum contribution twice.

Drill 22 Nesterov variation 1

Question

Question. J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity.