# Drill 20 Nesterov variation 1

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s20

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s20/

## Question

**Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity.

## Answer — step by step

### Given

Objective: J(θ) = ½θ².

Initial θ₀ = 1; η = 0.1; two iterations.

β = 0.9; initial signed velocity v₀ = 0.

### To find

Find every stored state needed by the method and the parameter after each of two updates.

### Formula — which one and why

**Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter.

θ\_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate.



$$
\theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1}
$$





$$
g_t=J\prime(\theta_{\mathrm{look}})
$$





$$
v_t=\beta v_{t-1}-\eta g_t
$$





$$
\theta_t=\theta_{t-1}+v_t
$$



**Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se.

### Solution — step by step

### Meaning of the symbols

θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.

### Differentiate the objective first



$$
J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0
$$



The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0.

### Update formulas

v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.



$$
\theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1},\quad g_t=J\prime(\theta_{\mathrm{look}})
$$





$$
v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t
$$



**Step 1: Update 1.**



$$
\theta_{\mathrm{look}}=1+0.9(0)=1
$$





$$
g_1=1-(0)=1
$$





$$
v_1=0.9(0)-0.1(1)=-0.1
$$





$$
\theta_1=1+(-0.1)=0.9
$$



**Step 2: Update 2.**



$$
\theta_{\mathrm{look}}=0.9+0.9(-0.1)=0.81
$$





$$
g_2=0.81-(0)=0.81
$$





$$
v_2=0.9(-0.1)-0.1(0.81)=-0.171
$$





$$
\theta_2=0.9+(-0.171)=0.729
$$



**Final parameter after two updates: 0.729.** The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead.