Deep Learning · TEE · section

Drill 22 Nesterov variation 1

Numerical practice from direct to mixed

Question

Question. J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity.

Answer — step by step

Given

Objective: J(θ) = ½θ².

Initial θ₀ = 1; η = 0.1; two iterations.

β = 0.9; initial signed velocity v₀ = 0.

To find

Find every stored state needed by the method and the parameter after each of two updates.

Formula — which one and why

Nesterov lookahead. First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter.

θ_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate.

\theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1}LaTeX: \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1}
g_t=J\prime(\theta_{\mathrm{look}})LaTeX: g_t=J\prime(\theta_{\mathrm{look}})
v_t=\beta v_{t-1}-\eta g_tLaTeX: v_t=\beta v_{t-1}-\eta g_t
\theta_t=\theta_{t-1}+v_tLaTeX: \theta_t=\theta_{t-1}+v_t

Yaad rakhne ka cue: Pehle look ahead; gradient wahan; update original position se.

Solution — step by step

Meaning of the symbols

θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.

Differentiate the objective first

J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0LaTeX: J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0

The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0.

Update formulas

v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.

\theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1},\quad g_t=J\prime(\theta_{\mathrm{look}})LaTeX: \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1},\quad g_t=J\prime(\theta_{\mathrm{look}})
v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_tLaTeX: v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t

Step 1: Update 1.

\theta_{\mathrm{look}}=1+0.9(0)=1LaTeX: \theta_{\mathrm{look}}=1+0.9(0)=1
g_1=1-(0)=1LaTeX: g_1=1-(0)=1
v_1=0.9(0)-0.1(1)=-0.1LaTeX: v_1=0.9(0)-0.1(1)=-0.1
\theta_1=1+(-0.1)=0.9LaTeX: \theta_1=1+(-0.1)=0.9

Step 2: Update 2.

\theta_{\mathrm{look}}=0.9+0.9(-0.1)=0.81LaTeX: \theta_{\mathrm{look}}=0.9+0.9(-0.1)=0.81
g_2=0.81-(0)=0.81LaTeX: g_2=0.81-(0)=0.81
v_2=0.9(-0.1)-0.1(0.81)=-0.171LaTeX: v_2=0.9(-0.1)-0.1(0.81)=-0.171
\theta_2=0.9+(-0.171)=0.729LaTeX: \theta_2=0.9+(-0.171)=0.729

Final parameter after two updates: 0.729. The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead.