Drill 20 Momentum variation 1

Answer — step by step

Given

Objective: J(θ) = ½θ².

Initial θ₀ = 1; η = 0.1; two iterations.

β = 0.9; initial signed velocity v₀ = 0.

To find

Find every stored state needed by the method and the parameter after each of two updates.

Formula — which one and why

Momentum with signed velocity. Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.

v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.

LaTeX: v_t=\beta v_{t-1}-\eta g_t

LaTeX: \theta_t=\theta_{t-1}+v_t

Yaad rakhne ka cue: Pehle velocity banao; phir old parameter mein velocity jodo.

Solution — step by step

Meaning of the symbols

θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.

Differentiate the objective first

LaTeX: J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0

The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0.

Update formulas

v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.

LaTeX: g_t=J\prime(\theta_{t-1})

LaTeX: v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t

Step 1: Update 1.

LaTeX: g_1=1-(0)=1

LaTeX: v_1=0.9(0)-0.1(1)=-0.1

LaTeX: \theta_1=1+(-0.1)=0.9

Step 2: Update 2.

LaTeX: g_2=0.9-(0)=0.9

LaTeX: v_2=0.9(-0.1)-0.1(0.9)=-0.18

LaTeX: \theta_2=0.9+(-0.18)=0.72

Final parameter after two updates: 0.72. The second gradient is evaluated again using the parameter obtained after the first update.