Deep Learning · MT2 · section

Drill 17 GD variation 2

MT2 numerical practice from direct to mixed

Question

Question. J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two GD updates.

Answer — step by step

Given

Objective: J(θ) = ½(θ−1)².

Initial θ₀ = −0.5; η = 0.1; two iterations.

To find

Find every stored state needed by the method and the parameter after each of two updates.

Formula — which one and why

Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.

θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.

g_t=J\prime(\theta_{t-1})LaTeX: g_t=J\prime(\theta_{t-1})
\theta_t=\theta_{t-1}-\eta g_tLaTeX: \theta_t=\theta_{t-1}-\eta g_t

Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.

Solution — step by step

Meaning of the symbols

θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.

Differentiate the objective first

J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1LaTeX: J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1

The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1.

Update formulas

g_t=J\prime(\theta_{t-1}),\quad \theta_t=\theta_{t-1}-\eta g_tLaTeX: g_t=J\prime(\theta_{t-1}),\quad \theta_t=\theta_{t-1}-\eta g_t

Step 1: Update 1.

g_1=-0.5-(1)=-1.5LaTeX: g_1=-0.5-(1)=-1.5
\theta_1=-0.5-0.1(-1.5)=-0.35LaTeX: \theta_1=-0.5-0.1(-1.5)=-0.35

Step 2: Update 2.

g_2=-0.35-(1)=-1.35LaTeX: g_2=-0.35-(1)=-1.35
\theta_2=-0.35-0.1(-1.35)=-0.215LaTeX: \theta_2=-0.35-0.1(-1.35)=-0.215

Final parameter after two updates: -0.215. The second gradient is evaluated again using the parameter obtained after the first update.