Deep Learning · MT2 · section

Drill 22 AdaGrad variation 1

MT2 numerical practice from direct to mixed

Question

Question. θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use AdaGrad for two updates, zero initial accumulators and ε = 0 for this hand calculation.

Answer — step by step

Given

Initial θ₀ = 1; η = 0.1.

Supplied gradients: [2,1].

Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation.

To find

Find every stored state needed by the method and the parameter after each of two updates.

Formula — which one and why

AdaGrad. Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude.

G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so.

$G_t=G_{t-1}+g_t^2$ LaTeX: G_t=G_{t-1}+g_t^2

$\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon}$ LaTeX: \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon}

Yaad rakhne ka cue: Square karo, purane sum mein jodo, root se scale karo.

Solution — step by step

Symbols and formulas used

θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise.

G is a running sum of squared gradients. Squaring removes the sign only inside G; keep the gradient sign in the update numerator. Start G₀ = 0.

$G_t=G_{t-1}+g_t^2$ LaTeX: G_t=G_{t-1}+g_t^2

$\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\varepsilon}$ LaTeX: \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\varepsilon}

Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result.

Step 1: Iteration 1 with supplied gradient 2.

Add the new squared gradient to all previous squares.

$G_1=0+(2)^2=4$ LaTeX: G_1=0+(2)^2=4

$\sqrt{G_1}=\sqrt{4}=2$ LaTeX: \sqrt{G_1}=\sqrt{4}=2

$\theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9$ LaTeX: \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9

Step 2: Iteration 2 with supplied gradient 1.

Add the new squared gradient to all previous squares.

$G_2=4+(1)^2=5$ LaTeX: G_2=4+(1)^2=5

$\sqrt{G_2}=\sqrt{5}=2.236067977$ LaTeX: \sqrt{G_2}=\sqrt{5}=2.236067977