Deep Learning · MT2 · section

Drill 26 Adam variation 1

MT2 numerical practice from direct to mixed

Question

Question. θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction.

Answer — step by step

Given

Initial θ₀ = 1; η = 0.1.

Supplied gradients: [2,1].

Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation.

β₁ = 0.9; β₂ = 0.999; use bias correction.

To find

Find every stored state needed by the method and the parameter after each of two updates.

Formula — which one and why

Adam with bias correction. Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update.

m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence.

$m_t=\beta_1m_{t-1}+(1-\beta_1)g_t$ LaTeX: m_t=\beta_1m_{t-1}+(1-\beta_1)g_t

$s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2$ LaTeX: s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2

$\hat m_t=\frac{m_t}{1-\beta_1^t}$ LaTeX: \hat m_t=\frac{m_t}{1-\beta_1^t}

$\hat s_t=\frac{s_t}{1-\beta_2^t}$ LaTeX: \hat s_t=\frac{s_t}{1-\beta_2^t}

$\theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon}$ LaTeX: \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon}

Yaad rakhne ka cue: Direction memory + square memory; hats lagao, phir update.

Solution — step by step

Symbols and formulas used

θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise.

m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999.

$m_t=\beta_1m_{t-1}+(1-\beta_1)g_t$ LaTeX: m_t=\beta_1m_{t-1}+(1-\beta_1)g_t

$v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2$ LaTeX: v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2

$\theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon}$ LaTeX: \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon}

Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result.

Step 1: Iteration 1 with supplied gradient 2.

First calculate the signed-gradient mean m and squared-gradient mean v.

$m_1=0.9(0)+0.1(2)=0.2$ LaTeX: m_1=0.9(0)+0.1(2)=0.2

Correct each zero-initialized mean using this iteration number in the power.

$\widehat m_1=\frac{0.2}{1-0.9^{1}}=2$ LaTeX: \widehat m_1=\frac{0.2}{1-0.9^{1}}=2

$\widehat v_1=\frac{0.004}{1-0.999^{1}}=4$ LaTeX: \widehat v_1=\frac{0.004}{1-0.999^{1}}=4

Divide the corrected direction by the corrected root, then subtract the scaled step.

$\theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9$ LaTeX: \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9

Step 2: Iteration 2 with supplied gradient 1.

First calculate the signed-gradient mean m and squared-gradient mean v.

Correct each zero-initialized mean using this iteration number in the power.

Divide the corrected direction by the corrected root, then subtract the scaled step.