# Adam remembers both direction and squared size Subject: Deep Learning Exam: TEE Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s14 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s14/ **Definition.** Adam, or adaptive moment estimation, uses bias-corrected exponentially weighted estimates of the first moment and the uncentered second moment of gradients to calculate parameter-specific updates. ### What the two states represent | State or coefficient | Meaning | | --- | --- | | mₜ | Moving average of signed gradients; retains directional information | | sₜ | Moving average of squared gradients; estimates their uncentered second moment | | β₁ | Decay coefficient for the first-moment estimate | | β₂ | Decay coefficient for the second-moment estimate | | t | Update number, starting at 1 for the first update | | η and ε | Base learning rate and small denominator stabilizer | The second moment is not the variance: it averages g² without subtracting a mean. Both states start at zero unless specified otherwise. These notes use s throughout this lesson; texts that use v for Adam's second moment mean the same state, not momentum's displacement velocity. ![The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.](https://deep-learning-study-room.vercel.app/assets/figures/v_adam.svg) Diagram: The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.. Labels: Adam keeps two different memories | Adam keeps two different memories | One describes signed direction. The other describes squared size. | Current gradient g | Example: g = −2 | First moment m | Average of signed gradients | Second moment s | Average of squared gradients | Correct both → divide m̂ by (√ŝ + ε) → update | For g = −2: the direction contribution is negative; the squared-size contribution | uses 4. The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update. Tap the diagram to enlarge. ### Recurrences, bias correction and update $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ Zero initialization reduces the total weight initially attached to observed gradients. Divide by the appropriate accumulated observation weight: $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ The update uses the corrected signed average in the numerator and the root of the corrected squared-gradient average in the denominator: $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\varepsilon} $$ The derivation in the next lesson explains why the correction factors contain powers of the iteration number. No additional multiplication by gₜ belongs in this final update: its contribution is already inside mₜ. ### Complete numerical: changing gradient magnitude **Question.** θ₀ = 1, m₀ = s₀ = 0, η = 0.1, β₁ = 0.9, β₂ = 0.999. Given g₁ = 2 and g₂ = 1, find two Adam updates, neglecting epsilon only in the numerical arithmetic. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step **Iteration 1 — step 1: update the two raw states.** $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ s_1=0.999(0)+0.001(2^2)=0.004 $$ **Step 2: correct both states using t = 1.** $$ \hat m_1=\frac{0.2}{1-0.9^1}=\frac{0.2}{0.1}=2 $$ $$ \hat s_1=\frac{0.004}{1-0.999^1}=\frac{0.004}{0.001}=4 $$ **Step 3: update the parameter.** $$ \theta_1=1-\frac{0.1(2)}{\sqrt4}=0.9 $$ Adam iteration 1: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**2**g1 Previous raw m**0** Previous raw s**0** Signed average m1**0.2** ↓ Bias-corrected m̂1**2** Squared-gradient average s1**0.004** ↓ Bias-corrected ŝ1**4** ↓ Combine: ηm̂ / √ŝ**0.1**ε is neglected for this specific numerical. Old θ**1** → New θ**0.9** **Carry forward raw states:** m = 0.2, s = 0.004. The corrected values are used for this update. **Iteration 2 — step 1: carry forward the raw states m₁ and s₁.** The recurrence uses the raw states 0.2 and 0.004, not the corrected values 2 and 4. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ s_2=0.999(0.004)+0.001(1^2)=0.004996 $$ **Step 2: correct both states using t = 2.** $$ 1-0.9^2=0.19,\qquad1-0.999^2=0.001999 $$ $$ \hat m_2=\frac{0.28}{0.19}\approx1.473684211 $$ $$ \hat s_2=\frac{0.004996}{0.001999}\approx2.499249625 $$ **Step 3: divide the corrected first moment by the corrected root.** $$ \frac{\eta\hat m_2}{\sqrt{\hat s_2}}\approx\frac{0.1(1.473684211)}{\sqrt{2.499249625}}\approx0.093217964 $$ **Step 4: update from θ₁.** $$ \theta_2\approx0.9-0.093217964\approx0.806782036 $$ Adam iteration 2: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**1**g2 Previous raw m**0.2** Previous raw s**0.004** Signed average m2**0.28** ↓ Bias-corrected m̂2**1.473684211** Squared-gradient average s2**0.004996** ↓ Bias-corrected ŝ2**2.499249625** ↓ Combine: ηm̂ / √ŝ**0.093217964**ε is neglected for this specific numerical. Old θ**0.9** → New θ**0.806782036** **Carry forward raw states:** m = 0.28, s = 0.004996. The corrected values are used for this update. | State | After iteration 1 | After iteration 2 | | --- | --- | --- | | Raw first moment m | 0.2 | 0.28 | | Raw second moment s | 0.004 | 0.004996 | | Corrected first moment | 2 | 1.473684211 | | Corrected second moment | 4 | 2.499249625 | | Parameter θ | 0.9 | 0.806782036 | **Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.806782. Store the raw moments for the next recurrence and use the corrected moments for the current parameter update. ### Properties and limitations Adam combines directional averaging with coordinate-specific scaling, which can be useful with noisy or differently scaled gradients. It requires two state values per parameter in addition to the parameter itself. η, β₁, β₂ and ε remain hyperparameters. Faster reduction of training loss does not establish better generalization, and Adam does not guarantee the global optimum of a general neural-network objective.