# AdaGrad

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/23

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/23/

AdaGrad stores a running sum of squared gradients for each parameter. Squaring removes the sign only in this stored total. The unsquared gradient in the numerator still decides the direction of the update.



$$
G_t=G_{t-1}+g_t^2
$$





$$
\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon}
$$



## AdaGrad adds every squared gradient

**Definition.** AdaGrad is an adaptive first-order optimizer that scales each parameter's update using the inverse square root of the sum of its previous and current squared gradients. Each parameter maintains its own accumulator.

### State, effective learning rate and update

For one parameter θ, let gₜ be its current gradient and s₀ = 0:



$$
s_t=s_{t-1}+g_t^2
$$





$$
\eta_{\mathrm{effective},t}=\frac{\eta}{\sqrt{s_t}+\varepsilon}
$$





$$
\theta_t=\theta_{t-1}-\eta_{\mathrm{effective},t}g_t
$$



s stores squared-gradient history. Both positive and negative gradients increase this nonnegative sum after squaring. The effective learning rate is the coefficient multiplying the signed gradient; the actual displacement also depends on that gradient's magnitude and sign.

![The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients.](https://deep-learning-study-room.vercel.app/assets/figures/v_adagrad.svg)

Diagram: The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients.. Labels: AdaGrad accumulates all past squared gradients | AdaGrad accumulates all past squared gradients | Use the worked question: initial s = 0; gradients 2, then 1. | First gradient | s₁ = 0 + 2² = 4 | Second gradient | s₂ = 4 + 1² = 5 | Scale each step | Divide g by (√s + ε) | The accumulator cannot decrease. Use the current accumulator in each parameter | update.

The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients. Tap the diagram to enlarge.

### Complete numerical: supplied gradients

**Question.** Start θ₀ = 1, s₀ = 0, η = 0.1. Given gradients g₁ = 2 and g₂ = 1, perform two AdaGrad updates. Ignore epsilon only for this hand calculation because neither denominator is zero.

### Given

θ₀ = 1; initial state(s) = 0; η = 0.1.

Supplied gradients g₁ = 2, g₂ = 1.

ε = 0 for this hand calculation.

### To find

Calculate all required states and the parameter after each of two updates.

### Formula — which one and why

**AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude.

G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so.



$$
G_t=G_{t-1}+g_t^2
$$





$$
\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon}
$$



**Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo.

### Solution — step by step

The gradients are supplied by the question. They are not computed from an assumed objective, and g₂ is not replaced by the updated parameter.

**Iteration 1 — step 1: update the accumulator.**



$$
s_1=0+2^2=4
$$



**Step 2: calculate the effective learning rate.**



$$
\eta_{\mathrm{effective},1}=\frac{0.1}{\sqrt4}=0.05
$$



**Step 3: apply the signed gradient.**



$$
\theta_1=1-0.05(2)=0.9
$$



**Iteration 2 — step 1: retain s₁ and add the new squared gradient.**



$$
s_2=4+1^2=5
$$



**Step 2: calculate the new coefficient.**



$$
\eta_{\mathrm{effective},2}=\frac{0.1}{\sqrt5}\approx0.044721360
$$



**Step 3: update from θ₁.**



$$
\theta_2=0.9-\frac{0.1(1)}{\sqrt5}\approx0.855278640
$$



**Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.855279, with accumulators s₁ = 4 and s₂ = 5.

### Advantage, limitation and epsilon convention

Coordinate-specific scaling can be useful when some features or gradients occur infrequently. Because s never decreases, its effective learning rate is nonincreasing when η and ε are fixed, and can become too small during prolonged training. The actual displacement magnitude need not decrease monotonically because gₜ also changes.

ε is normally a small positive stabilizer. These notes place it outside the square root. If a question places it inside, use that specified denominator consistently; the two placements are not numerically identical.

## Drill 22 AdaGrad variation 1

## Question

**Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use AdaGrad for two updates, zero initial accumulators and ε = 0 for this hand calculation.

## Answer — step by step

### Given

Initial θ₀ = 1; η = 0.1.

Supplied gradients: [2,1].

Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation.

### To find

Find every stored state needed by the method and the parameter after each of two updates.

### Formula — which one and why

**AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude.

G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so.



$$
G_t=G_{t-1}+g_t^2
$$





$$
\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon}
$$



**Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo.

### Solution — step by step

### Symbols and formulas used

θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise.

G is a running sum of squared gradients. Squaring removes the sign only inside G; keep the gradient sign in the update numerator. Start G₀ = 0.



$$
G_t=G_{t-1}+g_t^2
$$





$$
\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\varepsilon}
$$



Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result.

**Step 1: Iteration 1 with supplied gradient 2.**

Add the new squared gradient to all previous squares.



$$
G_1=0+(2)^2=4
$$





$$
\sqrt{G_1}=\sqrt{4}=2
$$





$$
\theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9
$$



**Step 2: Iteration 2 with supplied gradient 1.**

Add the new squared gradient to all previous squares.



$$
G_2=4+(1)^2=5
$$





$$
\sqrt{G_2}=\sqrt{5}=2.236067977
$$





$$
\theta_2=0.9-\frac{0.1\times(1)}{\sqrt{5}}=0.85527864
$$



## Drill 23 AdaGrad variation 2

## Question

**Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use AdaGrad for two updates, zero initial accumulators and ε = 0 for this hand calculation.

## Answer — step by step

### Given

Initial θ₀ = −0.5; η = 0.1.

Supplied gradients: [−1,2].

Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation.

### To find

Find every stored state needed by the method and the parameter after each of two updates.

### Formula — which one and why

**AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude.

G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so.



$$
G_t=G_{t-1}+g_t^2
$$





$$
\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon}
$$



**Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo.

### Solution — step by step

### Symbols and formulas used

θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise.

G is a running sum of squared gradients. Squaring removes the sign only inside G; keep the gradient sign in the update numerator. Start G₀ = 0.



$$
G_t=G_{t-1}+g_t^2
$$





$$
\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\varepsilon}
$$



Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result.

**Step 1: Iteration 1 with supplied gradient -1.**

Add the new squared gradient to all previous squares.



$$
G_1=0+(-1)^2=1
$$





$$
\sqrt{G_1}=\sqrt{1}=1
$$





$$
\theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{1}}=-0.4
$$



**Step 2: Iteration 2 with supplied gradient 2.**

Add the new squared gradient to all previous squares.



$$
G_2=1+(2)^2=5
$$





$$
\sqrt{G_2}=\sqrt{5}=2.236067977
$$





$$
\theta_2=-0.4-\frac{0.1\times(2)}{\sqrt{5}}=-0.489442719
$$