# Reading an optimizer formula without getting lost

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s0

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s0/

An optimizer updates the numbers the model is allowed to learn. Usually these are weights and biases. The letter θ is a convenient name for whichever parameter is currently being updated.

![Old value, current gradient and new value belong to different moments in the update.](https://deep-learning-study-room.vercel.app/assets/figures/v_optimizer_notation.svg)

Diagram: Old value, current gradient and new value belong to different moments in the update.. Labels: Read one update from old values to new values | Read one update from old values to new values | The subscript is a step label, not multiplication. | OLD | w at step t − 1 | CURRENT | gradient g at step t | NEW | w at step t | An optimizer may also carry a memory from the previous step.

Old value, current gradient and new value belong to different moments in the update. Tap the diagram to enlarge.

### Old, current and new

At update t, θₜ₋₁ is the value before the update and θₜ is the value after it. The symbol gₜ is the loss gradient used in that update. If a question already supplies gₜ, use it. If it supplies a loss J(θ), differentiate J and evaluate the derivative at the stated position.



$$
\theta_t=\theta_{t-1}-\eta g_t
$$



η, read eta, is the learning rate. It multiplies the gradient to determine the basic correction. A positive gradient is subtracted; a negative gradient makes the new value larger because subtracting a negative is addition.

### What a stored state means

An optimizer may retain a number from its previous update. This number is called its state or memory. It is an ordinary calculated number, not an additional neuron. Momentum stores movement; AdaGrad stores a sum of squared gradients; RMSProp stores a moving average of squares; Adam stores a moving average of gradients and another of squares.

### A weighted average from ordinary arithmetic

If the old state is 4, the new squared gradient is 9 and the decay is 0.9, retain 90% of the old state and add 10% of the new square.



$$
0.9\times4=3.6,\qquad(1-0.9)\times9=0.9
$$





$$
s_{\mathrm{new}}=3.6+0.9=4.5
$$



The coefficients 0.9 and 0.1 add to 1. As updates continue, a past contribution gets multiplied by 0.9 again, so its influence fades. AdaGrad instead gives 4 + 9 = 13 because it retains the full old sum.

### Squared memory and signed direction

For gradient −3, the square is 9. The stored scale is positive, but the update still needs the negative gradient to determine direction. Do not replace every occurrence of g with g².

### Epsilon and hats

ε, read epsilon, is a small positive number that prevents an exactly zero denominator. A hat in Adam, such as m with a hat, labels a bias-corrected average. A hat does not mean squaring. The power βᵗ uses the update number t, so its value changes from the first update to the second.