Deep Learning · MT2 · section

Reading an optimizer formula without getting lost

Optimization algorithms

An optimizer updates the numbers the model is allowed to learn. Usually these are weights and biases. The letter θ is a convenient name for whichever parameter is currently being updated.

Diagram: Old value, current gradient and new value belong to different moments in the update.. Labels: Read one update from old values to new values | Read one update from old values to new values | The subscript is a step label, not multiplication. | OLD | w at step t − 1 | CURRENT | gradient g at step t | NEW | w at step t | An optimizer may also carry a memory from the previous step.

Old, current and new

At update t, θₜ₋₁ is the value before the update and θₜ is the value after it. The symbol gₜ is the loss gradient used in that update. If a question already supplies gₜ, use it. If it supplies a loss J(θ), differentiate J and evaluate the derivative at the stated position.

LaTeX: \theta_t=\theta_{t-1}-\eta g_t

η, read eta, is the learning rate. It multiplies the gradient to determine the basic correction. A positive gradient is subtracted; a negative gradient makes the new value larger because subtracting a negative is addition.

What a stored state means

An optimizer may retain a number from its previous update. This number is called its state or memory. It is an ordinary calculated number, not an additional neuron. Momentum stores movement; AdaGrad stores a sum of squared gradients; RMSProp stores a moving average of squares; Adam stores a moving average of gradients and another of squares.

A weighted average from ordinary arithmetic

If the old state is 4, the new squared gradient is 9 and the decay is 0.9, retain 90% of the old state and add 10% of the new square.

LaTeX: 0.9\times4=3.6,\qquad(1-0.9)\times9=0.9

LaTeX: s_{\mathrm{new}}=3.6+0.9=4.5

The coefficients 0.9 and 0.1 add to 1. As updates continue, a past contribution gets multiplied by 0.9 again, so its influence fades. AdaGrad instead gives 4 + 9 = 13 because it retains the full old sum.

Squared memory and signed direction

For gradient −3, the square is 9. The stored scale is positive, but the update still needs the negative gradient to determine direction. Do not replace every occurrence of g with g².

Epsilon and hats

ε, read epsilon, is a small positive number that prevents an exactly zero denominator. A hat in Adam, such as m with a hat, labels a bias-corrected average. A hat does not mean squaring. The power βᵗ uses the update number t, so its value changes from the first update to the second.