Deep Learning · TEE · section

Optimizer notation: old value, memory, new value

Optimization algorithms

1. A parameter and a stored state are different

θ is the parameter we want to learn. A memory such as v, s or m helps an optimizer decide how to change θ; it is not a second prediction. Each trainable parameter usually has its own corresponding memory entries.

Diagram: The time label tells you which value to use. Carry the previous memory into the current update.. Labels: Read one update from old values to new values | Read one update from old values to new values | The subscript is a step label, not multiplication. | OLD | w at step t − 1 | CURRENT | gradient g at step t | NEW | w at step t | An optimizer may also carry a memory from the previous step.

At time t = 0, the problem supplies starting values. At update t = 1, read the old values, calculate the current gradient and new memory, then calculate the new parameter. At update t = 2, use the results of update 1 as the old values.

2. A supplied gradient versus a supplied objective

If the question says gradients are 2 and −1, use those values in order. If it instead gives J(θ) = θ², first differentiate to get g(θ) = 2θ. Recalculate g using the current parameter at each iteration. Nesterov evaluates at its lookahead point instead.

For θ₀ = 1 and η = 0.1 with ordinary GD, g₁ = 2 and θ₁ = 0.8. The next gradient is 2 × 0.8 = 1.6, not the old value 2. Therefore θ₂ = 0.8 − 0.1 × 1.6 = 0.64.

3. A squared gradient keeps size and loses sign

Both 2² and (−2)² equal 4. Squared-gradient memory tells us about scale. The signed numerator still decides the update direction. Never replace the gradient in every part of an optimizer by its square.

4. Square root and epsilon belong to the denominator

If s = 4 and ε = 0.01, the denominator √s + ε is 2 + 0.01 = 2.01. It is different from √(s + ε), which is √4.01. Use the placement specified by the formula. Small ε keeps a zero accumulator from causing division by zero.

5. A table prevents mixing iterations

Write one row per iteration. Ordinary GD needs current θ, gradient and new θ. Momentum adds old and new velocity. Adam adds both memories and their corrected values. Complete a row before starting the next; otherwise a new value from iteration 2 may accidentally be used inside iteration 1.

The worked examples below use this exact order, with every state shown.