# Momentum remembers movement

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s9

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s9/

**Definition.** Momentum is a gradient-based optimization method that combines the current gradient correction with a fraction of the previous parameter displacement. It can accumulate movement along persistent directions and damp some oscillations across alternating directions.

### Update equations and convention

This course uses **signed displacement velocity** v. It already includes the learning rate, and the parameter update adds it to the old parameter:



$$
g_t=J'(\theta_{t-1})
$$





$$
v_t=\beta v_{t-1}-\eta g_t
$$





$$
\theta_t=\theta_{t-1}+v_t
$$



θ is the parameter, η is the learning rate and β controls how much previous displacement is retained, normally with 0 ≤ β < 1. A negative v moves θ in the negative direction. Every parameter has its own velocity; this optimizer state is distinct from the network's trainable parameters.

![Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter.](https://deep-learning-study-room.vercel.app/assets/figures/v_momentum.svg)

Diagram: Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter.. Labels: Momentum remembers movement | Momentum remembers movement | Use signed velocity: v = βv\_old − ηg; then w\_new = w\_old + v. | Keep movement | β × old velocity | Add correction | Subtract η × gradient | Move weight | Add the new velocity | The memory carries direction. It is not a sum of squared gradients.

Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter. Tap the diagram to enlarge.

### Complete numerical: two iterations

**Question.** J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1, β = 0.9. Find two iterations using signed displacement velocity.

### Given

J(θ) = ½θ²; θ₀ = 1; v₀ = 0.

η = 0.1; β = 0.9; signed-displacement velocity.

### To find

Find the gradient, velocity and actual parameter after each of two iterations.

### Formula — which one and why

**Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.

v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.



$$
v_t=\beta v_{t-1}-\eta g_t
$$





$$
\theta_t=\theta_{t-1}+v_t
$$



**Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo.

### Solution — step by step

**Derivative used:** differentiating J gives J′(θ) = θ. The gradient must therefore be reevaluated at the current θ on each iteration.

**Step 1: Iteration 1: gradient, velocity, parameter.**



$$
g_1=J'(1)=1
$$





$$
v_1=0.9(0)-0.1(1)=-0.1
$$





$$
\theta_1=1+(-0.1)=0.9
$$



**Step 2: Iteration 2: use θ₁ and v₁.**



$$
g_2=J'(0.9)=0.9
$$





$$
v_2=0.9(-0.1)-0.1(0.9)=-0.18
$$





$$
\theta_2=0.9+(-0.18)=0.72
$$



| Iteration | Parameter where gradient is evaluated | Gradient | New velocity | New parameter |
| --- | --- | --- | --- | --- |
| 1 | 1 | 1 | −0.1 | 0.9 |
| 2 | 0.9 | 0.9 | −0.18 | 0.72 |

**Final answer:** θ₁ = 0.9 and θ₂ = 0.72. At iteration 2, the retained displacement contributes −0.09 and the new gradient correction contributes another −0.09. Their sum explains the larger displacement despite the smaller current gradient.

### Properties and alternative notation

Momentum can improve progress through long, poorly scaled valleys, but excessive learning rate or momentum can overshoot. It does not guarantee a global minimum for a general neural-network objective.

Some texts store a positive gradient accumulator u instead:



$$
u_t=\beta u_{t-1}+g_t,\qquad\theta_t=\theta_{t-1}-\eta u_t
$$



With constant η and consistent initialization, this corresponds to vₜ = −ηuₜ. A normalized moving average containing (1−β)gₜ is a different scaling convention. Use the exact recurrence stated in the question instead of combining pieces of different conventions.