# Momentum: signed velocity

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/21

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/21/

Momentum carries some of the previous movement into the next update. Here v is signed movement, so the new parameter is the old parameter plus v. A negative v moves the parameter down. Use the same convention throughout the calculation.



$$
v_t=\beta v_{t-1}-\eta g_t
$$





$$
\theta_t=\theta_{t-1}+v_t
$$



## Model answer: momentum and Nesterov

## Question

**Question.** Explain momentum and Nesterov accelerated gradient. How does the point at which the gradient is measured differ?

## Answer — step by step

Momentum retains a fraction of the previous movement and combines it with the current gradient correction. Movement in a persistent direction can build up, while alternating directions can partially cancel. This can reduce zig-zagging and improve progress through poorly scaled regions.

![Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum.](https://deep-learning-study-room.vercel.app/assets/figures/v_nesterov.svg)

Diagram: Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum.. Labels: Nesterov checks the slope at the look-ahead point | Nesterov checks the slope at the look-ahead point | Use the signed-velocity convention shown in the chapter. | Current w | Look-ahead | Updated w | Move by βv\_old | Correct using gradient here | Look-ahead = w + βv\_old | Nesterov measures the gradient after the look-ahead, before choosing its final | correction.

Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum. Tap the diagram to enlarge.

### Momentum formulas

This answer uses signed displacement velocity v: a negative velocity moves the parameter left. η is learning rate and β is the fraction of previous velocity retained.



$$
g_t=J'(\theta_{t-1})
$$





$$
v_t=\beta v_{t-1}-\eta g_t
$$





$$
\theta_t=\theta_{t-1}+v_t
$$



### Nesterov's change

Nesterov first forms a lookahead position using the old momentum. It measures the gradient there, then computes a new velocity and updates the actual old parameter.



$$
\theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1}
$$





$$
g_t=J'(\theta_{\mathrm{look}})
$$



The velocity and parameter update formulas remain as displayed above. Add the new velocity to θ at the previous iteration, not to the lookahead point.

| Basis | Momentum | Nesterov |
| --- | --- | --- |
| Gradient evaluated at | Current parameter | Lookahead parameter |
| Information retained | Previous velocity | Previous velocity |
| Intuition | Continue useful movement | Inspect the slope where old momentum points |
| Initial zero velocity | No movement contribution yet | First lookahead equals current parameter |

Both require suitable learning rate and momentum settings. Neither guarantees the global minimum of a general neural-network objective. Different books use different definitions of stored velocity, so the definition and update must be used together consistently.

## Momentum remembers movement

**Definition.** Momentum is a gradient-based optimization method that combines the current gradient correction with a fraction of the previous parameter displacement. It can accumulate movement along persistent directions and damp some oscillations across alternating directions.

### Update equations and convention

This course uses **signed displacement velocity** v. It already includes the learning rate, and the parameter update adds it to the old parameter:



$$
g_t=J'(\theta_{t-1})
$$





$$
v_t=\beta v_{t-1}-\eta g_t
$$





$$
\theta_t=\theta_{t-1}+v_t
$$



θ is the parameter, η is the learning rate and β controls how much previous displacement is retained, normally with 0 ≤ β < 1. A negative v moves θ in the negative direction. Every parameter has its own velocity; this optimizer state is distinct from the network's trainable parameters.

![Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter.](https://deep-learning-study-room.vercel.app/assets/figures/v_momentum.svg)

Diagram: Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter.. Labels: Momentum remembers movement | Momentum remembers movement | Use signed velocity: v = βv\_old − ηg; then w\_new = w\_old + v. | Keep movement | β × old velocity | Add correction | Subtract η × gradient | Move weight | Add the new velocity | The memory carries direction. It is not a sum of squared gradients.

Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter. Tap the diagram to enlarge.

### Complete numerical: two iterations

**Question.** J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1, β = 0.9. Find two iterations using signed displacement velocity.

### Given

J(θ) = ½θ²; θ₀ = 1; v₀ = 0.

η = 0.1; β = 0.9; signed-displacement velocity.

### To find

Find the gradient, velocity and actual parameter after each of two iterations.

### Formula — which one and why

**Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.

v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.



$$
v_t=\beta v_{t-1}-\eta g_t
$$





$$
\theta_t=\theta_{t-1}+v_t
$$



**Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo.

### Solution — step by step

**Derivative used:** differentiating J gives J′(θ) = θ. The gradient must therefore be reevaluated at the current θ on each iteration.

**Step 1: Iteration 1: gradient, velocity, parameter.**



$$
g_1=J'(1)=1
$$





$$
v_1=0.9(0)-0.1(1)=-0.1
$$





$$
\theta_1=1+(-0.1)=0.9
$$



**Step 2: Iteration 2: use θ₁ and v₁.**



$$
g_2=J'(0.9)=0.9
$$





$$
v_2=0.9(-0.1)-0.1(0.9)=-0.18
$$





$$
\theta_2=0.9+(-0.18)=0.72
$$



| Iteration | Parameter where gradient is evaluated | Gradient | New velocity | New parameter |
| --- | --- | --- | --- | --- |
| 1 | 1 | 1 | −0.1 | 0.9 |
| 2 | 0.9 | 0.9 | −0.18 | 0.72 |

**Final answer:** θ₁ = 0.9 and θ₂ = 0.72. At iteration 2, the retained displacement contributes −0.09 and the new gradient correction contributes another −0.09. Their sum explains the larger displacement despite the smaller current gradient.

### Properties and alternative notation

Momentum can improve progress through long, poorly scaled valleys, but excessive learning rate or momentum can overshoot. It does not guarantee a global minimum for a general neural-network objective.

Some texts store a positive gradient accumulator u instead:



$$
u_t=\beta u_{t-1}+g_t,\qquad\theta_t=\theta_{t-1}-\eta u_t
$$



With constant η and consistent initialization, this corresponds to vₜ = −ηuₜ. A normalized moving average containing (1−β)gₜ is a different scaling convention. Use the exact recurrence stated in the question instead of combining pieces of different conventions.

## Drill 20 Momentum variation 1

## Question

**Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity.

## Answer — step by step

### Given

Objective: J(θ) = ½θ².

Initial θ₀ = 1; η = 0.1; two iterations.

β = 0.9; initial signed velocity v₀ = 0.

### To find

Find every stored state needed by the method and the parameter after each of two updates.

### Formula — which one and why

**Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.

v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.



$$
v_t=\beta v_{t-1}-\eta g_t
$$





$$
\theta_t=\theta_{t-1}+v_t
$$



**Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo.

### Solution — step by step

### Meaning of the symbols

θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.

### Differentiate the objective first



$$
J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0
$$



The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0.

### Update formulas

v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.



$$
g_t=J\prime(\theta_{t-1})
$$





$$
v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t
$$



**Step 1: Update 1.**



$$
g_1=1-(0)=1
$$





$$
v_1=0.9(0)-0.1(1)=-0.1
$$





$$
\theta_1=1+(-0.1)=0.9
$$



**Step 2: Update 2.**



$$
g_2=0.9-(0)=0.9
$$





$$
v_2=0.9(-0.1)-0.1(0.9)=-0.18
$$





$$
\theta_2=0.9+(-0.18)=0.72
$$



**Final parameter after two updates: 0.72.** The second gradient is evaluated again using the parameter obtained after the first update.

## Drill 21 Momentum variation 2

## Question

**Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity.

## Answer — step by step

### Given

Objective: J(θ) = ½(θ−1)².

Initial θ₀ = −0.5; η = 0.1; two iterations.

β = 0.9; initial signed velocity v₀ = 0.

### To find

Find every stored state needed by the method and the parameter after each of two updates.

### Formula — which one and why

**Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.

v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.



$$
v_t=\beta v_{t-1}-\eta g_t
$$





$$
\theta_t=\theta_{t-1}+v_t
$$



**Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo.

### Solution — step by step

### Meaning of the symbols

θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.

### Differentiate the objective first



$$
J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1
$$



The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1.

### Update formulas

v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.



$$
g_t=J\prime(\theta_{t-1})
$$





$$
v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t
$$



**Step 1: Update 1.**



$$
g_1=-0.5-(1)=-1.5
$$





$$
v_1=0.9(0)-0.1(-1.5)=0.15
$$





$$
\theta_1=-0.5+(0.15)=-0.35
$$



**Step 2: Update 2.**



$$
g_2=-0.35-(1)=-1.35
$$





$$
v_2=0.9(0.15)-0.1(-1.35)=0.27
$$





$$
\theta_2=-0.35+(0.27)=-0.08
$$



**Final parameter after two updates: -0.08.** The second gradient is evaluated again using the parameter obtained after the first update.