# F25 Q4a

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/read/11/mt2-c11-s4

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c11-s4/

## Question

**Question (10 marks).** For N = 1,000 samples and batch size 100, count weight updates per epoch for batch GD, SGD and mini-batch GD. For J(θ) = 0.5 θ², η = 0.1, momentum 0.9, v₀ = 0, θ₀ = 1, compute two momentum iterations. List the required momentum-training hyperparameters.

## Answer — step by step

### Given

N = 1,000; mini-batch size = 100.

J(θ) = 0.5θ²; θ₀ = 1; v₀ = 0; η = 0.1; β = 0.9.

Use signed velocity.

### To find

Count updates per epoch for batch, stochastic and mini-batch GD; then perform two momentum iterations.

### Formula — which one and why

**Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.

N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.



$$
U_{\mathrm{batch}}=E
$$





$$
U_{\mathrm{SGD}}=EN
$$





$$
U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil
$$





$$
U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor
$$



**Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue.

**Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.

v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.



$$
v_t=\beta v_{t-1}-\eta g_t
$$





$$
\theta_t=\theta_{t-1}+v_t
$$



**Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo.

### Solution — step by step

**Step 1: count full-batch updates.**

All 1,000 samples contribute to one update: 1 per epoch.

**Step 2: count stochastic updates.**

One update per sample: 1,000 per epoch.

**Step 3: count mini-batches.**



$$
1000/100=10
$$



Use signed velocity vₜ = βvₜ₋₁−ηgₜ and θₜ = θₜ₋₁+vₜ; g(θ) = θ.

### Momentum iteration 1

**Step 1: evaluate the gradient at the current parameter.**



$$
g_1=1
$$



**Step 2: retain the previous displacement.**



$$
0.9(0)=0
$$



**Step 3: scale the current gradient.**



$$
0.1(1)=0.1
$$



**Step 4: combine to obtain the new signed velocity.**



$$
v_1=0-0.1=-0.1
$$



**Step 5: add new velocity to the actual old parameter.**



$$
\theta_1=1+(-0.1)=0.9
$$



### Momentum iteration 2

**Step 1: evaluate the gradient at the current parameter.**



$$
g_2=0.9
$$



**Step 2: retain the previous displacement.**



$$
0.9(-0.1)=-0.09
$$



**Step 3: scale the current gradient.**



$$
0.1(0.9)=0.09
$$



**Step 4: combine to obtain the new signed velocity.**



$$
v_2=-0.09-0.09=-0.18
$$



**Step 5: add new velocity to the actual old parameter.**



$$
\theta_2=0.9+(-0.18)=0.72
$$



**Hyperparameters:**  the basic momentum update has two main tunable coefficients, learning rate η and momentum β. A full training run also requires choices such as epochs and batch size (one for literal SGD), and may include a learning-rate schedule. There is no universal single count for all training configurations; state what is counted. Initial θ and v are initialization values, not learned outputs of the first update.