Deep Learning · MT2 · section

F25 Q4a

F25 selected MT2 practice

Question

Question (10 marks). For N = 1,000 samples and batch size 100, count weight updates per epoch for batch GD, SGD and mini-batch GD. For J(θ) = 0.5 θ², η = 0.1, momentum 0.9, v₀ = 0, θ₀ = 1, compute two momentum iterations. List the required momentum-training hyperparameters.

Answer — step by step

Given

N = 1,000; mini-batch size = 100.

J(θ) = 0.5θ²; θ₀ = 1; v₀ = 0; η = 0.1; β = 0.9.

Use signed velocity.

To find

Count updates per epoch for batch, stochastic and mini-batch GD; then perform two momentum iterations.

Formula — which one and why

Number of updates. An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.

N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.

$U_{\mathrm{batch}}=E$ LaTeX: U_{\mathrm{batch}}=E

$U_{\mathrm{SGD}}=EN$ LaTeX: U_{\mathrm{SGD}}=EN

$U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil$ LaTeX: U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil

$U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor$ LaTeX: U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor

Yaad rakhne ka cue: Kitne batches process hue, utne updates hue.

Momentum with signed velocity. Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.

v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.

$v_t=\beta v_{t-1}-\eta g_t$ LaTeX: v_t=\beta v_{t-1}-\eta g_t

$\theta_t=\theta_{t-1}+v_t$ LaTeX: \theta_t=\theta_{t-1}+v_t

Yaad rakhne ka cue: Pehle velocity banao; phir old parameter mein velocity jodo.

Solution — step by step

Step 1: count full-batch updates.

All 1,000 samples contribute to one update: 1 per epoch.

Step 2: count stochastic updates.

One update per sample: 1,000 per epoch.

Step 3: count mini-batches.

$1000/100=10$ LaTeX: 1000/100=10

Use signed velocity vₜ = βvₜ₋₁−ηgₜ and θₜ = θₜ₋₁+vₜ; g(θ) = θ.

Momentum iteration 1

Step 1: evaluate the gradient at the current parameter.

$g_1=1$ LaTeX: g_1=1

Step 2: retain the previous displacement.

$0.9(0)=0$ LaTeX: 0.9(0)=0

Step 3: scale the current gradient.

$0.1(1)=0.1$ LaTeX: 0.1(1)=0.1

Step 4: combine to obtain the new signed velocity.

$v_1=0-0.1=-0.1$ LaTeX: v_1=0-0.1=-0.1

Step 5: add new velocity to the actual old parameter.

$\theta_1=1+(-0.1)=0.9$ LaTeX: \theta_1=1+(-0.1)=0.9

Momentum iteration 2

Step 1: evaluate the gradient at the current parameter.

$g_2=0.9$ LaTeX: g_2=0.9

Step 2: retain the previous displacement.

$0.9(-0.1)=-0.09$ LaTeX: 0.9(-0.1)=-0.09

Step 3: scale the current gradient.

$0.1(0.9)=0.09$ LaTeX: 0.1(0.9)=0.09

Step 4: combine to obtain the new signed velocity.

$v_2=-0.09-0.09=-0.18$ LaTeX: v_2=-0.09-0.09=-0.18

Step 5: add new velocity to the actual old parameter.

$\theta_2=0.9+(-0.18)=0.72$ LaTeX: \theta_2=0.9+(-0.18)=0.72

Hyperparameters: the basic momentum update has two main tunable coefficients, learning rate η and momentum β. A full training run also requires choices such as epochs and batch size (one for literal SGD), and may include a learning-rate schedule. There is no universal single count for all training configurations; state what is counted. Initial θ and v are initialization values, not learned outputs of the first update.