Deep Learning · MT2 · section

Batch stochastic and mini batch gradient descent

Optimization algorithms

Definition. Batch, stochastic and mini-batch gradient descent differ in the subset of training examples used to estimate the gradient for each parameter update. Their update equation is the same; their gradient estimates and update frequency differ.

Explanation and working

All three use the same update idea. They differ in how much data contributes to one gradient.

Count outlined groups, not individual cells, to count gradient updates in one epoch. — Count outlined groups, not individual cells, to count gradient updates in one epoch. Tap the diagram to enlarge.

Method	Data for one update	Updates in one epoch with N samples	Typical behaviour
Batch GD	All N samples	1	Stable full-data gradient; costly updates
Stochastic GD	One sample	N	Frequent noisy updates
Mini-batch GD	B samples	Ceiling of N/B if final partial batch is retained	Efficient parallel calculation with moderate noise

Question. N = 1,000 and B = 100. Find updates per epoch.

Given

N = 1,000 examples; mini-batch size B = 100; one epoch.

To find

Count full-batch, stochastic and mini-batch updates per epoch.

Formula — which one and why

Number of updates. An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.

N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.

$U_{\mathrm{batch}}=E$ LaTeX: U_{\mathrm{batch}}=E

$U_{\mathrm{SGD}}=EN$ LaTeX: U_{\mathrm{SGD}}=EN

$U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil$ LaTeX: U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil

$U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor$ LaTeX: U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor

Yaad rakhne ka cue: Kitne batches process hue, utne updates hue.

Solution — step by step

Step 1: count full-batch updates.

All 1,000 samples form one group, so one epoch has 1 update.

Step 2: count stochastic updates.

One sample per update gives 1,000 updates per epoch.

Step 3: count mini-batches.

$\frac{N}{B}=\frac{1000}{100}=10$ LaTeX: \frac{N}{B}=\frac{1000}{100}=10

Final answer: batch = 1; SGD = 1,000; mini-batch = 10 updates per epoch.

Mini-batch calculation with a remainder

Question. A dataset contains 105 samples and the mini-batch size is 20. Keeping the incomplete last batch, find the updates per epoch.

Given

N = 105; B = 20; retain the final incomplete batch.

To find

The number of updates per epoch.

Formula — which one and why

Every processed group causes one update. A remaining nonempty group also counts.

The ceiling symbol rounds upward to the next integer.

$U=\lceil N/B\rceil$ LaTeX: U=\lceil N/B\rceil

Yaad rakhne ka cue: Bacha hua batch process kiya toh ek aur update.

Solution — step by step

Step 1: count complete batches.

$5(20)=100$ LaTeX: 5(20)=100

Step 2: count remaining examples.

$105-100=5$ LaTeX: 105-100=5

Step 3: include the nonempty last batch.

$U=5+1=6$ LaTeX: U=5+1=6

Final answer: 6 updates. If a question explicitly drops the final incomplete batch, it instead gives 5.