Deep Learning · MT2 · formulas

Batch and mini-batch gradient

A batch is a group of training examples used together for one update. An epoch is one pass through the dataset. Averaging gradients and counting updates are different questions, so both receive complete worked examples below.

$g_{batch}=\frac1B\sum_{i=1}^{B}g_i$ LaTeX: g_{batch}=\frac1B\sum_{i=1}^{B}g_i

$U_{epoch}=\left\lceil\frac NB\right\rceil$ LaTeX: U_{epoch}=\left\lceil\frac NB\right\rceil

Model answer: batch, stochastic and mini-batch training

Question

Question. Differentiate batch gradient descent, stochastic gradient descent and mini-batch gradient descent. Explain an epoch, a batch and an iteration.

Answer — step by step

All three methods update model parameters using gradients. Their difference is the number of training examples used to estimate the gradient for one update.

All methods visit the same six examples. Their grouping determines how often they update. Tap the diagram to enlarge.

A batch is the group of examples used together for an update. An iteration, in this context, is one optimizer update. An epoch is one complete pass through the chosen training dataset. One epoch usually contains several iterations.

Basis	Batch GD	Stochastic GD	Mini-batch GD
Examples in one gradient	Entire training dataset	One example	A smaller group of B examples
Updates per epoch	One	N	Ceiling of N/B if the remainder is retained
Gradient behavior	Full-data gradient	Noisy sample gradient	Intermediate averaging
Practical implication	Updates can be expensive on large datasets	Many small, sequential updates	Common compromise for vectorized computation

Example to distinguish the three counts

With 105 examples and batch size 20, five batches use 100 examples and a sixth uses the remaining 5. Therefore one epoch has 6 mini-batch updates. Three epochs give 18 updates. It would be 5 updates per epoch only if the problem explicitly discarded the incomplete batch.

The basic parameter update is unchanged:

$\theta_{\mathrm{new}}=\theta_{\mathrm{old}}-\eta g$ LaTeX: \theta_{\mathrm{new}}=\theta_{\mathrm{old}}-\eta g

For batch or mini-batch training, g normally represents the gradient of the stated aggregate loss over that group. For sequential SGD, update after each example and use the new parameter for the next example. Consequently, one SGD epoch need not equal one batch update even when both use the same examples.

A tiny dataset shows why the methods differ

Question. Train ŷ = wx on samples (1,2) and (2,4), starting at w = 1, with η = 0.1 and mean half squared loss. Compare one batch update with one SGD epoch in the listed order.

Given

Model ŷ = wx; initial w = 1; η = 0.1.

Samples in order: (1,2), (2,4).

Mean half squared loss for the batch.

To find

Compare one batch update with a sequential SGD epoch.

Formula — which one and why

Batch mean versus sequential SGD. A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter.

B = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x.

$g_{\mathrm{batch}}=\frac1B\sum_i g_i$ LaTeX: g_{\mathrm{batch}}=\frac1B\sum_i g_i

$g_i=(wx_i-y_i)x_i$ LaTeX: g_i=(wx_i-y_i)x_i

$w_{\mathrm{new}}=w-\eta g$ LaTeX: w_{\mathrm{new}}=w-\eta g

Yaad rakhne ka cue: Batch: same old weight par average; SGD: har sample ke baad weight change.

Solution — step by step

One batch update

Step 1: predict sample 1 with the original weight.

$\hat y_1=1(1)=1$ LaTeX: \hat y_1=1(1)=1

Step 2: calculate its error.

$e_1=1-2=-1$ LaTeX: e_1=1-2=-1

Step 3: multiply its error by its input.

$g_1=(-1)(1)=-1$ LaTeX: g_1=(-1)(1)=-1

Step 4: predict sample 2 with the same original weight.

$\hat y_2=1(2)=2$ LaTeX: \hat y_2=1(2)=2

Step 5: calculate its error.

$e_2=2-4=-2$ LaTeX: e_2=2-4=-2

Step 6: multiply its error by its input.

$g_2=(-2)(2)=-4$ LaTeX: g_2=(-2)(2)=-4

Step 7: average the two gradients.

$g=\frac{-1+(-4)}2=-2.5$ LaTeX: g=\frac{-1+(-4)}2=-2.5

Step 8: update once.

This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation. Tap the diagram to enlarge.

One sequential SGD epoch

Step 1: use the first sample gradient.

$g_1=-1$ LaTeX: g_1=-1

Step 2: update after the first sample.

$w_1=1-0.1(-1)=1.1$ LaTeX: w_1=1-0.1(-1)=1.1

Step 3: predict sample 2 using the changed weight.

$\hat y_2=1.1(2)=2.2$ LaTeX: \hat y_2=1.1(2)=2.2

Step 4: find sample 2 error.

$e_2=2.2-4=-1.8$ LaTeX: e_2=2.2-4=-1.8

Step 5: find the new sample 2 gradient.

$g_2=(-1.8)(2)=-3.6$ LaTeX: g_2=(-1.8)(2)=-3.6

Step 6: update after sample 2.

Final answer: batch weight = 1.25; sequential SGD weight = 1.46. SGD changes the weight between samples.

Larger mini-batches use more memory and often make better use of parallel hardware. Smaller batches provide noisier gradient estimates and more updates per epoch. Wall-clock speed and generalization depend on the implementation and problem, so “SGD always converges fastest” is not a safe claim.

Drill 14 Batch and sequential SGD

Question

Question. ŷ = wx, start w = 0, η = 0.1, data(1,1), (2,2). Use mean half squared loss for a batch. Compare one batch update with one sequential SGD epoch.