Deep Learning · TEE · section

Drill 17 Incomplete mini batch

Numerical practice from direct to mixed

Question

Question. N = 103, B = 20, epochs = 3. Count updates for batch GD, SGD, mini-batch keeping the remainder, and mini-batch dropping it.

Answer — step by step

Given

N = 103 samples; B = 20 samples per mini-batch.

E = 3 epochs.

Compare both remainder policies: keep or drop the final 3 examples.

To find

Find total updates for full-batch GD, SGD, and both mini-batch policies.

Formula — which one and why

Number of updates. An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.

N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.

U_{\mathrm{batch}}=ELaTeX: U_{\mathrm{batch}}=E
U_{\mathrm{SGD}}=ENLaTeX: U_{\mathrm{SGD}}=EN
U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceilLaTeX: U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil
U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloorLaTeX: U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor

Yaad rakhne ka cue: Kitne batches process hue, utne updates hue.

Solution — step by step

Meaning and formulas

An epoch means one pass through the dataset. An update happens once per batch used. N is the number of samples, B is the chosen batch size, and E is the number of epochs.

U_{\mathrm{batch}}=E,\qquad U_{\mathrm{SGD}}=NELaTeX: U_{\mathrm{batch}}=E,\qquad U_{\mathrm{SGD}}=NE
U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil,\qquad U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloorLaTeX: U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil,\qquad U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor

Ceiling rounds up to the next integer; floor rounds down. The final incomplete batch needs an update only if it is retained.

Step 1: batch GD

All 103 samples form one batch per epoch.

U_{\mathrm{batch}}=3\times1=3LaTeX: U_{\mathrm{batch}}=3\times1=3

Step 2: SGD

Each sample gets its own update.

U_{\mathrm{SGD}}=103\times3=309LaTeX: U_{\mathrm{SGD}}=103\times3=309

Step 3: mini-batches with the remainder

103=5\times20+3LaTeX: 103=5\times20+3

The batch sizes are 20, 20, 20, 20, 20 and 3. Six batches per epoch give:

U_{\mathrm{keep}}=6\times3=18LaTeX: U_{\mathrm{keep}}=6\times3=18

Step 4: mini-batches without the remainder

Only the five complete batches are used each epoch.

U_{\mathrm{drop}}=5\times3=15LaTeX: U_{\mathrm{drop}}=5\times3=15

Final answer: 3, 309, 18 and 15 updates respectively.