# Model answer: batch, stochastic and mini-batch training

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s5

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s5/

## Question

**Question.** Differentiate batch gradient descent, stochastic gradient descent and mini-batch gradient descent. Explain an epoch, a batch and an iteration.

## Answer — step by step

All three methods update model parameters using gradients. Their difference is the number of training examples used to estimate the gradient for one update.

![All methods visit the same six examples. Their grouping determines how often they update.](https://deep-learning-study-room.vercel.app/assets/figures/v_batches.svg)

Diagram: All methods visit the same six examples. Their grouping determines how often they update.. Labels: Same 6 samples, different update groups | Same 6 samples, different update groups | Each outlined group produces one gradient update. | Batch GD | 1 | 2 | 3 | 4 | 5 | 6 | 1 update | Mini-batch, size 2 | 1 | 2 | 3 | 4 | 5 | 6 | 3 updates | SGD, size 1 | 1 | 2 | 3 | 4 | 5 | 6 | 6 updates | One epoch visits all six samples. The number of updates depends on the group | size.

All methods visit the same six examples. Their grouping determines how often they update. Tap the diagram to enlarge.

A **batch** is the group of examples used together for an update. An **iteration**, in this context, is one optimizer update. An **epoch** is one complete pass through the chosen training dataset. One epoch usually contains several iterations.

| Basis | Batch GD | Stochastic GD | Mini-batch GD |
| --- | --- | --- | --- |
| Examples in one gradient | Entire training dataset | One example | A smaller group of B examples |
| Updates per epoch | One | N | Ceiling of N/B if the remainder is retained |
| Gradient behavior | Full-data gradient | Noisy sample gradient | Intermediate averaging |
| Practical implication | Updates can be expensive on large datasets | Many small, sequential updates | Common compromise for vectorized computation |

### Example to distinguish the three counts

With 105 examples and batch size 20, five batches use 100 examples and a sixth uses the remaining 5. Therefore one epoch has 6 mini-batch updates. Three epochs give 18 updates. It would be 5 updates per epoch only if the problem explicitly discarded the incomplete batch.

The basic parameter update is unchanged:



$$
\theta_{\mathrm{new}}=\theta_{\mathrm{old}}-\eta g
$$



For batch or mini-batch training, g normally represents the gradient of the stated aggregate loss over that group. For sequential SGD, update after each example and use the new parameter for the next example. Consequently, one SGD epoch need not equal one batch update even when both use the same examples.