# Batch stochastic and mini batch gradient descent

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s4

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s4/

**Definition.** Batch, stochastic and mini-batch gradient descent differ in the subset of training examples used to estimate the gradient for each parameter update. Their update equation is the same; their gradient estimates and update frequency differ.

### Explanation and working

All three use the same update idea. They differ in how much data contributes to one gradient.

![Count outlined groups, not individual cells, to count gradient updates in one epoch.](https://deep-learning-study-room.vercel.app/assets/figures/v_batches.svg)

Diagram: Count outlined groups, not individual cells, to count gradient updates in one epoch.. Labels: Same 6 samples, different update groups | Same 6 samples, different update groups | Each outlined group produces one gradient update. | Batch GD | 1 | 2 | 3 | 4 | 5 | 6 | 1 update | Mini-batch, size 2 | 1 | 2 | 3 | 4 | 5 | 6 | 3 updates | SGD, size 1 | 1 | 2 | 3 | 4 | 5 | 6 | 6 updates | One epoch visits all six samples. The number of updates depends on the group | size.

Count outlined groups, not individual cells, to count gradient updates in one epoch. Tap the diagram to enlarge.

| Method | Data for one update | Updates in one epoch with N samples | Typical behaviour |
| --- | --- | --- | --- |
| Batch GD | All N samples | 1 | Stable full-data gradient; costly updates |
| Stochastic GD | One sample | N | Frequent noisy updates |
| Mini-batch GD | B samples | Ceiling of N/B if final partial batch is retained | Efficient parallel calculation with moderate noise |

**Question.** N = 1,000 and B = 100. Find updates per epoch.

### Given

N = 1,000 examples; mini-batch size B = 100; one epoch.

### To find

Count full-batch, stochastic and mini-batch updates per epoch.

### Formula — which one and why

**Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.

N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.



$$
U_{\mathrm{batch}}=E
$$





$$
U_{\mathrm{SGD}}=EN
$$





$$
U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil
$$





$$
U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor
$$



**Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue.

### Solution — step by step

**Step 1: count full-batch updates.**

All 1,000 samples form one group, so one epoch has 1 update.

**Step 2: count stochastic updates.**

One sample per update gives 1,000 updates per epoch.

**Step 3: count mini-batches.**



$$
\frac{N}{B}=\frac{1000}{100}=10
$$



**Final answer:** batch = 1; SGD = 1,000; mini-batch = 10 updates per epoch.

### Mini-batch calculation with a remainder

**Question. A dataset contains 105 samples and the mini-batch size is 20. Keeping the incomplete last batch, find the updates per epoch.**

### Given

N = 105; B = 20; retain the final incomplete batch.

### To find

The number of updates per epoch.

### Formula — which one and why

Every processed group causes one update. A remaining nonempty group also counts.

The ceiling symbol rounds upward to the next integer.



$$
U=\lceil N/B\rceil
$$



**Yaad rakhne ka cue:** Bacha hua batch process kiya toh ek aur update.

### Solution — step by step

**Step 1: count complete batches.**



$$
5(20)=100
$$



**Step 2: count remaining examples.**



$$
105-100=5
$$



**Step 3: include the nonempty last batch.**



$$
U=5+1=6
$$



**Final answer:** 6 updates. If a question explicitly drops the final incomplete batch, it instead gives 5.