# A tiny dataset shows why the methods differ

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s6

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s6/

**Question.** Train ŷ = wx on samples (1,2) and (2,4), starting at w = 1, with η = 0.1 and mean half squared loss. Compare one batch update with one SGD epoch in the listed order.

### Given

Model ŷ = wx; initial w = 1; η = 0.1.

Samples in order: (1,2), (2,4).

Mean half squared loss for the batch.

### To find

Compare one batch update with a sequential SGD epoch.

### Formula — which one and why

**Batch mean versus sequential SGD.** A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter.

B = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x.



$$
g_{\mathrm{batch}}=\frac1B\sum_i g_i
$$





$$
g_i=(wx_i-y_i)x_i
$$





$$
w_{\mathrm{new}}=w-\eta g
$$



**Yaad rakhne ka cue:** Batch: same old weight par average; SGD: har sample ke baad weight change.

### Solution — step by step

### One batch update

**Step 1: predict sample 1 with the original weight.**



$$
\hat y_1=1(1)=1
$$



**Step 2: calculate its error.**



$$
e_1=1-2=-1
$$



**Step 3: multiply its error by its input.**



$$
g_1=(-1)(1)=-1
$$



**Step 4: predict sample 2 with the same original weight.**



$$
\hat y_2=1(2)=2
$$



**Step 5: calculate its error.**



$$
e_2=2-4=-2
$$



**Step 6: multiply its error by its input.**



$$
g_2=(-2)(2)=-4
$$



**Step 7: average the two gradients.**



$$
g=\frac{-1+(-4)}2=-2.5
$$



**Step 8: update once.**



$$
w_{\mathrm{new}}=1-0.1(-2.5)=1.25
$$



![This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation.](https://deep-learning-study-room.vercel.app/assets/figures/v_batches.svg)

Diagram: This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation.. Labels: Same 6 samples, different update groups | Same 6 samples, different update groups | Each outlined group produces one gradient update. | Batch GD | 1 | 2 | 3 | 4 | 5 | 6 | 1 update | Mini-batch, size 2 | 1 | 2 | 3 | 4 | 5 | 6 | 3 updates | SGD, size 1 | 1 | 2 | 3 | 4 | 5 | 6 | 6 updates | One epoch visits all six samples. The number of updates depends on the group | size.

This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation. Tap the diagram to enlarge.

### One sequential SGD epoch

**Step 1: use the first sample gradient.**



$$
g_1=-1
$$



**Step 2: update after the first sample.**



$$
w_1=1-0.1(-1)=1.1
$$



**Step 3: predict sample 2 using the changed weight.**



$$
\hat y_2=1.1(2)=2.2
$$



**Step 4: find sample 2 error.**



$$
e_2=2.2-4=-1.8
$$



**Step 5: find the new sample 2 gradient.**



$$
g_2=(-1.8)(2)=-3.6
$$



**Step 6: update after sample 2.**



$$
w_2=1.1-0.1(-3.6)=1.46
$$



**Final answer:** batch weight = 1.25; sequential SGD weight = 1.46. SGD changes the weight between samples.

Larger mini-batches use more memory and often make better use of parallel hardware. Smaller batches provide noisier gradient estimates and more updates per epoch. Wall-clock speed and generalization depend on the implementation and problem, so “SGD always converges fastest” is not a safe claim.