Deep Learning · TEE · section

A tiny dataset shows why the methods differ

Optimization algorithms

Question. Train ŷ = wx on samples (1,2) and (2,4), starting at w = 1, with η = 0.1 and mean half squared loss. Compare one batch update with one SGD epoch in the listed order.

Given

Model ŷ = wx; initial w = 1; η = 0.1.

Samples in order: (1,2), (2,4).

Mean half squared loss for the batch.

To find

Compare one batch update with a sequential SGD epoch.

Formula — which one and why

Batch mean versus sequential SGD. A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter.

B = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x.

LaTeX: g_{\mathrm{batch}}=\frac1B\sum_i g_i

LaTeX: g_i=(wx_i-y_i)x_i

LaTeX: w_{\mathrm{new}}=w-\eta g

Yaad rakhne ka cue: Batch: same old weight par average; SGD: har sample ke baad weight change.

Solution — step by step

One batch update

Step 1: predict sample 1 with the original weight.

LaTeX: \hat y_1=1(1)=1

Step 2: calculate its error.

LaTeX: e_1=1-2=-1

Step 3: multiply its error by its input.

LaTeX: g_1=(-1)(1)=-1

Step 4: predict sample 2 with the same original weight.

LaTeX: \hat y_2=1(2)=2

Step 5: calculate its error.

LaTeX: e_2=2-4=-2

Step 6: multiply its error by its input.

LaTeX: g_2=(-2)(2)=-4

Step 7: average the two gradients.

LaTeX: g=\frac{-1+(-4)}2=-2.5

Step 8: update once.

LaTeX: w_{\mathrm{new}}=1-0.1(-2.5)=1.25

Diagram: This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation.. Labels: Same 6 samples, different update groups | Same 6 samples, different update groups | Each outlined group produces one gradient update. | Batch GD | 1 | 2 | 3 | 4 | 5 | 6 | 1 update | Mini-batch, size 2 | 1 | 2 | 3 | 4 | 5 | 6 | 3 updates | SGD, size 1 | 1 | 2 | 3 | 4 | 5 | 6 | 6 updates | One epoch visits all six samples. The number of updates depends on the group | size.

One sequential SGD epoch

Step 1: use the first sample gradient.

LaTeX: g_1=-1

Step 2: update after the first sample.

LaTeX: w_1=1-0.1(-1)=1.1

Step 3: predict sample 2 using the changed weight.

LaTeX: \hat y_2=1.1(2)=2.2

Step 4: find sample 2 error.

LaTeX: e_2=2.2-4=-1.8

Step 5: find the new sample 2 gradient.

LaTeX: g_2=(-1.8)(2)=-3.6

Step 6: update after sample 2.

LaTeX: w_2=1.1-0.1(-3.6)=1.46

Final answer: batch weight = 1.25; sequential SGD weight = 1.46. SGD changes the weight between samples.

Larger mini-batches use more memory and often make better use of parallel hardware. Smaller batches provide noisier gradient estimates and more updates per epoch. Wall-clock speed and generalization depend on the implementation and problem, so “SGD always converges fastest” is not a safe claim.