Deep Learning · TEE · section
R23 Q1d
Re examination January 2023 fully solved
Question
Question (4 marks). Evaluate how mini-batch size affects optimization and parallel processing.
Answer — step by step
A mini-batch forms one gradient from B examples. Smaller B usually gives noisier gradients, lower activation memory and more updates per epoch. Larger B averages more examples, often uses vectorized hardware more efficiently up to resource limits, but needs more memory and gives fewer updates per epoch. N examples produce ceil(N/B) updates if the last partial batch is kept. Too-large B does not guarantee faster wall-clock progress or better generalization. Tune batch size with learning rate and resource constraints. Computation within a batch can be parallelized; successive optimizer steps remain dependent on the updated parameters.