F22 Q5b

Answer — step by step

The learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable.

Diagram: Illustrative full batch and stochastic traces distinguish smoothness from progress. Labels: Loss | Updates | Full batch trend | SGD fluctuations | Illustration only: smoothness alone does not prove better learning.

Batch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory.

Momentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories.

Diagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD.

Momentum uses vnew = βv−ηg and θ new = θ+vnew; consistent gradients accumulate direction, whereas some alternating components cancel. Plain mini-batch descent uses only the current batch gradient. Both can be combined: momentum with mini-batch gradients is a standard coherent choice.