Deep Learning · TEE · section

R23 Q6a

Re examination January 2023 fully solved

Question

Question (10 marks). Suggest and justify remedies when (1) loss does not minimize after many epochs, (2) gradients are noisy and loss oscillates, (3) loss falls then rises because updates miss a minimum.

Answer — step by step

The learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable.

Illustrative full batch and stochastic traces distinguish smoothness from progress Tap the diagram to enlarge.

Batch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory.

Momentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories.

Diagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD.

Scenario 1: check correctness and gradient magnitudes, then tune learning rate, activation, initialization or optimizer. Scenario 2: consider an appropriate larger mini-batch or momentum; distinguish ordinary stochastic noise from divergence. Scenario 3: reduce learning rate or use a schedule; momentum/Nesterov can help but excessive momentum can worsen overshoot. Judge training and validation curves separately.