F23 Q3b

Answer — step by step

Adam combines an exponentially weighted first moment of gradients with a second moment of squared gradients. Both start at zero; early bias correction compensates for this zero initialization.

LaTeX: m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,\quad v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2

LaTeX: \widehat m_t=\frac{m_t}{1-\beta_1^t},\quad\widehat v_t=\frac{v_t}{1-\beta_2^t},\quad\theta_t=\theta_{t-1}-\eta\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon}

Use update count t = 1 for the first update; t = 0 would make the correction denominator zero. m stores direction with momentum-like smoothing; v stores gradient magnitude information; division by its square root adapts coordinate step sizes. Do not multiply the final fraction by another copy of g.

For g₁ = 2, β₁ = 0.9 and β₂ = 0.999, m₁ = 0.2 and v₁ = 0.004. Corrected moments are 2 and 4. With θ₀ = 1, η = 0.1 and negligible ε, θ₁ = 1−0.1 × 2/2 = 0.9.

Compared with ordinary SGD, Adam often provides a useful adaptive starting point and can cope with uneven or sparse gradient scales. It needs two state arrays per parameter, several hyperparameters and validation of generalization; it is not universally more accurate or guaranteed to converge to the global optimum.