Deep Learning · MT2 · section

R23 Q2a

R23 selected MT2 practice

Question

Question (10 marks). A CNN with convolution, pooling, dense and output layers receives 127 × 127 images for 5 classes and trains with Adam. Identify learned parameters and tunable hyperparameters.

Answer — step by step

Convolutional kernel entries and filter biases are learned parameters. Dense weight matrices and biases are learned parameters. Pooling, flattening and ordinary activations have no learned weights. The output has 5 class scores and usually softmax with categorical cross-entropy. Adam’s m, v are running optimizer states, not extra predictive parameters.

Hyperparameters include kernel sizes, filter counts, strides, padding, pooling configuration, dense widths, activations, learning rate, Adam β₁/β₂/ε, batch size, epochs and regularization. Architecture choices control representation capacity and resource demand; optimizer settings control update behavior. No filter counts or dense widths are specified, so a unique numerical parameter total cannot be calculated from 127 × 127 and 5 classes alone.

Adam combines an exponentially weighted first moment of gradients with a second moment of squared gradients. Both start at zero; early bias correction compensates for this zero initialization.

m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,\quad v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2LaTeX: m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,\quad v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2
\widehat m_t=\frac{m_t}{1-\beta_1^t},\quad\widehat v_t=\frac{v_t}{1-\beta_2^t},\quad\theta_t=\theta_{t-1}-\eta\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon}LaTeX: \widehat m_t=\frac{m_t}{1-\beta_1^t},\quad\widehat v_t=\frac{v_t}{1-\beta_2^t},\quad\theta_t=\theta_{t-1}-\eta\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon}

Use update count t = 1 for the first update; t = 0 would make the correction denominator zero. m stores direction with momentum-like smoothing; v stores gradient magnitude information; division by its square root adapts coordinate step sizes. Do not multiply the final fraction by another copy of g.

For g₁ = 2, β₁ = 0.9 and β₂ = 0.999, m₁ = 0.2 and v₁ = 0.004. Corrected moments are 2 and 4. With θ₀ = 1, η = 0.1 and negligible ε, θ₁ = 1−0.1 × 2/2 = 0.9.

Compared with ordinary SGD, Adam often provides a useful adaptive starting point and can cope with uneven or sparse gradient scales. It needs two state arrays per parameter, several hyperparameters and validation of generalization; it is not universally more accurate or guaranteed to converge to the global optimum.