Drill 49 Mixed CNN dimensions softmax and loss

Answer — step by step

Given

Input = 8 × 8 × 3.

Conv: kernel 3, stride 1, padding 1, 2 filters.

Pool: kernel 2, stride 2; flatten; dense layer with 3 outputs.

Logits = [0,1,−1]; true class is the second class (one-hot target [0,1,0]).

To find

Find shapes, parameter counts, probabilities and CCE.

Formula — which one and why

Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.

H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.

LaTeX: N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1

LaTeX: P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}}

LaTeX: n_{\mathrm{flat}}=HWC

LaTeX: P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}}

LaTeX: P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0

Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.

Stable softmax. Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.

zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.

LaTeX: M=\max_j z_j

LaTeX: p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}}

Yaad rakhne ka cue: Largest score ghatao, exp nikalo, total se sabko divide karo.

Categorical cross-entropy. For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.

yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.

LaTeX: L=-\sum_j y_j\ln p_j

LaTeX: L=-\ln p_c

Yaad rakhne ka cue: One-hot mein sirf correct class ki probability ka minus ln.

Solution — step by step

Formulas used

Separate shape from learned parameter count. H, W and C mean height, width and channels. K, P and S mean filter size, padding per side and stride.

LaTeX: N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1

LaTeX: P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}}

LaTeX: n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}}

The +1 counts one bias per receiving filter or neuron. Pooling and flattening add no learned parameters. Softmax converts scores to probabilities; CCE uses the probability of the correct class.

LaTeX: p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\quad c=\max_jz_j,\quad L=-\ln p_{\mathrm{correct}}

Conv size = (8+2−3)+1 = 8; output 8 × 8 × 2; parameters = (3 × 3 × 3+1) × 2 = 56. Pool 4 × 4 × 2; parameters 0. Flatten 32. Dense 3 parameters = (32+1) × 3 = 99. Total = 155.

Step 1 stabilize: subtract the largest logit 1 from every value: [-1, 0, -2]. This does not change softmax.

Step 2 exponentiate: exp values = [0.367879441, 1, 0.135335283].

Step 3 denominator: sum = 1.503214724.

Step 4 divide each value by this same sum: probabilities = [0.244728471, 0.665240956, 0.090030573]. Check: all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability.

Correct class probability = 0.665240956; CCE = −ln(0.665240956) = 0.407605964.