Deep Learning · TEE · section

R23 Q4b

Re examination January 2023 fully solved

Question

Question (10 marks). Estimate parameters of the AlexNet diagram. Use Input 227 × 227 × 3. → Conv K = 11, filters = 96, stride = 4, padding = 0. → Pool K = 3, stride = 2, padding = 0. → Conv K = 5, filters = 256, stride = 1, padding = 2. → Pool K = 3, stride = 2, padding = 0. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 256, stride = 1, padding = 1. → Pool K = 3, stride = 2, padding = 0. → Flatten. → Dense 4096 units. → Dense 4096 units. → Dense 1000 units.

Answer — step by step

Given

Use Input 227 × 227 × 3.

Conv K = 11, filters = 96, stride = 4, padding = 0.

Pool K = 3, stride = 2, padding = 0.

Conv K = 5, filters = 256, stride = 1, padding = 2.

Pool K = 3, stride = 2, padding = 0.

Conv K = 3, filters = 384, stride = 1, padding = 1.

Conv K = 3, filters = 384, stride = 1, padding = 1.

Conv K = 3, filters = 256, stride = 1, padding = 1.

Pool K = 3, stride = 2, padding = 0.

Dense 4096 units.

Dense 4096 units.

Dense 1000 units.

To find

Find every layer shape, its parameter count and the network total.

Formula — which one and why

Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.

H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.

N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1LaTeX: N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1
P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}}LaTeX: P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}}
n_{\mathrm{flat}}=HWCLaTeX: n_{\mathrm{flat}}=HWC
P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}}LaTeX: P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}}
P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0LaTeX: P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0

Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.

Solution — step by step

Use output size \left\lfloor\frac{N+2P-K}{S}\right\rfloor+1LaTeX: \left\lfloor\frac{N+2P-K}{S}\right\rfloor+1, Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.

Layer-by-layer calculation

Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters.

Layer 1 — convolution

Layer 1: what changes and what is learned
Input shape227 × 227 × 3height × width × channels
Convolution96 filters · 11 × 11 · S = 4 · P = 0
Output shape55 × 55 × 96height × width × channels
Weights34,848
Biases96
Trainable parameters34,944

Spatial positions share the filter weights. Output depth equals the number of filters.

Input: 227 × 227 × 3. Use 96 filters of spatial size 11 × 11, stride 4, and padding 0 on each side.

1. Calculate height and width.

N_{\mathrm{out}}=\left\lfloor\frac{227+2(0)-11}{4}\right\rfloor+1=55LaTeX: N_{\mathrm{out}}=\left\lfloor\frac{227+2(0)-11}{4}\right\rfloor+1=55

2. Determine output channels. Each filter produces one feature map. Therefore the output is 55 × 55 × 96.

3. Count trainable values. Each filter receives all 3 input channels. Its weights are reused at every spatial position.

N_{\mathrm{weights}}=11\times11\times3\times96=34{,}848LaTeX: N_{\mathrm{weights}}=11\times11\times3\times96=34{,}848
N_{\mathrm{biases}}=96LaTeX: N_{\mathrm{biases}}=96
P_1=34{,}848+96=34{,}944LaTeX: P_1=34{,}848+96=34{,}944

Layer 2 — pooling

Layer 2: what changes and what is learned
Input shape55 × 55 × 96height × width × channels
Pooling3 × 3 window · S = 2
Output shape27 × 27 × 96height × width × channels
Weights0
Biases0
Trainable parameters0

Spatial dimensions change; the channel count stays the same. Pooling learns no weights.

Input: 55 × 55 × 96. Use a 3 × 3 window, stride 2, and zero padding.

N_{\mathrm{out}}=\left\lfloor\frac{55-3}{2}\right\rfloor+1=27LaTeX: N_{\mathrm{out}}=\left\lfloor\frac{55-3}{2}\right\rfloor+1=27

Pooling acts independently on each channel. The output is 27 × 27 × 96.

Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.

Layer 3 — convolution

Layer 3: what changes and what is learned
Input shape27 × 27 × 96height × width × channels
Convolution256 filters · 5 × 5 · S = 1 · P = 2
Output shape27 × 27 × 256height × width × channels
Weights614,400
Biases256
Trainable parameters614,656

Spatial positions share the filter weights. Output depth equals the number of filters.

Input: 27 × 27 × 96. Use 256 filters of spatial size 5 × 5, stride 1, and padding 2 on each side.

1. Calculate height and width.

N_{\mathrm{out}}=\left\lfloor\frac{27+2(2)-5}{1}\right\rfloor+1=27LaTeX: N_{\mathrm{out}}=\left\lfloor\frac{27+2(2)-5}{1}\right\rfloor+1=27

2. Determine output channels. Each filter produces one feature map. Therefore the output is 27 × 27 × 256.

3. Count trainable values. Each filter receives all 96 input channels. Its weights are reused at every spatial position.

N_{\mathrm{weights}}=5\times5\times96\times256=614{,}400LaTeX: N_{\mathrm{weights}}=5\times5\times96\times256=614{,}400
N_{\mathrm{biases}}=256LaTeX: N_{\mathrm{biases}}=256
P_3=614{,}400+256=614{,}656LaTeX: P_3=614{,}400+256=614{,}656

Layer 4 — pooling

Layer 4: what changes and what is learned
Input shape27 × 27 × 256height × width × channels
Pooling3 × 3 window · S = 2
Output shape13 × 13 × 256height × width × channels
Weights0
Biases0
Trainable parameters0

Spatial dimensions change; the channel count stays the same. Pooling learns no weights.

Input: 27 × 27 × 256. Use a 3 × 3 window, stride 2, and zero padding.

N_{\mathrm{out}}=\left\lfloor\frac{27-3}{2}\right\rfloor+1=13LaTeX: N_{\mathrm{out}}=\left\lfloor\frac{27-3}{2}\right\rfloor+1=13

Pooling acts independently on each channel. The output is 13 × 13 × 256.

Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.

Layer 5 — convolution

Layer 5: what changes and what is learned
Input shape13 × 13 × 256height × width × channels
Convolution384 filters · 3 × 3 · S = 1 · P = 1
Output shape13 × 13 × 384height × width × channels
Weights884,736
Biases384
Trainable parameters885,120

Spatial positions share the filter weights. Output depth equals the number of filters.

Input: 13 × 13 × 256. Use 384 filters of spatial size 3 × 3, stride 1, and padding 1 on each side.

1. Calculate height and width.

N_{\mathrm{out}}=\left\lfloor\frac{13+2(1)-3}{1}\right\rfloor+1=13LaTeX: N_{\mathrm{out}}=\left\lfloor\frac{13+2(1)-3}{1}\right\rfloor+1=13

2. Determine output channels. Each filter produces one feature map. Therefore the output is 13 × 13 × 384.

3. Count trainable values. Each filter receives all 256 input channels. Its weights are reused at every spatial position.

N_{\mathrm{weights}}=3\times3\times256\times384=884{,}736LaTeX: N_{\mathrm{weights}}=3\times3\times256\times384=884{,}736
N_{\mathrm{biases}}=384LaTeX: N_{\mathrm{biases}}=384
P_5=884{,}736+384=885{,}120LaTeX: P_5=884{,}736+384=885{,}120

Layer 6 — convolution

Layer 6: what changes and what is learned
Input shape13 × 13 × 384height × width × channels
Convolution384 filters · 3 × 3 · S = 1 · P = 1
Output shape13 × 13 × 384height × width × channels
Weights1,327,104
Biases384
Trainable parameters1,327,488

Spatial positions share the filter weights. Output depth equals the number of filters.

Input: 13 × 13 × 384. Use 384 filters of spatial size 3 × 3, stride 1, and padding 1 on each side.

1. Calculate height and width.

N_{\mathrm{out}}=\left\lfloor\frac{13+2(1)-3}{1}\right\rfloor+1=13LaTeX: N_{\mathrm{out}}=\left\lfloor\frac{13+2(1)-3}{1}\right\rfloor+1=13

2. Determine output channels. Each filter produces one feature map. Therefore the output is 13 × 13 × 384.

3. Count trainable values. Each filter receives all 384 input channels. Its weights are reused at every spatial position.

N_{\mathrm{weights}}=3\times3\times384\times384=1{,}327{,}104LaTeX: N_{\mathrm{weights}}=3\times3\times384\times384=1{,}327{,}104
N_{\mathrm{biases}}=384LaTeX: N_{\mathrm{biases}}=384
P_6=1{,}327{,}104+384=1{,}327{,}488LaTeX: P_6=1{,}327{,}104+384=1{,}327{,}488

Layer 7 — convolution

Layer 7: what changes and what is learned
Input shape13 × 13 × 384height × width × channels
Convolution256 filters · 3 × 3 · S = 1 · P = 1
Output shape13 × 13 × 256height × width × channels
Weights884,736
Biases256
Trainable parameters884,992

Spatial positions share the filter weights. Output depth equals the number of filters.

Input: 13 × 13 × 384. Use 256 filters of spatial size 3 × 3, stride 1, and padding 1 on each side.

1. Calculate height and width.

N_{\mathrm{out}}=\left\lfloor\frac{13+2(1)-3}{1}\right\rfloor+1=13LaTeX: N_{\mathrm{out}}=\left\lfloor\frac{13+2(1)-3}{1}\right\rfloor+1=13

2. Determine output channels. Each filter produces one feature map. Therefore the output is 13 × 13 × 256.

3. Count trainable values. Each filter receives all 384 input channels. Its weights are reused at every spatial position.

N_{\mathrm{weights}}=3\times3\times384\times256=884{,}736LaTeX: N_{\mathrm{weights}}=3\times3\times384\times256=884{,}736
N_{\mathrm{biases}}=256LaTeX: N_{\mathrm{biases}}=256
P_7=884{,}736+256=884{,}992LaTeX: P_7=884{,}736+256=884{,}992

Layer 8 — pooling

Layer 8: what changes and what is learned
Input shape13 × 13 × 256height × width × channels
Pooling3 × 3 window · S = 2
Output shape6 × 6 × 256height × width × channels
Weights0
Biases0
Trainable parameters0

Spatial dimensions change; the channel count stays the same. Pooling learns no weights.

Input: 13 × 13 × 256. Use a 3 × 3 window, stride 2, and zero padding.

N_{\mathrm{out}}=\left\lfloor\frac{13-3}{2}\right\rfloor+1=6LaTeX: N_{\mathrm{out}}=\left\lfloor\frac{13-3}{2}\right\rfloor+1=6

Pooling acts independently on each channel. The output is 6 × 6 × 256.

Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.

Layer 9 — flatten

Layer 9: what changes and what is learned
Input shape6 × 6 × 256height × width × channels
FlattenRearrange activations
Output shape9,216output features
Weights0
Biases0
Trainable parameters0

The same activation values are rearranged into one vector. No parameter is learned.

Input: 6 × 6 × 256. Rearrange all activation values into one vector.

n_{\mathrm{flat}}=6\times6\times256=9{,}216LaTeX: n_{\mathrm{flat}}=6\times6\times256=9{,}216

Output: a vector with 9,216 entries. Flattening changes the arrangement, so it adds 0 trainable parameters.

Layer 10 — fully connected layer

Layer 10: what changes and what is learned
Input shape9,216input features
Dense layer4,096 neurons
Output shape4,096output features
Weights37,748,736
Biases4,096
Trainable parameters37,752,832

Every output neuron receives every input feature, with one bias per output neuron.

Input: 9,216 features. Output: 4,096 neurons. Every output neuron connects to every input feature and has one bias.

N_{\mathrm{weights}}=9{,}216\times4{,}096=37{,}748{,}736LaTeX: N_{\mathrm{weights}}=9{,}216\times4{,}096=37{,}748{,}736
N_{\mathrm{biases}}=4{,}096LaTeX: N_{\mathrm{biases}}=4{,}096
P_10=37{,}748{,}736+4{,}096=37{,}752{,}832LaTeX: P_10=37{,}748{,}736+4{,}096=37{,}752{,}832

Layer 11 — fully connected layer

Layer 11: what changes and what is learned
Input shape4,096input features
Dense layer4,096 neurons
Output shape4,096output features
Weights16,777,216
Biases4,096
Trainable parameters16,781,312

Every output neuron receives every input feature, with one bias per output neuron.

Input: 4,096 features. Output: 4,096 neurons. Every output neuron connects to every input feature and has one bias.

N_{\mathrm{weights}}=4{,}096\times4{,}096=16{,}777{,}216LaTeX: N_{\mathrm{weights}}=4{,}096\times4{,}096=16{,}777{,}216
N_{\mathrm{biases}}=4{,}096LaTeX: N_{\mathrm{biases}}=4{,}096
P_11=16{,}777{,}216+4{,}096=16{,}781{,}312LaTeX: P_11=16{,}777{,}216+4{,}096=16{,}781{,}312

Layer 12 — fully connected layer

Layer 12: what changes and what is learned
Input shape4,096input features
Dense layer1,000 neurons
Output shape1,000output features
Weights4,096,000
Biases1,000
Trainable parameters4,097,000

Every output neuron receives every input feature, with one bias per output neuron.

Input: 4,096 features. Output: 1,000 neurons. Every output neuron connects to every input feature and has one bias.

N_{\mathrm{weights}}=4{,}096\times1{,}000=4{,}096{,}000LaTeX: N_{\mathrm{weights}}=4{,}096\times1{,}000=4{,}096{,}000
N_{\mathrm{biases}}=1{,}000LaTeX: N_{\mathrm{biases}}=1{,}000
P_12=4{,}096{,}000+1{,}000=4{,}097{,}000LaTeX: P_12=4{,}096{,}000+1{,}000=4{,}097{,}000

Summary of the complete network

LayerSpatial or flatten calculationOutputWeightsBiasesTotal
1 conv\left\lfloor\frac{227+2 \times 0-11}{4}\right\rfloor+1LaTeX: \left\lfloor\frac{227+2 \times 0-11}{4}\right\rfloor+155 x 55 x 9634,8489634,944
2 pool\left\lfloor\frac{55-3}{2}\right\rfloor+1LaTeX: \left\lfloor\frac{55-3}{2}\right\rfloor+127 x 27 x 96000
3 conv\left\lfloor\frac{27+2 \times 2-5}{1}\right\rfloor+1LaTeX: \left\lfloor\frac{27+2 \times 2-5}{1}\right\rfloor+127 x 27 x 256614,400256614,656
4 pool\left\lfloor\frac{27-3}{2}\right\rfloor+1LaTeX: \left\lfloor\frac{27-3}{2}\right\rfloor+113 x 13 x 256000
5 conv\left\lfloor\frac{13+2 \times 1-3}{1}\right\rfloor+1LaTeX: \left\lfloor\frac{13+2 \times 1-3}{1}\right\rfloor+113 x 13 x 384884,736384885,120
6 conv\left\lfloor\frac{13+2 \times 1-3}{1}\right\rfloor+1LaTeX: \left\lfloor\frac{13+2 \times 1-3}{1}\right\rfloor+113 x 13 x 3841,327,1043841,327,488
7 conv\left\lfloor\frac{13+2 \times 1-3}{1}\right\rfloor+1LaTeX: \left\lfloor\frac{13+2 \times 1-3}{1}\right\rfloor+113 x 13 x 256884,736256884,992
8 pool\left\lfloor\frac{13-3}{2}\right\rfloor+1LaTeX: \left\lfloor\frac{13-3}{2}\right\rfloor+16 x 6 x 256000
9 flat6 × 6 × 2569216000
10 dense9216 → 4096409637,748,7364,09637,752,832
11 dense4096 → 4096409616,777,2164,09616,781,312
12 dense4096 → 100010004,096,0001,0004,097,000

Parameter substitutions, in layer order:

Layer 1: (11 x 11 x 3 + 1) x 96 = 34,944.

Layer 3: (5 x 5 x 96 + 1) x 256 = 614,656.

Layer 5: (3 x 3 x 256 + 1) x 384 = 885,120.

Layer 6: (3 x 3 x 384 + 1) x 384 = 1,327,488.

Layer 7: (3 x 3 x 384 + 1) x 256 = 884,992.

Layer 10: (9216 + 1) x 4096 = 37,752,832.

Layer 11: (4096 + 1) x 4096 = 16,781,312.

Layer 12: (4096 + 1) x 1000 = 4,097,000.

Total: 34,944 + 614,656 + 885,120 + 1,327,488 + 884,992 + 37,752,832 + 16,781,312 + 4,097,000 = 62,378,344 trainable parameters. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.

The diagram labels its final pooled map 6 × 6 × 256; a 3 × 3, stride 2 pool from 13 × 13 produces 6. If a small printed stride label is read as 3, it conflicts with the shown 6 × 6 map. The answer uses the consistent classic stride 2 interpretation. Original grouped AlexNet counts are a different architecture assumption.