# TEE paper rehearsal

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/mocks/TEE

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/mocks/tee/

3 hours · 100 marks. Q1 compulsory + any four from Q2–Q7. Historical paper rehearsal.

## F25 Q1c

## Question

**Question (5 marks).** Which regularizer leads to sparsity and why? For an MNIST CNN with high training accuracy and lower validation accuracy, explain how dropout works during training and how it can improve generalization.

## Answer — step by step

Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.



$$
J_{L1}=L+\lambda\sum_j|w_j|,\qquad J_{L2}=L+\frac{\lambda}{2}\sum_jw_j^2
$$



L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.

Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.

## F25 Q1d

## Question

**Question (5 marks).** Describe sequence processing, context and variable-length sentences in RNNs. Explain their use in language translation, as illustrated by the historical reference to Google Translate.

## Answer — step by step

A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.



$$
h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad p_t=\operatorname{softmax}(W_{hy}h_t+b_y)
$$



![Current input and previous state jointly determine the new state](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg)

Diagram: Current input and previous state jointly determine the new state. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ←

Current input and previous state jointly determine the new state Tap the diagram to enlarge.

At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.

For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.

A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.

## F25 Q2b

## Question

**Question (10 marks).** Explain how ResNet skip connections influence gradient flow, feature reuse and vanishing gradients. Compare InceptionNet and AlexNet learning and feature extraction.

## Answer — step by step

A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.

![The shortcut adds the input to the learned residual before the next stage](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg)

Diagram: The shortcut adds the input to the learned residual before the next stage. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate.

The shortcut adds the input to the learned residual before the next stage Tap the diagram to enlarge.



$$
\frac{\partial L}{\partial x}=\frac{\partial L}{\partial y}\left(\frac{\partial F}{\partial x}+I\right)
$$



The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.

Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].

An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.

![Parallel branches preserve height and width then concatenate channels](https://deep-learning-study-room.vercel.app/assets/figures/inception.svg)

Diagram: Parallel branches preserve height and width then concatenate channels. Labels: Input | 1 × 1 | 1 × 1 reduce | 3 × 3 | 1 × 1 reduce | 5 × 5 | 3 × 3 pool | 1 × 1 | Concatenate channels | All branch height/width values must match.

Parallel branches preserve height and width then concatenate channels Tap the diagram to enlarge.

A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.

For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.

AlexNet learns features along a mainly sequential stack of convolutional layers. Inception learns several spatial scales in parallel within a module and combines their channels. Both learn filter weights through backpropagation; their structural organization differs.

## F25 Q3b

## Question

**Question (10 marks).** Design face-recognition attendance using a CNN when the supplied dataset is very small. Discuss ways to address the issue and explain transfer learning.

## Answer — step by step

Transfer learning reuses features learned on a source task to solve a related target task. For a small face-attendance dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset.

1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. For evaluation on enrolled identities, use different capture sessions in each split.

2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes.

3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy.

4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage.

5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization.

6. Evaluate on held-out sessions, inspect per-class errors, and reject low-confidence/unregistered cases if the attendance design requires that behavior.

Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition.

## F25 Q4a

## Question

**Question (10 marks).** For N = 1,000 samples and batch size 100, count weight updates per epoch for batch GD, SGD and mini-batch GD. For J(θ) = 0.5 θ², η = 0.1, momentum 0.9, v₀ = 0, θ₀ = 1, compute two momentum iterations. List the required momentum-training hyperparameters.

## Answer — step by step

### Given

N = 1,000; mini-batch size = 100.

J(θ) = 0.5θ²; θ₀ = 1; v₀ = 0; η = 0.1; β = 0.9.

Use signed velocity.

### To find

Count updates per epoch for batch, stochastic and mini-batch GD; then perform two momentum iterations.

### Formula — which one and why

**Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.

N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.



$$
U_{\mathrm{batch}}=E
$$





$$
U_{\mathrm{SGD}}=EN
$$





$$
U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil
$$





$$
U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor
$$



**Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue.

**Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.

v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.



$$
v_t=\beta v_{t-1}-\eta g_t
$$





$$
\theta_t=\theta_{t-1}+v_t
$$



**Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo.

### Solution — step by step

**Step 1: count full-batch updates.**

All 1,000 samples contribute to one update: 1 per epoch.

**Step 2: count stochastic updates.**

One update per sample: 1,000 per epoch.

**Step 3: count mini-batches.**



$$
1000/100=10
$$



Use signed velocity vₜ = βvₜ₋₁−ηgₜ and θₜ = θₜ₋₁+vₜ; g(θ) = θ.

### Momentum iteration 1

**Step 1: evaluate the gradient at the current parameter.**



$$
g_1=1
$$



**Step 2: retain the previous displacement.**



$$
0.9(0)=0
$$



**Step 3: scale the current gradient.**



$$
0.1(1)=0.1
$$



**Step 4: combine to obtain the new signed velocity.**



$$
v_1=0-0.1=-0.1
$$



**Step 5: add new velocity to the actual old parameter.**



$$
\theta_1=1+(-0.1)=0.9
$$



### Momentum iteration 2

**Step 1: evaluate the gradient at the current parameter.**



$$
g_2=0.9
$$



**Step 2: retain the previous displacement.**



$$
0.9(-0.1)=-0.09
$$



**Step 3: scale the current gradient.**



$$
0.1(0.9)=0.09
$$



**Step 4: combine to obtain the new signed velocity.**



$$
v_2=-0.09-0.09=-0.18
$$



**Step 5: add new velocity to the actual old parameter.**



$$
\theta_2=0.9+(-0.18)=0.72
$$



**Hyperparameters:**  the basic momentum update has two main tunable coefficients, learning rate η and momentum β. A full training run also requires choices such as epochs and batch size (one for literal SGD), and may include a learning-rate schedule. There is no universal single count for all training configurations; state what is counted. Initial θ and v are initialization values, not learned outputs of the first update.

## F25 Q5b

## Question

**Question (10 marks).** Explain AdaGrad, give its update rule and discuss advantages and disadvantages compared with standard SGD.

## Answer — step by step

AdaGrad gives each parameter its own effective learning rate based on the sum of its past squared gradients. Start G₀ = 0, choose η and a small positive ε, then apply elementwise:



$$
G_t=G_{t-1}+g_t^2,\quad\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon}
$$



Frequently updated coordinates accumulate a larger G and take smaller steps. Rarely active coordinates keep a relatively larger effective learning rate, which can be useful for sparse features. The square removes the sign in the accumulator; the unsquared current gradient still determines the direction of the parameter change.

Example with θ₀ = 1, η = 0.1, gradients 2 then 1 and negligible ε: first G = 4, denominator = 2, change = 0.1 × 2/2 = 0.1, θ₁ = 0.9. Next G = 5, denominator = √5 = 2.236068, change = 0.1/2.236068 = 0.04472136, θ₂ = 0.85527864.

Advantages over ordinary SGD include coordinatewise adaptation and usefulness for sparse gradients. Disadvantages include extra accumulator memory and continuously shrinking effective learning rates: G never decreases, so learning may become too slow. AdaGrad still requires a base learning rate and does not guarantee a global minimum. RMSProp addresses indefinite accumulation by replacing the sum with an exponentially weighted average.

## F25 Q6a

## Question

**Question (10 marks).** Calculate every activation shape and trainable-parameter count for: Input 128 × 128 × 3. → Conv K = 3, filters = 16, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 3, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 240 units. → Dense 120 units. → Dense 5 units. Also convolve the input A with the kernel K shown below, valid padding and stride 2.

Given matrices

Input A

Input A

|  |  |  |  |  |
| --- | --- | --- | --- | --- |
| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |

Kernel K

Kernel K

|  |  |  |
| --- | --- | --- |
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |

Stride 2; P = 0 (no padding).

## Answer — step by step

### Given

Input 128 × 128 × 3.

Conv K = 3, filters = 16, stride = 1, padding = 0.

Pool K = 2, stride = 2, padding = 0.

Conv K = 3, filters = 32, stride = 1, padding = 0.

Pool K = 2, stride = 2, padding = 0.

Dense 240 units.

Dense 120 units.

Dense 5 units.

Also convolve the input A with the kernel K shown below, valid padding and stride 2.

### To find

Find every layer shape, its parameter count and the network total. Find every convolution output cell.

### Formula — which one and why

**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.

H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.



$$
N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1
$$





$$
P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}}
$$





$$
n_{\mathrm{flat}}=HWC
$$





$$
P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}}
$$





$$
P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0
$$



**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.

**Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat.

X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention.



$$
Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b
$$



**Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar.

### Solution — step by step

Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs.

Use output size  $\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1$ , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.

### Layer-by-layer calculation

Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters.

### Layer 1 — convolution

Layer 1: what changes and what is learned

Input shape**128 × 128 × 3**height × width × channels

→

Convolution**16 filters · 3 × 3 · S = 1 · P = 0**

→

Output shape**126 × 126 × 16**height × width × channels

Weights**432**

+

Biases**16**

=

Trainable parameters**448**

Spatial positions share the filter weights. Output depth equals the number of filters.

Input: **128 × 128 × 3**. Use 16 filters of spatial size 3 × 3, stride 1, and padding 0 on each side.

**1. Calculate height and width.**



$$
N_{\mathrm{out}}=\left\lfloor\frac{128+2(0)-3}{1}\right\rfloor+1=126
$$



**2. Determine output channels.** Each filter produces one feature map. Therefore the output is **126 × 126 × 16**.

**3. Count trainable values.** Each filter receives all 3 input channels. Its weights are reused at every spatial position.



$$
N_{\mathrm{weights}}=3\times3\times3\times16=432
$$





$$
N_{\mathrm{biases}}=16
$$





$$
P_1=432+16=448
$$



### Layer 2 — pooling

Layer 2: what changes and what is learned

Input shape**126 × 126 × 16**height × width × channels

→

Pooling**2 × 2 window · S = 2**

→

Output shape**63 × 63 × 16**height × width × channels

Weights**0**

+

Biases**0**

=

Trainable parameters**0**

Spatial dimensions change; the channel count stays the same. Pooling learns no weights.

Input: **126 × 126 × 16**. Use a 2 × 2 window, stride 2, and zero padding.



$$
N_{\mathrm{out}}=\left\lfloor\frac{126-2}{2}\right\rfloor+1=63
$$



Pooling acts independently on each channel. The output is **63 × 63 × 16**.

**Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases.

### Layer 3 — convolution

Layer 3: what changes and what is learned

Input shape**63 × 63 × 16**height × width × channels

→

Convolution**32 filters · 3 × 3 · S = 1 · P = 0**

→

Output shape**61 × 61 × 32**height × width × channels

Weights**4,608**

+

Biases**32**

=

Trainable parameters**4,640**

Spatial positions share the filter weights. Output depth equals the number of filters.

Input: **63 × 63 × 16**. Use 32 filters of spatial size 3 × 3, stride 1, and padding 0 on each side.

**1. Calculate height and width.**



$$
N_{\mathrm{out}}=\left\lfloor\frac{63+2(0)-3}{1}\right\rfloor+1=61
$$



**2. Determine output channels.** Each filter produces one feature map. Therefore the output is **61 × 61 × 32**.

**3. Count trainable values.** Each filter receives all 16 input channels. Its weights are reused at every spatial position.



$$
N_{\mathrm{weights}}=3\times3\times16\times32=4{,}608
$$





$$
N_{\mathrm{biases}}=32
$$





$$
P_3=4{,}608+32=4{,}640
$$



### Layer 4 — pooling

Layer 4: what changes and what is learned

Input shape**61 × 61 × 32**height × width × channels

→

Pooling**2 × 2 window · S = 2**

→

Output shape**30 × 30 × 32**height × width × channels

Weights**0**

+

Biases**0**

=

Trainable parameters**0**

Spatial dimensions change; the channel count stays the same. Pooling learns no weights.

Input: **61 × 61 × 32**. Use a 2 × 2 window, stride 2, and zero padding.



$$
N_{\mathrm{out}}=\left\lfloor\frac{61-2}{2}\right\rfloor+1=30
$$



Pooling acts independently on each channel. The output is **30 × 30 × 32**.

**Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases.

The division gives 29.5; take its floor before adding 1. A partial window outside the unpadded input is not included.

### Layer 5 — flatten

Layer 5: what changes and what is learned

Input shape**30 × 30 × 32**height × width × channels

→

Flatten**Rearrange activations**

→

Output shape**28,800**output features

Weights**0**

+

Biases**0**

=

Trainable parameters**0**

The same activation values are rearranged into one vector. No parameter is learned.

Input: **30 × 30 × 32**. Rearrange all activation values into one vector.



$$
n_{\mathrm{flat}}=30\times30\times32=28{,}800
$$



Output: a vector with **28,800 entries**. Flattening changes the arrangement, so it adds **0 trainable parameters**.

### Layer 6 — fully connected layer

Layer 6: what changes and what is learned

Input shape**28,800**input features

→

Dense layer**240 neurons**

→

Output shape**240**output features

Weights**6,912,000**

+

Biases**240**

=

Trainable parameters**6,912,240**

Every output neuron receives every input feature, with one bias per output neuron.

Input: **28,800 features**. Output: **240 neurons**. Every output neuron connects to every input feature and has one bias.



$$
N_{\mathrm{weights}}=28{,}800\times240=6{,}912{,}000
$$





$$
N_{\mathrm{biases}}=240
$$





$$
P_6=6{,}912{,}000+240=6{,}912{,}240
$$



### Layer 7 — fully connected layer

Layer 7: what changes and what is learned

Input shape**240**input features

→

Dense layer**120 neurons**

→

Output shape**120**output features

Weights**28,800**

+

Biases**120**

=

Trainable parameters**28,920**

Every output neuron receives every input feature, with one bias per output neuron.

Input: **240 features**. Output: **120 neurons**. Every output neuron connects to every input feature and has one bias.



$$
N_{\mathrm{weights}}=240\times120=28{,}800
$$





$$
N_{\mathrm{biases}}=120
$$





$$
P_7=28{,}800+120=28{,}920
$$



### Layer 8 — fully connected layer

Layer 8: what changes and what is learned

Input shape**120**input features

→

Dense layer**5 neurons**

→

Output shape**5**output features

Weights**600**

+

Biases**5**

=

Trainable parameters**605**

Every output neuron receives every input feature, with one bias per output neuron.

Input: **120 features**. Output: **5 neurons**. Every output neuron connects to every input feature and has one bias.



$$
N_{\mathrm{weights}}=120\times5=600
$$





$$
N_{\mathrm{biases}}=5
$$





$$
P_8=600+5=605
$$



### Summary of the complete network

| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |
| --- | --- | --- | --- | --- | --- |
| 1 conv |  $\left\lfloor\frac{128+2 \times  0-3}{1}\right\rfloor+1$  | 126 x 126 x 16 | 432 | 16 | 448 |
| 2 pool |  $\left\lfloor\frac{126-2}{2}\right\rfloor+1$  | 63 x 63 x 16 | 0 | 0 | 0 |
| 3 conv |  $\left\lfloor\frac{63+2 \times  0-3}{1}\right\rfloor+1$  | 61 x 61 x 32 | 4,608 | 32 | 4,640 |
| 4 pool |  $\left\lfloor\frac{61-2}{2}\right\rfloor+1$  | 30 x 30 x 32 | 0 | 0 | 0 |
| 5 flat | 30 × 30 × 32 | 28800 | 0 | 0 | 0 |
| 6 dense | 28800 → 240 | 240 | 6,912,000 | 240 | 6,912,240 |
| 7 dense | 240 → 120 | 120 | 28,800 | 120 | 28,920 |
| 8 dense | 120 → 5 | 5 | 600 | 5 | 605 |

**Parameter substitutions, in layer order:**

Layer 1: (3 x 3 x 3 + 1) x 16 = 448.

Layer 3: (3 x 3 x 16 + 1) x 32 = 4,640.

Layer 6: (28800 + 1) x 240 = 6,912,240.

Layer 7: (240 + 1) x 120 = 28,920.

Layer 8: (120 + 1) x 5 = 605.

**Total:**  448 + 4,640 + 6,912,240 + 28,920 + 605 = **6,946,853 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.

**Separate convolution calculation:**

**Step 1 shape:**   $\left\lfloor\frac{5+2 \times  0-3}{2}\right\rfloor+1$  = 2 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel.

### Output position (1, 1)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (1, 1) into one output cell

Input rows **1–3**, columns **1–3**. The window moves **2** cells per output step.

1. Select the input window

1. Select the input window

|  |  |  |  |  |
| --- | --- | --- | --- | --- |
| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |

Only the outlined cells belong to this output.

2. Match the kernel

2. Match the kernel

|  |  |  |
| --- | --- | --- |
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell

3. Multiply cell by cell

|  |  |  |
| --- | --- | --- |
| 3 | 0 | −2 |
| 4 | 0 | −4 |
| 3 | 0 | −1 |

Each cell is input × matching kernel coefficient.

4. Add the products row by row

Row 1**3 + 0 + (−2) = 1**

Row 2**4 + 0 + (−4) = 0**

Row 3**3 + 0 + (−1) = 2**

**1 + 0 + 2 = 3**

5. Place it in the output

5. Place it in the output

|  |  |
| --- | --- |
| 3 | 5 |
| 3 | −7 |

Output row 1, column 1.

**Window row 1.**



$$
R_1=(3)\times(1)+(1)\times(0)+(2)\times(-1)=1
$$



**Window row 2.**



$$
R_2=(2)\times(2)+(1)\times(0)+(2)\times(-2)=0
$$



**Window row 3.**



$$
R_3=(3)\times(1)+(2)\times(0)+(1)\times(-1)=2
$$



**Add the rows to obtain one output cell.**



$$
Y_{1,1}=(1)+(0)+(2)=3
$$



### Output position (1, 2)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (1, 2) into one output cell

Input rows **1–3**, columns **3–5**. The window moves **2** cells per output step.

1. Select the input window

1. Select the input window

|  |  |  |  |  |
| --- | --- | --- | --- | --- |
| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |

Only the outlined cells belong to this output.

2. Match the kernel

2. Match the kernel

|  |  |  |
| --- | --- | --- |
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell

3. Multiply cell by cell

|  |  |  |
| --- | --- | --- |
| 2 | 0 | −1 |
| 4 | 0 | 0 |
| 1 | 0 | −1 |

Each cell is input × matching kernel coefficient.

4. Add the products row by row

Row 1**2 + 0 + (−1) = 1**

Row 2**4 + 0 + 0 = 4**

Row 3**1 + 0 + (−1) = 0**

**1 + 4 + 0 = 5**

5. Place it in the output

5. Place it in the output

|  |  |
| --- | --- |
| 3 | 5 |
| 3 | −7 |

Output row 1, column 2.

**Window row 1.**



$$
R_1=(2)\times(1)+(2)\times(0)+(1)\times(-1)=1
$$



**Window row 2.**



$$
R_2=(2)\times(2)+(1)\times(0)+(0)\times(-2)=4
$$



**Window row 3.**



$$
R_3=(1)\times(1)+(0)\times(0)+(1)\times(-1)=0
$$



**Add the rows to obtain one output cell.**



$$
Y_{1,2}=(1)+(4)+(0)=5
$$



### Output position (2, 1)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (2, 1) into one output cell

Input rows **3–5**, columns **1–3**. The window moves **2** cells per output step.

1. Select the input window

1. Select the input window

|  |  |  |  |  |
| --- | --- | --- | --- | --- |
| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |

Only the outlined cells belong to this output.

2. Match the kernel

2. Match the kernel

|  |  |  |
| --- | --- | --- |
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell

3. Multiply cell by cell

|  |  |  |
| --- | --- | --- |
| 3 | 0 | −1 |
| 0 | 0 | 0 |
| 1 | 0 | 0 |

Each cell is input × matching kernel coefficient.

4. Add the products row by row

Row 1**3 + 0 + (−1) = 2**

Row 2**0 + 0 + 0 = 0**

Row 3**1 + 0 + 0 = 1**

**2 + 0 + 1 = 3**

5. Place it in the output

5. Place it in the output

|  |  |
| --- | --- |
| 3 | 5 |
| 3 | −7 |

Output row 2, column 1.

**Window row 1.**



$$
R_1=(3)\times(1)+(2)\times(0)+(1)\times(-1)=2
$$



**Window row 2.**



$$
R_2=(0)\times(2)+(3)\times(0)+(0)\times(-2)=0
$$



**Window row 3.**



$$
R_3=(1)\times(1)+(1)\times(0)+(0)\times(-1)=1
$$



**Add the rows to obtain one output cell.**



$$
Y_{2,1}=(2)+(0)+(1)=3
$$



### Output position (2, 2)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (2, 2) into one output cell

Input rows **3–5**, columns **3–5**. The window moves **2** cells per output step.

1. Select the input window

1. Select the input window

|  |  |  |  |  |
| --- | --- | --- | --- | --- |
| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |

Only the outlined cells belong to this output.

2. Match the kernel

2. Match the kernel

|  |  |  |
| --- | --- | --- |
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell

3. Multiply cell by cell

|  |  |  |
| --- | --- | --- |
| 1 | 0 | −1 |
| 0 | 0 | −4 |
| 0 | 0 | −3 |

Each cell is input × matching kernel coefficient.

4. Add the products row by row

Row 1**1 + 0 + (−1) = 0**

Row 2**0 + 0 + (−4) = −4**

Row 3**0 + 0 + (−3) = −3**

**0 + (−4) + (−3) = −7**

5. Place it in the output

5. Place it in the output

|  |  |
| --- | --- |
| 3 | 5 |
| 3 | −7 |

Output row 2, column 2.

**Window row 1.**



$$
R_1=(1)\times(1)+(0)\times(0)+(1)\times(-1)=0
$$



**Window row 2.**



$$
R_2=(0)\times(2)+(1)\times(0)+(2)\times(-2)=-4
$$



**Window row 3.**



$$
R_3=(0)\times(1)+(2)\times(0)+(3)\times(-1)=-3
$$



**Add the rows to obtain one output cell.**



$$
Y_{2,2}=(0)+(-4)+(-3)=-7
$$



**Final output:**   $\begin{bmatrix}3 & 5\\3 & -7\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule.

## F25 Q7b

## Question

**Question (10 marks).** Explain hierarchical CNN feature extraction and the roles of convolution, pooling and fully connected layers. Why prefer CNNs over fully connected ANNs for images? Give medical-image and autonomous-vehicle applications and advantages.

## Answer — step by step

A convolutional neural network processes local spatial neighborhoods using learnable filters. A filter slides across the image; the same weights are reused at different locations. Each filter produces one output channel. Early features often respond to edges, subsequent layers combine local patterns into textures or parts, and deeper features support object-level classification.

![One filter is reused at successive spatial positions](https://deep-learning-study-room.vercel.app/assets/figures/convolution.svg)

Diagram: One filter is reused at successive spatial positions. Labels: Step 1: start (1,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 1×1 + 2×0 + 4×0 | + 5×(−1) = −4 | Step 2: start (1,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 2×1 + 3×0 + 5×0 | + 6×(−1) = −4 | Step 3: start (2,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 4×1 + 5×0 + 7×0 | + 8×(−1) = −4 | Step 4: start (2,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 5×1 + 6×0 + 8×0 | + 9×(−1) = −4 | Output: [−4, −4; −4, −4]. Kernel is not flipped.

One filter is reused at successive spatial positions Tap the diagram to enlarge.

Convolution extracts local patterns, a nonlinear activation lets the network combine them nonlinearly, and pooling reduces spatial resolution and sensitivity to small local shifts. Flattening or global pooling converts feature maps into a classification representation; a dense output layer produces class scores or probabilities.

Compared with a fully connected image network, CNNs exploit locality and weight sharing, often requiring far fewer parameters for early image processing. An RGB filter extends through all input channels. Pooling does not change the number of channels and normally has no trainable parameters. A CNN is not automatically invariant to every transformation.

For medical-image analysis, a CNN can classify a chest X-ray or identify suspicious regions from labeled scans, learning useful spatial patterns. For autonomous vehicles, CNN-based perception can detect road signs, pedestrians or lanes from camera images. Benefits include learned hierarchical visual features and reuse across image positions; both applications require held-out evaluation under relevant acquisition conditions.