Deep Learning · TEE · formulas

Convolution value

One convolution output is a sum of matched input–kernel products. Move the window and repeat that exact rule for the next output. With several input channels, add contributions from every channel and add the filter bias once.

Y_{ij,o}=\sum_c\sum_u\sum_v K_{uvco}X_{iS+u-P,jS+v-P,c}+b_oLaTeX: Y_{ij,o}=\sum_c\sum_u\sum_v K_{uvco}X_{iS+u-P,jS+v-P,c}+b_o

One filter over multiple channels: all contributions go into one cell

Definition. Multi-channel convolution contracts the kernel's spatial and input-channel dimensions to produce one output value per filter at each spatial location. Different filters have independent learned kernels.

Explanation and working

1. Separate spatial size from depth

Imagine a 1 × 1 spatial location with three input channels [2, 5, 1]. A 1 × 1 filter still has one weight for each channel. Suppose its weights are [0.1, −0.2, 0.5] and its bias is 0.3.

Multiply each channel by its own weight, add the three contributions, then add the single bias. These are the numbers in the example. Tap the diagram to enlarge.
z=0.1(2)+(-0.2)(5)+0.5(1)+0.3LaTeX: z=0.1(2)+(-0.2)(5)+0.5(1)+0.3
z=0.2-1+0.5+0.3=0LaTeX: z=0.2-1+0.5+0.3=0

The three channels contributed to one output value. A single filter creates one output channel, even when its input has several channels.

2. A larger filter has one spatial grid per input channel

A 3 × 3 filter on a three-channel input has three 3 × 3 weight grids, totaling 27 weights. At one output location, calculate the nine products for channel 1, nine for channel 2 and nine for channel 3. Add all 27 products and then add the filter's one bias.

The channel-wise sums are intermediate contributions; do not treat them as three final output channels unless three distinct filters were specified.

3. What changes when there are several filters?

Each filter has its own full-depth weight set and bias. Four filters give four independent sums at each output location, so the result has four channels. The filters all receive the same input depth.

For three input channels and four 3 × 3 filters, parameters are (27 + 1) × 4 = 112. Moving those filters over more image locations reuses these same 112 parameters.

4. One-by-one does not mean one weight

A 1 × 1 convolution with 64 input channels and 16 filters has 64 weights and one bias per filter: (64 + 1) × 16 = 1,040 parameters. It mixes channel information at each spatial position. With 16 outputs it reduces depth from 64 to 16; height and width stay unchanged when stride is 1 and no padding is needed.

This is the channel-reduction idea used by Inception bottlenecks and some residual projections. Spatial kernel size, input depth and output-channel count must all be read separately.

Faculty example: a complete binary-image convolution

Question. Apply the shown 3 × 3 filter to the 5 × 5 image. Use stride 1, no padding and zero bias. Calculate every output value without flipping the filter. (Practice: 5 marks)

Given

Image X = \begin{bmatrix}1 & 1 & 1 & 0 & 0\\0 & 1 & 1 & 1 & 0\\0 & 0 & 1 & 1 & 1\\0 & 0 & 1 & 1 & 0\\0 & 1 & 1 & 0 & 0\end{bmatrix}LaTeX: \begin{bmatrix}1 & 1 & 1 & 0 & 0\\0 & 1 & 1 & 1 & 0\\0 & 0 & 1 & 1 & 1\\0 & 0 & 1 & 1 & 0\\0 & 1 & 1 & 0 & 0\end{bmatrix}.

Filter K = \begin{bmatrix}1 & 0 & 1\\0 & 1 & 0\\1 & 0 & 1\end{bmatrix}LaTeX: \begin{bmatrix}1 & 0 & 1\\0 & 1 & 0\\1 & 0 & 1\end{bmatrix}. These are the values in the Unit 4 lecture example.

To find

The output shape and all nine feature-map values.

Formula — which one and why

The output-size formula counts complete filter placements. At each placement, multiply matching image/filter cells and add the products. A filter entry of zero makes that particular product zero; it does not remove the image cell from the input.

N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1LaTeX: N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1

Here N is input side, K is kernel side, P is padding per side and S is stride. The symbol K in the shape formula is the size 3, whereas the matrix K above contains the filter's coefficients.

Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+bLaTeX: Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b

For this particular filter, only the four corners and the centre of each 3 × 3 window contribute. We can therefore write each sum using those five entries after explaining the four zero products. Below, their order is always top-left, top-right, centre, bottom-left, bottom-right within the current window.

Each output cell corresponds to a different 3 × 3 window. The filter coefficients are reused at all nine positions. Tap the diagram to enlarge.

Solution — step by step

Step 1: find the shape. (5 − 3)/1 + 1 = 3. The output is 3 × 3.

Step 2: top-left output, showing all nine products. 1×1 + 1×0 + 1×1 + 0×0 + 1×1 + 1×0 + 0×1 + 0×0 + 1×1 = 1+0+1+0+1+0+0+0+1 = 4.

Step 3: top-middle output. Move the window one column right. Corners and centre are 1,0,1,0,1. Sum = 1+0+1+0+1 = 3.

Step 4: top-right output. Move one more column right. Corners and centre are 1,0,1,1,1. Sum = 1+0+1+1+1 = 4.

Step 5: middle-left output. Return to column 1 and move down one row. Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = 2.

Step 6: middle-middle output. Corners and centre are 1,1,1,0,1. Sum = 1+1+1+0+1 = 4.

Step 7: middle-right output. Corners and centre are 1,0,1,1,0. Sum = 1+0+1+1+0 = 3.

Step 8: bottom-left output. Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = 2.

Step 9: bottom-middle output. Corners and centre are 0,1,1,1,0. Sum = 0+1+1+1+0 = 3.

Step 10: bottom-right output. Corners and centre are 1,1,1,1,0. Sum = 1+1+1+1+0 = 4.

Final answer: Y = \begin{bmatrix}4 & 3 & 4\\2 & 4 & 3\\2 & 3 & 4\end{bmatrix}LaTeX: \begin{bmatrix}4 & 3 & 4\\2 & 4 & 3\\2 & 3 & 4\end{bmatrix}. All nine outputs use the same nine filter weights. They do not introduce nine different filters.

Yaad rakho: Window badalta hai; filter ke numbers same rehte hain.

Faculty example: combine three RGB channel sums

Question. At one position, a convolutional filter receives the following red, green and blue 3 × 3 patches and matching kernel slices. Calculate one output value with bias 1. (Practice: 3 marks)

Given

Red patch = \begin{bmatrix}0 & 0 & 0\\0 & 156 & 155\\0 & 153 & 154\end{bmatrix}LaTeX: \begin{bmatrix}0 & 0 & 0\\0 & 156 & 155\\0 & 153 & 154\end{bmatrix}; red kernel = \begin{bmatrix}-1 & -1 & 1\\0 & 1 & -1\\0 & 1 & 1\end{bmatrix}LaTeX: \begin{bmatrix}-1 & -1 & 1\\0 & 1 & -1\\0 & 1 & 1\end{bmatrix}.

Green patch = \begin{bmatrix}0 & 0 & 0\\0 & 167 & 166\\0 & 164 & 165\end{bmatrix}LaTeX: \begin{bmatrix}0 & 0 & 0\\0 & 167 & 166\\0 & 164 & 165\end{bmatrix}; green kernel = \begin{bmatrix}1 & 0 & 0\\1 & -1 & -1\\1 & 0 & -1\end{bmatrix}LaTeX: \begin{bmatrix}1 & 0 & 0\\1 & -1 & -1\\1 & 0 & -1\end{bmatrix}.

Blue patch = \begin{bmatrix}0 & 0 & 0\\0 & 163 & 162\\0 & 160 & 161\end{bmatrix}LaTeX: \begin{bmatrix}0 & 0 & 0\\0 & 163 & 162\\0 & 160 & 161\end{bmatrix}; blue kernel = \begin{bmatrix}0 & 1 & 1\\0 & 1 & 0\\1 & -1 & 1\end{bmatrix}LaTeX: \begin{bmatrix}0 & 1 & 1\\0 & 1 & 0\\1 & -1 & 1\end{bmatrix}. Bias = 1.

These are three slices of one full-depth filter, not three independent output filters.

To find

The contribution of each input channel and the final scalar output at this position.

Formula — which one and why

Multiply corresponding entries within each channel. Add the three channel sums because this filter spans all input channels. Add its single bias once, after accumulating the channel contributions.

z=s_R+s_G+s_B+bLaTeX: z=s_R+s_G+s_B+b

The quantities s_R, s_G and s_B are intermediate weighted sums, not separate final output channels.

Solution — step by step

Step 1: red-channel products. The top row and first column contribute zero. The remaining terms are 156×1 + 155×(−1) + 153×1 + 154×1.

Step 2: red-channel sum. 156 − 155 + 153 + 154 = 308.

Step 3: green-channel products. The remaining patch terms are 167×(−1) + 166×(−1) + 164×0 + 165×(−1).

Step 4: green-channel sum. −167 −166 +0 −165 = −498.

Step 5: blue-channel products. The nonzero-patch terms are 163×1 + 162×0 + 160×(−1) + 161×1.

Step 6: blue-channel sum. 163 +0 −160 +161 = 164.

Step 7: add the channel contributions. 308 −498 +164 = −26.

Step 8: add one filter bias. −26 +1 = −25.

Final answer: The convolution output is −25. If a subsequent ReLU is explicitly requested, that activation produces max(0,−25) = 0. The convolution value and activated value are different stages.

Yaad rakho: Teen channels ka contribution jodo; ek filter ka bias ek baar jodo.

The convolution calculation

Definition. A convolutional layer calculates local weighted sums with a spatially shared kernel and optional bias. Standard deep-learning implementations ordinarily use cross-correlation, without flipping the kernel; examination calculations must follow the stated convention.

Explanation and working

In deep-learning software, the operation normally called convolution is cross-correlation: use the kernel as written, without flipping it. This is the convention in this course’s CNN numericals and the supplied latest answer key. Mathematical convolution would rotate the kernel; state that convention if a question explicitly requests it.

Question. Apply a 2 × 2 kernel Matrix 1 shown below to the 3 × 3 image Matrix 2 shown below, with stride 1, no padding and zero bias.

Given

Input X = \begin{bmatrix}1 & 2 & 3\\4 & 5 & 6\\7 & 8 & 9\end{bmatrix}LaTeX: \begin{bmatrix}1 & 2 & 3\\4 & 5 & 6\\7 & 8 & 9\end{bmatrix}.

Kernel K = \begin{bmatrix}1 & 0\\0 & -1\end{bmatrix}LaTeX: \begin{bmatrix}1 & 0\\0 & -1\end{bmatrix}.

Stride = 1; padding = 0; bias = 0; no kernel flip.

To find

Find the 2 × 2 output map.

Formula — which one and why

Convolution values. At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat.

X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention.

Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+bLaTeX: Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b

Yaad rakhne ka cue: Matching cells multiply; sab products add; bias sirf ek baar.

Solution — step by step

Given matrices for this question Tap the diagram to enlarge.
X=\begin{bmatrix}1&2&3\\4&5&6\\7&8&9\end{bmatrix},\quad K=\begin{bmatrix}1&0\\0&-1\end{bmatrix}LaTeX: X=\begin{bmatrix}1&2&3\\4&5&6\\7&8&9\end{bmatrix},\quad K=\begin{bmatrix}1&0\\0&-1\end{bmatrix}

Step 1. Top-left window. 1(1) + 2(0) + 4(0) + 5(−1) = 1 − 5 = −4.

Step 2. Move right by one. 2(1) + 3(0) + 5(0) + 6(−1) = 2 − 6 = −4.

Step 3. Move down and return left. 4(1) + 5(0) + 7(0) + 8(−1) = 4 − 8 = −4.

Step 4. Bottom-right window. 5(1) + 6(0) + 8(0) + 9(−1) = 5 − 9 = −4. Output is \begin{bmatrix}-4 & -4\\-4 & -4\end{bmatrix}LaTeX: \begin{bmatrix}-4 & -4\\-4 & -4\end{bmatrix}. If ReLU follows, that next layer changes all four entries to zero; do not silently apply it when it was not requested.

Four windows produce four output cells Tap the diagram to enlarge.

Drill 34 Convolution with a negative coefficient

Question

Question. Use the input A and kernel K shown below, stride 1, padding 0. Compute all outputs.

Given matrices
Input A
1201
0132
2101
1023
Kernel K
1−1
01

Stride 1; P = 0 (no padding).

Answer — step by step

Given

Input A and kernel K are printed with the question.

Stride S = 1; padding P = 0; bias = 0.

Use the kernel without flipping it.

To find

Calculate all nine output cells.

Formula — which one and why

Convolution values. At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat.

X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention.

Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+bLaTeX: Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b

Yaad rakhne ka cue: Matching cells multiply; sab products add; bias sirf ek baar.

Solution — step by step

Formulas and the operation we are performing

A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation.

Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+bLaTeX: Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+b

Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability.

N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1LaTeX: N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1

N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one.

Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs.

Step 1 shape: \left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1LaTeX: \left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1 = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel.

Output position (1, 1)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (1, 1) into one output cell

Input rows 1–2, columns 1–2. The window moves 1 cell per output step.

1. Select the input window
1201
0132
2101
1023

Only the outlined cells belong to this output.

2. Match the kernel
1−1
01

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell
1−2
01

Each cell is input × matching kernel coefficient.

4. Add the products row by row
Row 11 + (−2) = −1
Row 20 + 1 = 1

(−1) + 1 = 0

5. Place it in the output
051
0−22
132

Output row 1, column 1.

Window row 1.

R_1=(1)\times(1)+(2)\times(-1)=-1LaTeX: R_1=(1)\times(1)+(2)\times(-1)=-1

Window row 2.

R_2=(0)\times(0)+(1)\times(1)=1LaTeX: R_2=(0)\times(0)+(1)\times(1)=1

Add the rows to obtain one output cell.

Y_{1,1}=(-1)+(1)=0LaTeX: Y_{1,1}=(-1)+(1)=0

Output position (1, 2)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (1, 2) into one output cell

Input rows 1–2, columns 2–3. The window moves 1 cell per output step.

1. Select the input window
1201
0132
2101
1023

Only the outlined cells belong to this output.

2. Match the kernel
1−1
01

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell
20
03

Each cell is input × matching kernel coefficient.

4. Add the products row by row
Row 12 + 0 = 2
Row 20 + 3 = 3

2 + 3 = 5

5. Place it in the output
051
0−22
132

Output row 1, column 2.

Window row 1.

R_1=(2)\times(1)+(0)\times(-1)=2LaTeX: R_1=(2)\times(1)+(0)\times(-1)=2

Window row 2.

R_2=(1)\times(0)+(3)\times(1)=3LaTeX: R_2=(1)\times(0)+(3)\times(1)=3

Add the rows to obtain one output cell.

Y_{1,2}=(2)+(3)=5LaTeX: Y_{1,2}=(2)+(3)=5

Output position (1, 3)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (1, 3) into one output cell

Input rows 1–2, columns 3–4. The window moves 1 cell per output step.

1. Select the input window
1201
0132
2101
1023

Only the outlined cells belong to this output.

2. Match the kernel
1−1
01

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell
0−1
02

Each cell is input × matching kernel coefficient.

4. Add the products row by row
Row 10 + (−1) = −1
Row 20 + 2 = 2

(−1) + 2 = 1

5. Place it in the output
051
0−22
132

Output row 1, column 3.

Window row 1.

R_1=(0)\times(1)+(1)\times(-1)=-1LaTeX: R_1=(0)\times(1)+(1)\times(-1)=-1

Window row 2.

R_2=(3)\times(0)+(2)\times(1)=2LaTeX: R_2=(3)\times(0)+(2)\times(1)=2

Add the rows to obtain one output cell.

Y_{1,3}=(-1)+(2)=1LaTeX: Y_{1,3}=(-1)+(2)=1

Output position (2, 1)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (2, 1) into one output cell

Input rows 2–3, columns 1–2. The window moves 1 cell per output step.

1. Select the input window
1201
0132
2101
1023

Only the outlined cells belong to this output.

2. Match the kernel
1−1
01

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell
0−1
01

Each cell is input × matching kernel coefficient.

4. Add the products row by row
Row 10 + (−1) = −1
Row 20 + 1 = 1

(−1) + 1 = 0

5. Place it in the output
051
0−22
132

Output row 2, column 1.

Window row 1.

R_1=(0)\times(1)+(1)\times(-1)=-1LaTeX: R_1=(0)\times(1)+(1)\times(-1)=-1

Window row 2.

R_2=(2)\times(0)+(1)\times(1)=1LaTeX: R_2=(2)\times(0)+(1)\times(1)=1

Add the rows to obtain one output cell.

Y_{2,1}=(-1)+(1)=0LaTeX: Y_{2,1}=(-1)+(1)=0

Output position (2, 2)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (2, 2) into one output cell

Input rows 2–3, columns 2–3. The window moves 1 cell per output step.

1. Select the input window
1201
0132
2101
1023

Only the outlined cells belong to this output.

2. Match the kernel
1−1
01

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell
1−3
00

Each cell is input × matching kernel coefficient.

4. Add the products row by row
Row 11 + (−3) = −2
Row 20 + 0 = 0

(−2) + 0 = −2

5. Place it in the output
051
0−22
132

Output row 2, column 2.

Window row 1.

R_1=(1)\times(1)+(3)\times(-1)=-2LaTeX: R_1=(1)\times(1)+(3)\times(-1)=-2

Window row 2.

R_2=(1)\times(0)+(0)\times(1)=0LaTeX: R_2=(1)\times(0)+(0)\times(1)=0

Add the rows to obtain one output cell.

Y_{2,2}=(-2)+(0)=-2LaTeX: Y_{2,2}=(-2)+(0)=-2

Output position (2, 3)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (2, 3) into one output cell

Input rows 2–3, columns 3–4. The window moves 1 cell per output step.

1. Select the input window
1201
0132
2101
1023

Only the outlined cells belong to this output.

2. Match the kernel
1−1
01

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell
3−2
01

Each cell is input × matching kernel coefficient.

4. Add the products row by row
Row 13 + (−2) = 1
Row 20 + 1 = 1

1 + 1 = 2

5. Place it in the output
051
0−22
132

Output row 2, column 3.

Window row 1.

R_1=(3)\times(1)+(2)\times(-1)=1LaTeX: R_1=(3)\times(1)+(2)\times(-1)=1

Window row 2.

R_2=(0)\times(0)+(1)\times(1)=1LaTeX: R_2=(0)\times(0)+(1)\times(1)=1

Add the rows to obtain one output cell.

Y_{2,3}=(1)+(1)=2LaTeX: Y_{2,3}=(1)+(1)=2

Output position (3, 1)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (3, 1) into one output cell

Input rows 3–4, columns 1–2. The window moves 1 cell per output step.

1. Select the input window
1201
0132
2101
1023

Only the outlined cells belong to this output.

2. Match the kernel
1−1
01

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell
2−1
00

Each cell is input × matching kernel coefficient.

4. Add the products row by row
Row 12 + (−1) = 1
Row 20 + 0 = 0

1 + 0 = 1

5. Place it in the output
051
0−22
132

Output row 3, column 1.

Window row 1.

R_1=(2)\times(1)+(1)\times(-1)=1LaTeX: R_1=(2)\times(1)+(1)\times(-1)=1

Window row 2.

R_2=(1)\times(0)+(0)\times(1)=0LaTeX: R_2=(1)\times(0)+(0)\times(1)=0

Add the rows to obtain one output cell.

Y_{3,1}=(1)+(0)=1LaTeX: Y_{3,1}=(1)+(0)=1

Output position (3, 2)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (3, 2) into one output cell

Input rows 3–4, columns 2–3. The window moves 1 cell per output step.

1. Select the input window
1201
0132
2101
1023

Only the outlined cells belong to this output.

2. Match the kernel
1−1
01

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell
10
02

Each cell is input × matching kernel coefficient.

4. Add the products row by row
Row 11 + 0 = 1
Row 20 + 2 = 2

1 + 2 = 3

5. Place it in the output
051
0−22
132

Output row 3, column 2.

Window row 1.

R_1=(1)\times(1)+(0)\times(-1)=1LaTeX: R_1=(1)\times(1)+(0)\times(-1)=1

Window row 2.

R_2=(0)\times(0)+(2)\times(1)=2LaTeX: R_2=(0)\times(0)+(2)\times(1)=2

Add the rows to obtain one output cell.

Y_{3,2}=(1)+(2)=3LaTeX: Y_{3,2}=(1)+(2)=3

Output position (3, 3)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (3, 3) into one output cell

Input rows 3–4, columns 3–4. The window moves 1 cell per output step.

1. Select the input window
1201
0132
2101
1023

Only the outlined cells belong to this output.

2. Match the kernel
1−1
01

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell
0−1
03

Each cell is input × matching kernel coefficient.

4. Add the products row by row
Row 10 + (−1) = −1
Row 20 + 3 = 3

(−1) + 3 = 2

5. Place it in the output
051
0−22
132

Output row 3, column 3.

Window row 1.

R_1=(0)\times(1)+(1)\times(-1)=-1LaTeX: R_1=(0)\times(1)+(1)\times(-1)=-1

Window row 2.

R_2=(2)\times(0)+(3)\times(1)=3LaTeX: R_2=(2)\times(0)+(3)\times(1)=3

Add the rows to obtain one output cell.

Y_{3,3}=(-1)+(3)=2LaTeX: Y_{3,3}=(-1)+(3)=2

Final output: \begin{bmatrix}0 & 5 & 1\\0 & -2 & 2\\1 & 3 & 2\end{bmatrix}LaTeX: \begin{bmatrix}0 & 5 & 1\\0 & -2 & 2\\1 & 3 & 2\end{bmatrix}. Each output came from its own complete window; stride changes the next window position, not the multiplication rule.

Drill 46 Mixed convolution pool dense and loss

Question

Question. A is shown below, kernel Matrix 2 shown below, stride 1, valid. Apply ReLU, then 2 × 2 max pool stride 1. Flatten rowwise, use weight vector [0.1,0.2,0.3,0.4], bias 0, sigmoid output, target 1. Find all stages and BCE.

Given matrices
Input A
1201
0132
2101
1023
Kernel K · Matrix 2
10
0−1

Stride 1; P = 0 (no padding).

Answer — step by step

Given

Input A and kernel Matrix 2 are printed with the question.

Conv: stride 1, valid padding; follow with ReLU.

Max pool: 2 × 2, stride 1; flatten row by row.

Dense weights = [0.1,0.2,0.3,0.4]; bias = 0; sigmoid output.

Binary target = 1.

To find

Find convolution, activated map, pooled map, flattened values, prediction and BCE.

Formula — which one and why

Convolution values. At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat.

X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention.

Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+bLaTeX: Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b

Yaad rakhne ka cue: Matching cells multiply; sab products add; bias sirf ek baar.

Pooling. Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately.

V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases.

y_{\max}=\max VLaTeX: y_{\max}=\max V
y_{\mathrm{avg}}=\frac1n\sum_{v\in V}vLaTeX: y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v

Yaad rakhne ka cue: Max mein biggest; average mein total ÷ window ka count.

Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.

xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.

z=\sum_iw_ix_i+bLaTeX: z=\sum_iw_ix_i+b
a=f(z)LaTeX: a=f(z)

Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.

Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.

p=\frac{1}{1+e^{-z}}LaTeX: p=\frac{1}{1+e^{-z}}
\frac{dp}{dz}=p(1-p)LaTeX: \frac{dp}{dz}=p(1-p)

Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).

Binary cross-entropy. For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y.

y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative.

L=-[y\ln p+(1-y)\ln(1-p)]LaTeX: L=-[y\ln p+(1-y)\ln(1-p)]
\frac{\partial L}{\partial z}=p-yLaTeX: \frac{\partial L}{\partial z}=p-y

Yaad rakhne ka cue: Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p).

Solution — step by step

Formulas and the operation we are performing

A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation.

Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+bLaTeX: Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+b

Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability.

N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1LaTeX: N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1

N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one.

Formulas and meaning

Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer.

p_{\mathrm{max}}=\max(a_1,a_2,a_3,a_4)LaTeX: p_{\mathrm{max}}=\max(a_1,a_2,a_3,a_4)
p_{\mathrm{avg}}=\frac{a_1+a_2+a_3+a_4}{4}LaTeX: p_{\mathrm{avg}}=\frac{a_1+a_2+a_3+a_4}{4}
N_{\mathrm{pool}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1LaTeX: N_{\mathrm{pool}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1

Use the input size at the pooling layer. The output keeps the same channel count.

Completing the network after pooling

Flatten row by row. Multiply the resulting entries by the dense weights and add bias. Apply sigmoid to produce p. For target 1, binary cross-entropy is minus ln(p).

z=\sum_iw_ix_i+b,\qquad p=\frac1{1+e^{-z}},\qquad L=-\ln pLaTeX: z=\sum_iw_ix_i+b,\qquad p=\frac1{1+e^{-z}},\qquad L=-\ln p

Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs.

Step 1 shape: \left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1LaTeX: \left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1 = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel.

Output position (1, 1)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (1, 1) into one output cell

Input rows 1–2, columns 1–2. The window moves 1 cell per output step.

1. Select the input window
1201
0132
2101
1023

Only the outlined cells belong to this output.

2. Match the kernel
10
0−1

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell
10
0−1

Each cell is input × matching kernel coefficient.

4. Add the products row by row
Row 11 + 0 = 1
Row 20 + (−1) = −1

1 + (−1) = 0

5. Place it in the output
0−1−2
−112
2−1−3

Output row 1, column 1.

Window row 1.

R_1=(1)\times(1)+(2)\times(0)=1LaTeX: R_1=(1)\times(1)+(2)\times(0)=1

Window row 2.

R_2=(0)\times(0)+(1)\times(-1)=-1LaTeX: R_2=(0)\times(0)+(1)\times(-1)=-1

Add the rows to obtain one output cell.

Y_{1,1}=(1)+(-1)=0LaTeX: Y_{1,1}=(1)+(-1)=0

Output position (1, 2)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (1, 2) into one output cell

Input rows 1–2, columns 2–3. The window moves 1 cell per output step.

1. Select the input window
1201
0132
2101
1023

Only the outlined cells belong to this output.

2. Match the kernel
10
0−1

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell
20
0−3

Each cell is input × matching kernel coefficient.

4. Add the products row by row
Row 12 + 0 = 2
Row 20 + (−3) = −3

2 + (−3) = −1

5. Place it in the output
0−1−2
−112
2−1−3

Output row 1, column 2.

Window row 1.

R_1=(2)\times(1)+(0)\times(0)=2LaTeX: R_1=(2)\times(1)+(0)\times(0)=2

Window row 2.

R_2=(1)\times(0)+(3)\times(-1)=-3LaTeX: R_2=(1)\times(0)+(3)\times(-1)=-3

Add the rows to obtain one output cell.

Y_{1,2}=(2)+(-3)=-1LaTeX: Y_{1,2}=(2)+(-3)=-1

Output position (1, 3)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (1, 3) into one output cell

Input rows 1–2, columns 3–4. The window moves 1 cell per output step.

1. Select the input window
1201
0132
2101
1023

Only the outlined cells belong to this output.

2. Match the kernel
10
0−1

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell
00
0−2

Each cell is input × matching kernel coefficient.

4. Add the products row by row
Row 10 + 0 = 0
Row 20 + (−2) = −2

0 + (−2) = −2

5. Place it in the output
0−1−2
−112
2−1−3

Output row 1, column 3.

Window row 1.

R_1=(0)\times(1)+(1)\times(0)=0LaTeX: R_1=(0)\times(1)+(1)\times(0)=0

Window row 2.

R_2=(3)\times(0)+(2)\times(-1)=-2LaTeX: R_2=(3)\times(0)+(2)\times(-1)=-2

Add the rows to obtain one output cell.

Y_{1,3}=(0)+(-2)=-2LaTeX: Y_{1,3}=(0)+(-2)=-2

Output position (2, 1)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (2, 1) into one output cell

Input rows 2–3, columns 1–2. The window moves 1 cell per output step.

1. Select the input window
1201
0132
2101
1023

Only the outlined cells belong to this output.

2. Match the kernel
10
0−1

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell
00
0−1

Each cell is input × matching kernel coefficient.

4. Add the products row by row
Row 10 + 0 = 0
Row 20 + (−1) = −1

0 + (−1) = −1

5. Place it in the output
0−1−2
−112
2−1−3

Output row 2, column 1.

Window row 1.

R_1=(0)\times(1)+(1)\times(0)=0LaTeX: R_1=(0)\times(1)+(1)\times(0)=0

Window row 2.

R_2=(2)\times(0)+(1)\times(-1)=-1LaTeX: R_2=(2)\times(0)+(1)\times(-1)=-1

Add the rows to obtain one output cell.

Y_{2,1}=(0)+(-1)=-1LaTeX: Y_{2,1}=(0)+(-1)=-1

Output position (2, 2)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (2, 2) into one output cell

Input rows 2–3, columns 2–3. The window moves 1 cell per output step.

1. Select the input window
1201
0132
2101
1023

Only the outlined cells belong to this output.

2. Match the kernel
10
0−1

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell
10
00

Each cell is input × matching kernel coefficient.

4. Add the products row by row
Row 11 + 0 = 1
Row 20 + 0 = 0

1 + 0 = 1

5. Place it in the output
0−1−2
−112
2−1−3

Output row 2, column 2.

Window row 1.

R_1=(1)\times(1)+(3)\times(0)=1LaTeX: R_1=(1)\times(1)+(3)\times(0)=1

Window row 2.

R_2=(1)\times(0)+(0)\times(-1)=0LaTeX: R_2=(1)\times(0)+(0)\times(-1)=0

Add the rows to obtain one output cell.

Y_{2,2}=(1)+(0)=1LaTeX: Y_{2,2}=(1)+(0)=1

Output position (2, 3)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (2, 3) into one output cell

Input rows 2–3, columns 3–4. The window moves 1 cell per output step.

1. Select the input window
1201
0132
2101
1023

Only the outlined cells belong to this output.

2. Match the kernel
10
0−1

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell
30
0−1

Each cell is input × matching kernel coefficient.

4. Add the products row by row
Row 13 + 0 = 3
Row 20 + (−1) = −1

3 + (−1) = 2

5. Place it in the output
0−1−2
−112
2−1−3

Output row 2, column 3.

Window row 1.

R_1=(3)\times(1)+(2)\times(0)=3LaTeX: R_1=(3)\times(1)+(2)\times(0)=3

Window row 2.

R_2=(0)\times(0)+(1)\times(-1)=-1LaTeX: R_2=(0)\times(0)+(1)\times(-1)=-1

Add the rows to obtain one output cell.

Y_{2,3}=(3)+(-1)=2LaTeX: Y_{2,3}=(3)+(-1)=2

Output position (3, 1)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (3, 1) into one output cell

Input rows 3–4, columns 1–2. The window moves 1 cell per output step.

1. Select the input window
1201
0132
2101
1023

Only the outlined cells belong to this output.

2. Match the kernel
10
0−1

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell
20
00

Each cell is input × matching kernel coefficient.

4. Add the products row by row
Row 12 + 0 = 2
Row 20 + 0 = 0

2 + 0 = 2

5. Place it in the output
0−1−2
−112
2−1−3

Output row 3, column 1.

Window row 1.

R_1=(2)\times(1)+(1)\times(0)=2LaTeX: R_1=(2)\times(1)+(1)\times(0)=2

Window row 2.

R_2=(1)\times(0)+(0)\times(-1)=0LaTeX: R_2=(1)\times(0)+(0)\times(-1)=0

Add the rows to obtain one output cell.

Y_{3,1}=(2)+(0)=2LaTeX: Y_{3,1}=(2)+(0)=2

Output position (3, 2)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (3, 2) into one output cell

Input rows 3–4, columns 2–3. The window moves 1 cell per output step.

1. Select the input window
1201
0132
2101
1023

Only the outlined cells belong to this output.

2. Match the kernel
10
0−1

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell
10
0−2

Each cell is input × matching kernel coefficient.

4. Add the products row by row
Row 11 + 0 = 1
Row 20 + (−2) = −2

1 + (−2) = −1

5. Place it in the output
0−1−2
−112
2−1−3

Output row 3, column 2.

Window row 1.

R_1=(1)\times(1)+(0)\times(0)=1LaTeX: R_1=(1)\times(1)+(0)\times(0)=1

Window row 2.

R_2=(0)\times(0)+(2)\times(-1)=-2LaTeX: R_2=(0)\times(0)+(2)\times(-1)=-2

Add the rows to obtain one output cell.

Y_{3,2}=(1)+(-2)=-1LaTeX: Y_{3,2}=(1)+(-2)=-1

Output position (3, 3)

Use this position's input window. Multiply corresponding entries row by row; then add all row totals.

Follow window (3, 3) into one output cell

Input rows 3–4, columns 3–4. The window moves 1 cell per output step.

1. Select the input window
1201
0132
2101
1023

Only the outlined cells belong to this output.

2. Match the kernel
10
0−1

Same row and column positions; no matrix multiplication.

3. Multiply cell by cell
00
0−3

Each cell is input × matching kernel coefficient.

4. Add the products row by row
Row 10 + 0 = 0
Row 20 + (−3) = −3

0 + (−3) = −3

5. Place it in the output
0−1−2
−112
2−1−3

Output row 3, column 3.

Window row 1.

R_1=(0)\times(1)+(1)\times(0)=0LaTeX: R_1=(0)\times(1)+(1)\times(0)=0

Window row 2.

R_2=(2)\times(0)+(3)\times(-1)=-3LaTeX: R_2=(2)\times(0)+(3)\times(-1)=-3

Add the rows to obtain one output cell.

Y_{3,3}=(0)+(-3)=-3LaTeX: Y_{3,3}=(0)+(-3)=-3

Final output: \begin{bmatrix}0 & -1 & -2\\-1 & 1 & 2\\2 & -1 & -3\end{bmatrix}LaTeX: \begin{bmatrix}0 & -1 & -2\\-1 & 1 & 2\\2 & -1 & -3\end{bmatrix}. Each output came from its own complete window; stride changes the next window position, not the multiplication rule.

ReLU turns negative entries into 0: \begin{bmatrix}0 & 0 & 0\\0 & 1 & 2\\2 & 0 & 0\end{bmatrix}LaTeX: \begin{bmatrix}0 & 0 & 0\\0 & 1 & 2\\2 & 0 & 0\end{bmatrix}.

Shape: \left\lfloor\frac{3-2}{1}\right\rfloor+1LaTeX: \left\lfloor\frac{3-2}{1}\right\rfloor+1 = 2 on each axis; channels stay unchanged.

Pooling window (1,1)

Step 1: select the entries.

[0, 0, 0, 1]

Step 2: add all selected entries.

0 + 0 + 0 + 1 = 1

Step 3: divide by the 4 entries in this window.

1/4 = 0.25. Maximum = 1

Pooling window (1,2)

Step 1: select the entries.

[0, 0, 1, 2]

Step 2: add all selected entries.

0 + 0 + 1 + 2 = 3

Step 3: divide by the 4 entries in this window.

3/4 = 0.75. Maximum = 2

Pooling window (2,1)

Step 1: select the entries.

[0, 1, 2, 0]

Step 2: add all selected entries.

0 + 1 + 2 + 0 = 3

Step 3: divide by the 4 entries in this window.

3/4 = 0.75. Maximum = 2

Pooling window (2,2)

Step 1: select the entries.

[1, 2, 0, 0]

Step 2: add all selected entries.

1 + 2 + 0 + 0 = 3

Step 3: divide by the 4 entries in this window.

3/4 = 0.75. Maximum = 2

Average output: \begin{bmatrix}0.25 & 0.75\\0.75 & 0.75\end{bmatrix}LaTeX: \begin{bmatrix}0.25 & 0.75\\0.75 & 0.75\end{bmatrix}. Max output: \begin{bmatrix}1 & 2\\2 & 2\end{bmatrix}LaTeX: \begin{bmatrix}1 & 2\\2 & 2\end{bmatrix}. Pooling normally has zero trainable parameters and reduces spatial dimensions.

Use MAX output and flatten rowwise: [1, 2, 2, 2]. Dense z = 0.1 × 1 + 0.2 × 2 + 0.3 × 2 + 0.4 × 2 = 1.9. p = σ(1.9) = 0.869891526. BCE = −ln(p) = 0.139386758. Average pool values were computed for contrast; the question requires max pool.