Deep Learning · TEE · formulas

Concatenation and residual addition

Concatenation and addition combine branches differently. Concatenation places channel groups next to each other and increases the channel count. Residual addition adds matching entries, so the two full shapes must be compatible.

C_{concat}=\sum_b C_bLaTeX: C_{concat}=\sum_b C_b
y_{res}=F(x)+xLaTeX: y_{res}=F(x)+x

Inception concatenation and residual addition

Definition. Channel concatenation joins feature tensors while preserving their separate channels. Residual addition sums corresponding entries. These operations impose different shape constraints and produce different output depths.

Explanation and working

Concatenation joins feature channels; elementwise addition sums matching entries. They are different operations.

Concatenation stacks channels. Residual addition combines matching values and preserves the matched shape. Tap the diagram to enlarge.

Four Inception branches of spatial size 14 × 14 have 16, 32, 8 and 8 channels. First add the channel counts: 16 + 32 = 48; 48 + 8 = 56; 56 + 8 = 64. Height and width stay 14. The concatenated output is 14 × 14 × 64.

A residual addition requires matching shapes. A 14 × 14 × 64 tensor cannot be directly added to a 14 × 14 × 32 tensor. A projection, often a 1 × 1 convolution, can transform one path to the matching shape before addition.

ResNet adds a shortcut

Definition. A residual network uses shortcut connections to learn a residual mapping relative to a block's input. A basic residual block adds the block transformation F(x) to a shortcut representation of x.

y=F(x)+xLaTeX: y=F(x)+x

If the desired mapping is H(x), the residual branch represents F(x) = H(x) − x. When the useful mapping is close to identity, a small residual can be easier to optimize than reconstructing identity through the entire branch.

The two paths must have equal output shape before elementwise addition. Tap the diagram to enlarge.

Effect on gradient propagation

For the identity-shortcut formulation, the derivative includes a direct identity contribution.

\frac{\partial y}{\partial x}=\frac{\partial F}{\partial x}+ILaTeX: \frac{\partial y}{\partial x}=\frac{\partial F}{\partial x}+I

This provides a path for information and derivatives that does not require passing through every weight layer in the residual branch. It helps optimize deep networks and addresses the degradation problem. It does not imply that every possible ResNet is immune to vanishing gradients.

Projection when dimensions differ

If channels or spatial dimensions change, use a projection shortcut, often a 1 × 1 convolution with suitable stride.

y=F(x)+W_sxLaTeX: y=F(x)+W_sx

An identity shortcut has no learned parameters. A projection does. For Cᵢ input channels and Cₒ output channels, a biased 1 × 1 projection has CᵢCₒ + Cₒ parameters. The residual branch's output must match the projection output for addition.

Question. Let x = [1, 2, 3] and F(x) = [0.1, −0.2, 0.3]. Find the residual sum.

Given

x = [1,2,3]; residual branch F(x) = [0.1,−0.2,0.3].

To find

Find the three components of x+F(x).

Formula — which one and why

Residual addition. Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested.

x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition.

y=x+F(x)LaTeX: y=x+F(x)

Yaad rakhne ka cue: Same position ke numbers add; shape same rehni chahiye.

Solution — step by step

Step 1: add the first matching pair.

1+0.1=1.1LaTeX: 1+0.1=1.1

Step 2: add the second matching pair.

2+(-0.2)=1.8LaTeX: 2+(-0.2)=1.8

Step 3: add the third matching pair.

3+0.3=3.3LaTeX: 3+0.3=3.3

Step 4: assemble in the original order.

y=[1.1,1.8,3.3]LaTeX: y=[1.1,1.8,3.3]

The vector length remains 3. Concatenating these vectors would give length 6 and would implement a different operation. Any post-addition activation specified in the architecture is applied after this sum.

Drill 51 Inception concatenation and parameter count

Question

Question. Input 10 × 10 × 8. Parallel branches: A uses 1 × 1 conv 4 filters ; B uses 3 × 3 same Conv 6 filters; C max pool 3 same, stride 1 then 1 × 1 conv 2 filters. Find combined shape and trainable parameters.

Answer — step by step

Given

Input = 10 × 10 × 8.

Branch A: 1 × 1 convolution, 4 filters.

Branch B: 3 × 3 same convolution, 6 filters.

Branch C: 3 × 3 same max pool with stride 1, then 1 × 1 convolution, 2 filters.

To find

Find each branch shape and parameters, then the concatenated shape and total parameters.

Formula — which one and why

Inception channel concatenation. Parallel branches retain their separate features. Add their channel counts while keeping the common height and width. Add trainable parameters across branches.

C₁,C₂,… = branch output channels. Concatenation requires matching spatial dimensions.

C_{\mathrm{out}}=\sum_k C_kLaTeX: C_{\mathrm{out}}=\sum_k C_k
P_{\mathrm{total}}=\sum_k P_kLaTeX: P_{\mathrm{total}}=\sum_k P_k

Yaad rakhne ka cue: Channels jodo; height-width same rakho.

Convolution parameters. Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.

K_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.

N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}}LaTeX: N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}}
N_b=C_{\mathrm{out}}LaTeX: N_b=C_{\mathrm{out}}
P=N_w+N_bLaTeX: P=N_w+N_b

Yaad rakhne ka cue: Filter area × input channels × filters; phir ek bias per filter.

Solution — step by step

Meaning and formulas

Each parallel branch receives the same original 10 × 10 × 8 input. “Same” with stride 1 preserves height and width. Concatenation joins the channel lists without adding the feature values.

C_{\mathrm{combined}}=C_A+C_B+C_CLaTeX: C_{\mathrm{combined}}=C_A+C_B+C_C
P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}}LaTeX: P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}}

Assume one bias per filter. Pooling has no learned parameters.

Step 1: Branch A.

Its 4 filters produce shape 10 × 10 × 4.

P_A=(1\times1\times8+1)\times4=9\times4=36LaTeX: P_A=(1\times1\times8+1)\times4=9\times4=36

Step 2: Branch B.

Its 6 filters produce shape 10 × 10 × 6.

P_B=(3\times3\times8+1)\times6=73\times6=438LaTeX: P_B=(3\times3\times8+1)\times6=73\times6=438

Step 3: Branch C.

Pooling preserves 10 × 10 × 8 here because stride is 1 and padding is same. Its parameter count is zero. The following 2-filter projection creates 10 × 10 × 2.

P_C=0+(1\times1\times8+1)\times2=18LaTeX: P_C=0+(1\times1\times8+1)\times2=18

Step 4: Join the shapes and add the parameters.

C_{\mathrm{combined}}=4+6+2=12LaTeX: C_{\mathrm{combined}}=4+6+2=12
P_{\mathrm{total}}=36+438+18=492LaTeX: P_{\mathrm{total}}=36+438+18=492

Final answer: combined shape 10 × 10 × 12; 492 parameters. Branch B does not take branch A's four channels as input: they are parallel branches.

Drill 52 Projection shortcut

Question

Question. A residual main path outputs 8 × 8 × 16 but input is 16 × 16 × 8. Design a 1 × 1 projection shortcut with stride 2, zero padding. Find its output and parameters.

Answer — step by step

Given

Residual input = 16 × 16 × 8; main-path output = 8 × 8 × 16.

Shortcut kernel = 1 × 1; stride 2; padding 0.

Use one bias per projection filter.

To find

Choose the number of filters and find projection output shape and parameters.

Formula — which one and why

Convolution/pooling spatial size. Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1.

N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here.

N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1LaTeX: N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1

Yaad rakhne ka cue: Input + dono sides ki padding − kernel; stride se divide; floor + 1.

Convolution parameters. Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.

K_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.

N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}}LaTeX: N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}}
N_b=C_{\mathrm{out}}LaTeX: N_b=C_{\mathrm{out}}
P=N_w+N_bLaTeX: P=N_w+N_b

Yaad rakhne ka cue: Filter area × input channels × filters; phir ek bias per filter.

Solution — step by step

Meaning and formulas

Residual addition needs identical shapes on both paths. The projection must reduce height and width from 16 to 8 and increase channels from 8 to 16.

N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1LaTeX: N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1
P_{\mathrm{projection}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}}LaTeX: P_{\mathrm{projection}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}}

Step 1: choose 16 filters

One filter creates one channel, so 16 filters give the required 16 output channels.

Step 2: use the specified 1 × 1 kernel and stride 2

N_{\mathrm{out}}=\left\lfloor\frac{16+0-1}{2}\right\rfloor+1LaTeX: N_{\mathrm{out}}=\left\lfloor\frac{16+0-1}{2}\right\rfloor+1
N_{\mathrm{out}}=\lfloor7.5\rfloor+1=7+1=8LaTeX: N_{\mathrm{out}}=\lfloor7.5\rfloor+1=7+1=8

The floor operation drops the fractional part; it does not round 7.5 to 8 before adding 1.

Step 3: count weights and biases

\text{weights}=1\times1\times8\times16=128LaTeX: \text{weights}=1\times1\times8\times16=128
\text{biases}=16,\qquad \text{total}=128+16=144LaTeX: \text{biases}=16,\qquad \text{total}=128+16=144

Final answer: projection output 8 × 8 × 16 and 144 parameters with biases. The projected shortcut now matches the main path, so elementwise addition is valid.

Drill 40 Residual addition

Question

Question. x = [2,−3,1], F(x) = [−1,1,0.5]. Calculate residual output before and after ReLU and compare the plain block.

Answer — step by step

Given

x = [2,−3,1]; F(x) = [−1,1,0.5].

To find

Find x+F(x), apply ReLU, and compare with ReLU(F(x)).

Formula — which one and why

Residual addition. Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested.

x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition.

y=x+F(x)LaTeX: y=x+F(x)

Yaad rakhne ka cue: Same position ke numbers add; shape same rehni chahiye.

ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.

\operatorname{ReLU}(z)=\max(0,z)LaTeX: \operatorname{ReLU}(z)=\max(0,z)
\operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}

Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

Solution — step by step

Meaning and formulas

The main branch produces F(x). The shortcut carries x. A residual join adds matching entries from these equally sized vectors. ReLU is a separate operation after that join.

H(x)=x+F(x),\qquad a_j=\max(0,H_j)LaTeX: H(x)=x+F(x),\qquad a_j=\max(0,H_j)

Step 1: add each pair

H_1=2+(-1)=1LaTeX: H_1=2+(-1)=1
H_2=-3+1=-2LaTeX: H_2=-3+1=-2
H_3=1+0.5=1.5LaTeX: H_3=1+0.5=1.5

The residual output before ReLU is [1, −2, 1.5].

Step 2: apply ReLU

[\max(0,1),\max(0,-2),\max(0,1.5)]=[1,0,1.5]LaTeX: [\max(0,1),\max(0,-2),\max(0,1.5)]=[1,0,1.5]

Step 3: compare with the plain branch

Without a shortcut, the raw output is just F(x) = [−1, 1, 0.5]. Its ReLU output is [0, 1, 0.5].

Residual addition keeps three entries. Concatenation would join the vectors into six entries and is a different operation.