# Drill 38 Residual addition

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s38

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s38/

## Question

**Question.** x = [2,−3,1], F(x) = [−1,1,0.5]. Calculate residual output before and after ReLU and compare the plain block.

## Answer — step by step

### Given

x = [2,−3,1]; F(x) = [−1,1,0.5].

### To find

Find x+F(x), apply ReLU, and compare with ReLU(F(x)).

### Formula — which one and why

**Residual addition.** Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested.

x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition.



$$
y=x+F(x)
$$



**Yaad rakhne ka cue:** Same position ke numbers add; shape same rehni chahiye.

**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.



$$
\operatorname{ReLU}(z)=\max(0,z)
$$





$$
\operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}
$$



**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

### Solution — step by step

### Meaning and formulas

The main branch produces F(x). The shortcut carries x. A residual join adds matching entries from these equally sized vectors. ReLU is a separate operation after that join.



$$
H(x)=x+F(x),\qquad a_j=\max(0,H_j)
$$



### Step 1: add each pair



$$
H_1=2+(-1)=1
$$





$$
H_2=-3+1=-2
$$





$$
H_3=1+0.5=1.5
$$



The residual output before ReLU is **[1, −2, 1.5]**.

### Step 2: apply ReLU



$$
[\max(0,1),\max(0,-2),\max(0,1.5)]=[1,0,1.5]
$$



### Step 3: compare with the plain branch

Without a shortcut, the raw output is just F(x) = [−1, 1, 0.5]. Its ReLU output is **[0, 1, 0.5]**.

Residual addition keeps three entries. Concatenation would join the vectors into six entries and is a different operation.