Drill 38 Residual addition

Answer — step by step

Given

x = [2,−3,1]; F(x) = [−1,1,0.5].

To find

Find x+F(x), apply ReLU, and compare with ReLU(F(x)).

Formula — which one and why

Residual addition. Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested.

x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition.

LaTeX: y=x+F(x)

Yaad rakhne ka cue: Same position ke numbers add; shape same rehni chahiye.

ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.

LaTeX: \operatorname{ReLU}(z)=\max(0,z)

LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}

Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

Solution — step by step

Meaning and formulas

The main branch produces F(x). The shortcut carries x. A residual join adds matching entries from these equally sized vectors. ReLU is a separate operation after that join.

LaTeX: H(x)=x+F(x),\qquad a_j=\max(0,H_j)

Step 1: add each pair

LaTeX: H_1=2+(-1)=1

LaTeX: H_2=-3+1=-2

LaTeX: H_3=1+0.5=1.5

The residual output before ReLU is [1, −2, 1.5].

Step 2: apply ReLU

LaTeX: [\max(0,1),\max(0,-2),\max(0,1.5)]=[1,0,1.5]

Step 3: compare with the plain branch

Without a shortcut, the raw output is just F(x) = [−1, 1, 0.5]. Its ReLU output is [0, 1, 0.5].

Residual addition keeps three entries. Concatenation would join the vectors into six entries and is a different operation.