Deep Learning · TEE · section

Drill 15 A negative ReLU blocks this path

Numerical practice from direct to mixed

Question

Question. x = 2, w1 = −0.4, b1 = 0.1, w2 = 0.5, b2 = 0.2; hidden ReLU, linear output, y = 1, half squared loss. Find dw 1 and dw 2.

Answer — step by step

Given

x = 2; w₁ = −0.4; b₁ = 0.1; w₂ = 0.5; b₂ = 0.2.

Hidden activation: ReLU; output activation: linear.

Target y = 1; loss: half squared error.

To find

Find the gradients of w₁ and w₂ and explain any zero gradients.

Formula — which one and why

Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.

x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.

$z_h=W_1x+b_1$ LaTeX: z_h=W_1x+b_1

$h=f_1(z_h)$ LaTeX: h=f_1(z_h)

$z_o=W_2h+b_2$ LaTeX: z_o=W_2h+b_2

$o=f_2(z_o)$ LaTeX: o=f_2(z_o)

Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.

Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.

oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.

$L=\frac12\sum_j(o_j-y_j)^2$ LaTeX: L=\frac12\sum_j(o_j-y_j)^2

$\frac{\partial L}{\partial o_j}=o_j-y_j$ LaTeX: \frac{\partial L}{\partial o_j}=o_j-y_j

Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.

Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.

δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.

$\delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})$ LaTeX: \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})

$\delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}$ LaTeX: \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}

$\frac{\partial L}{\partial W_{ji}}=\delta_ja_i$ LaTeX: \frac{\partial L}{\partial W_{ji}}=\delta_ja_i

$\frac{\partial L}{\partial b_j}=\delta_j$ LaTeX: \frac{\partial L}{\partial b_j}=\delta_j

$W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}$ LaTeX: W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}

Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.

Solution — step by step

Formulas used

The hidden neuron uses ReLU, but the output is linear. “Linear output” means its weighted sum is already its output.

Step 1: hidden weighted sum and activation

$h=\max(0,-0.7)=0$ LaTeX: h=\max(0,-0.7)=0

Because the raw hidden score is negative, ReLU's derivative at this point is 0.

Step 2: output prediction and output delta

$o=0.5\times0+0.2=0.2$ LaTeX: o=0.5\times0+0.2=0.2

$\delta_o=0.2-1=-0.8$ LaTeX: \delta_o=0.2-1=-0.8

Step 3: output weight gradient

$\frac{\partial L}{\partial w_2}=(-0.8)\times0=0$ LaTeX: \frac{\partial L}{\partial w_2}=(-0.8)\times0=0

This is zero because the input to that connection, h, is zero.

Step 4: hidden delta and input weight gradient

$\delta_h=(-0.8)\times0.5\times0=0$ LaTeX: \delta_h=(-0.8)\times0.5\times0=0

$\frac{\partial L}{\partial w_1}=0\times2=0$ LaTeX: \frac{\partial L}{\partial w_1}=0\times2=0

Final answer: both requested weight gradients are zero. The output bias gradient equals −0.8, so a zero weight gradient does not mean every parameter's gradient is zero.