Deep Learning · TEE · formulas

Hidden delta, weight gradient and bias gradient

A hidden neuron can influence several outputs. Add all those backward contributions before multiplying by its own activation derivative. For each connection, use the receiving neuron’s delta and the sending neuron’s activation. Keep the old weights unchanged until all gradients have been found.

$\delta_j^h=f\prime(z_j^h)\sum_k w_{kj}^{out}\delta_k^{out}$ LaTeX: \delta_j^h=f\prime(z_j^h)\sum_k w_{kj}^{out}\delta_k^{out}

$\frac{\partial L}{\partial w_{ji}}=\delta_j a_i$ LaTeX: \frac{\partial L}{\partial w_{ji}}=\delta_j a_i

$\frac{\partial L}{\partial b_j}=\delta_j$ LaTeX: \frac{\partial L}{\partial b_j}=\delta_j

More outputs and vectorized backpropagation

Definition. Vectorized backpropagation expresses the chain rule using matrix products and elementwise derivatives. When a hidden activation affects several outputs, its derivative contains the sum of the contributions from every downstream path.

Why the hidden derivative is a sum

If hidden activation hⱼ connects to output preactivations zₒ₁ and zₒ₂, changing hⱼ changes both outputs. The multivariable chain rule gives:

$\frac{\partial L}{\partial h_j}=W_{2,1j}\delta_{o_1}+W_{2,2j}\delta_{o_2}$ LaTeX: \frac{\partial L}{\partial h_j}=W_{2,1j}\delta_{o_1}+W_{2,2j}\delta_{o_2}

For any number of connected outputs, sum their contributions and then differentiate the hidden activation:

\delta_{h_j}=\left(\sum_k W_{2,kj}\delta_{o_k}\right)f'(z_{h_j})

Use the connecting weights from the original forward pass. Adding only the contribution of the first output would omit a dependency of the loss.

The outgoing paths both contribute to the hidden derivative. Add the weighted output deltas, then apply the hidden activation derivative. Tap the diagram to enlarge.

Matrix equations for one training sample

Use column vectors throughout. Let W₁ map d inputs to m hidden units and W₂ map those units to k outputs. The following output-delta formula assumes half sum of squared errors and an elementwise output activation f₂:

$L=\frac12\sum_{j=1}^{k}(\hat y_j-y_j)^2$ LaTeX: L=\frac12\sum_{j=1}^{k}(\hat y_j-y_j)^2

$\delta_2=(\hat y-y)\odot f_2'(z_2)$ LaTeX: \delta_2=(\hat y-y)\odot f_2'(z_2)

$\delta_1=(W_2^T\delta_2)\odot f_1'(z_1)$ LaTeX: \delta_1=(W_2^T\delta_2)\odot f_1'(z_1)

Here ⊙ means elementwise multiplication. The product δ₂a₁ᵀ is an outer product: every receiving delta is multiplied by every sending activation.

Quantity	Shape	Reason
W₁ and its gradient	m × d	m receiving hidden neurons, d inputs
W₂ and its gradient	k × m	k receiving output neurons, m hidden activations
δ₂	k × 1	One derivative per output preactivation
W₂ᵀδ₂	m × 1	The summed downstream derivative for each hidden unit
δ₂a₁ᵀ	(k × 1)(1 × m) = k × m	The output-weight gradient has the same shape as W₂

How the loss changes the output delta

For MSE averaged over k output components, the squared-error derivative has the factor 2/k. For sigmoid with binary cross-entropy, or softmax with categorical cross-entropy and a normalized target, the paired derivative simplifies to prediction minus target:

$\delta_{\mathrm{output}}=\hat y-y$ LaTeX: \delta_{\mathrm{output}}=\hat y-y

That simplification already includes differentiation of the paired activation and loss. Multiplying by a sigmoid derivative again would be incorrect. For sigmoid with half squared error, the sigmoid derivative is still required.

For a batch, sum per-example parameter gradients; divide by the batch size only if the objective is a batch mean. State the output reduction and batch reduction separately when both appear in a question.

Backpropagation with one hidden neuron

Definition. Backpropagation computes the derivatives of a scalar objective with respect to network parameters by applying the chain rule in reverse through the computational graph. An optimizer then uses these derivatives to update the parameters.

Network equations and the meaning of delta

Consider one input, one hidden neuron and one output neuron:

For the half squared loss used in this example:

$L=\frac12(\hat y-y)^2$ LaTeX: L=\frac12(\hat y-y)^2

Define a neuron's delta as the derivative of the loss with respect to its preactivation, not with respect to its output:

$\delta_o=\frac{\partial L}{\partial z_o}=(\hat y-y)g'(z_o)$ LaTeX: \delta_o=\frac{\partial L}{\partial z_o}=(\hat y-y)g'(z_o)

The output-weight derivative contains the hidden activation because zₒ = w₂h + b₂:

$\frac{\partial L}{\partial w_2}=\delta_o h,\qquad\frac{\partial L}{\partial b_2}=\delta_o$ LaTeX: \frac{\partial L}{\partial w_2}=\delta_o h,\qquad\frac{\partial L}{\partial b_2}=\delta_o

To reach the hidden neuron, continue backwards through the old output weight and the hidden activation:

$\delta_h=\delta_o w_2 f'(z_h)$ LaTeX: \delta_h=\delta_o w_2 f'(z_h)

$\frac{\partial L}{\partial w_1}=\delta_h x,\qquad\frac{\partial L}{\partial b_1}=\delta_h$ LaTeX: \frac{\partial L}{\partial w_1}=\delta_h x,\qquad\frac{\partial L}{\partial b_1}=\delta_h

The bias derivative equals the corresponding delta because the derivative of z with respect to its additive bias is 1. All these derivatives belong to the same forward pass; an updated weight must not be used partway through this backward pass.

Complete worked numerical

Question. x = 2, y = 1; w₁ = 0.5, b₁ = 0.1; w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1.

Given

x = 2, y = 1.

w₁ = 0.5, b₁ = 0.1.

w₂ = 0.8, b₂ = 0.2.

Both neurons use ReLU.

Use half squared loss.

Find the output and all four parameter gradients, then update all four parameters using η = 0.1.

To find

Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters.

Formula — which one and why

Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.

x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.

$z_h=W_1x+b_1$ LaTeX: z_h=W_1x+b_1

$h=f_1(z_h)$ LaTeX: h=f_1(z_h)

$z_o=W_2h+b_2$ LaTeX: z_o=W_2h+b_2

$o=f_2(z_o)$ LaTeX: o=f_2(z_o)

Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.

Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.

oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.

$L=\frac12\sum_j(o_j-y_j)^2$ LaTeX: L=\frac12\sum_j(o_j-y_j)^2

$\frac{\partial L}{\partial o_j}=o_j-y_j$ LaTeX: \frac{\partial L}{\partial o_j}=o_j-y_j

Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.

Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.

δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.

$\delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})$ LaTeX: \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})

$\delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}$ LaTeX: \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}

$\frac{\partial L}{\partial W_{ji}}=\delta_ja_i$ LaTeX: \frac{\partial L}{\partial W_{ji}}=\delta_ja_i

$\frac{\partial L}{\partial b_j}=\delta_j$ LaTeX: \frac{\partial L}{\partial b_j}=\delta_j

$W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}$ LaTeX: W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}

Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.

Solution — step by step

The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights. Tap the diagram to enlarge.

Step 1: forward propagation through the hidden neuron.

$z_h=0.5(2)+0.1=1.1$ LaTeX: z_h=0.5(2)+0.1=1.1

$h=\operatorname{ReLU}(1.1)=1.1$ LaTeX: h=\operatorname{ReLU}(1.1)=1.1

Step 2: forward propagation through the output neuron.

$z_o=0.8(1.1)+0.2=1.08$ LaTeX: z_o=0.8(1.1)+0.2=1.08

$\hat y=\operatorname{ReLU}(1.08)=1.08$ LaTeX: \hat y=\operatorname{ReLU}(1.08)=1.08

ReLU is not bounded above by 1, so this output is valid. It is not being interpreted as a probability here.

Step 3: calculate the loss and output delta.

$L=\frac12(1.08-1)^2=0.0032$ LaTeX: L=\frac12(1.08-1)^2=0.0032

Both preactivations are positive, so both ReLU derivatives are 1.

$\delta_o=(1.08-1)(1)=0.08$ LaTeX: \delta_o=(1.08-1)(1)=0.08

Step 4: calculate the output-layer parameter gradients.

$\frac{\partial L}{\partial w_2}=0.08(1.1)=0.088$ LaTeX: \frac{\partial L}{\partial w_2}=0.08(1.1)=0.088

$\frac{\partial L}{\partial b_2}=0.08$ LaTeX: \frac{\partial L}{\partial b_2}=0.08

Step 5: propagate the derivative to the hidden neuron.

Use w₂ = 0.8, its value during the forward pass:

$\delta_h=0.08(0.8)(1)=0.064$ LaTeX: \delta_h=0.08(0.8)(1)=0.064

$\frac{\partial L}{\partial w_1}=0.064(2)=0.128$ LaTeX: \frac{\partial L}{\partial w_1}=0.064(2)=0.128

$\frac{\partial L}{\partial b_1}=0.064$ LaTeX: \frac{\partial L}{\partial b_1}=0.064

Step 6: update each parameter using its own gradient.

Parameter	Gradient from the original forward pass	Updated value
w₁	0.128	0.4872
b₁	0.064	0.0936
w₂	0.088	0.7912
b₂	0.08	0.192

Final answer: the original prediction is 1.08 and its loss is 0.0032. The table contains all requested gradients and updated parameters. A new forward pass would use all four new parameters together.

Drill 13 Full ReLU backpropagation

Question

Question. Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases.

Worked example: all incoming weights and biases Tap the diagram to enlarge.