Deep Learning · TEE · section

Teams photos full sigmoid example

Current MT1 and the two teacher backprop examples

The three photographs show this 2 → 2 → 2 network and its chain rule. The photo label for w₃ is read as 0.25, consistent with the visible output/error values and the matching worked example in S05. Learning rate is not legible/specified in these photos; η = 0.5 below is an explicit practice assumption.

Question. Input x = [0.05, 0.1]. Each row below lists the incoming weights of one receiving neuron. W₁ = $\begin{bmatrix}0.15 & 0.2\\0.25 & 0.3\end{bmatrix}$ ; b₁ = [0.35, 0.35]; W₂ = $\begin{bmatrix}0.4 & 0.45\\0.5 & 0.55\end{bmatrix}$ ; b₂ = [0.6, 0.6]. Hidden and output activations are sigmoid. Target y = [0.01, 0.99]. Learning rate η = 0.5. Compute all outputs, total half squared loss, all eight weight gradients and one simultaneous weight update. Map weights in order: w₁ = x₁ → h₁, w₂ = x₂ → h₁, w₃ = x₁ → h₂, w₄ = x₂ → h₂; w₅ = h₁ → o₁, w₆ = h₂ → o₁, w₇ = h₁ → o₂, w₈ = h₂ → o₂.

Given

Input x = [0.05, 0.1].

W₁ = $\begin{bmatrix}0.15 & 0.2\\0.25 & 0.3\end{bmatrix}$ .

b₁ = [0.35, 0.35].

W₂ = $\begin{bmatrix}0.4 & 0.45\\0.5 & 0.55\end{bmatrix}$ .

b₂ = [0.6, 0.6].

Hidden and output activations are sigmoid.

Target y = [0.01, 0.99].

Learning rate η = 0.5.

To find

Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.

Formula — which one and why

Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.

x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.

$z_h=W_1x+b_1$ LaTeX: z_h=W_1x+b_1

$h=f_1(z_h)$ LaTeX: h=f_1(z_h)

$z_o=W_2h+b_2$ LaTeX: z_o=W_2h+b_2

$o=f_2(z_o)$ LaTeX: o=f_2(z_o)

Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.

Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.

oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.

$L=\frac12\sum_j(o_j-y_j)^2$ LaTeX: L=\frac12\sum_j(o_j-y_j)^2

$\frac{\partial L}{\partial o_j}=o_j-y_j$ LaTeX: \frac{\partial L}{\partial o_j}=o_j-y_j

Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.

Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.

$p=\frac{1}{1+e^{-z}}$ LaTeX: p=\frac{1}{1+e^{-z}}

$\frac{dp}{dz}=p(1-p)$ LaTeX: \frac{dp}{dz}=p(1-p)

Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).

Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.

δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.

$\delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})$ LaTeX: \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})

$\delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}$ LaTeX: \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}

$\frac{\partial L}{\partial W_{ji}}=\delta_ja_i$ LaTeX: \frac{\partial L}{\partial W_{ji}}=\delta_ja_i

$\frac{\partial L}{\partial b_j}=\delta_j$ LaTeX: \frac{\partial L}{\partial b_j}=\delta_j

$W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}$ LaTeX: W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}

Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.

Solution — step by step

Every arrow in the teacher example is labeled so the two backward paths can be traced Tap the diagram to enlarge.

Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.

Hidden neuron 1:

Hidden neuron 2:

Output neuron 1:

Output neuron 2:

Step 2 loss convention. The photos explicitly use ½(t₁−o₁)² + ½(t₂−o₂)². With exactly two output components, this equals their mean squared error, 0.298371109. The unhalved sum is 0.596742218.

Prediction minus target = [0.74136507, -0.217071535]. Squared errors = [0.549622166, 0.047120051]. Half-sum squared loss = 0.298371109; mean squared error = 0.298371109. These are different unless the number of output components makes the scale coincide.

Output delta 1:

The derivative is o(1−o).

Output delta 2:

The derivative is o(1−o).

Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.

h1 → o1:

The calculation on h1 → o1

Sending activation0.593269992h1

Receiving delta0.138498562δ at o1

Connection gradient0.082167041h1 → o1

Now update this weight

Old weight0.4

New weight0.35891648

Subtract η × gradient = 0.04108352. This weight decreases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.

$g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1$ LaTeX: g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1

Apply the learning rate to this gradient and subtract the result from the old weight.

h2 → o1:

The calculation on h2 → o1

Sending activation0.596884378h2

Receiving delta0.138498562δ at o1

Connection gradient0.082667628h2 → o1

Now update this weight

Old weight0.45

New weight0.408666186

Subtract η × gradient = 0.041333814. This weight decreases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.

$g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2$ LaTeX: g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2

Apply the learning rate to this gradient and subtract the result from the old weight.

h1 → o2:

The calculation on h1 → o2

Sending activation0.593269992h1

Receiving delta−0.038098237δ at o2

Connection gradient−0.02260254h1 → o2

Now update this weight

Old weight0.5

New weight0.51130127

Subtract η × gradient = −0.01130127. This weight increases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.

$g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1$ LaTeX: g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1

Apply the learning rate to this gradient and subtract the result from the old weight.

h2 → o2:

The calculation on h2 → o2

Sending activation0.596884378h2

Receiving delta−0.038098237δ at o2

Connection gradient−0.022740242h2 → o2

Now update this weight

Old weight0.55

New weight0.561370121

Subtract η × gradient = −0.011370121. This weight increases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.

$g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2$ LaTeX: g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2

Apply the learning rate to this gradient and subtract the result from the old weight.

Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.

Hidden 1 downstream contribution:

Calculate each outgoing path using the old outgoing weight and that output's delta.

Several output paths meet at this hidden neuron

Output 1

Output delta0.138498562

Old outgoing weight0.4

Path contribution0.055399425

Output 2

Output delta−0.038098237

Old outgoing weight0.5

Path contribution−0.019049118

Add every path contribution0.036350306Then multiply by the hidden activation derivative.

Multiply within each path. Add the resulting contributions at the shared hidden neuron.

Contribution from output 1.

Contribution from output 2.

Accumulate the contributions, then apply the hidden activation derivative below.

Hidden 2 downstream contribution:

Calculate each outgoing path using the old outgoing weight and that output's delta.

Several output paths meet at this hidden neuron

Output 1

Output delta0.138498562

Old outgoing weight0.45

Path contribution0.062324353

Output 2

Output delta−0.038098237

Old outgoing weight0.55

Path contribution−0.02095403

Add every path contribution0.041370323Then multiply by the hidden activation derivative.

Multiply within each path. Add the resulting contributions at the shared hidden neuron.

Contribution from output 1.

Contribution from output 2.

Accumulate the contributions, then apply the hidden activation derivative below.

Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.

x1 → h1:

The calculation on x1 → h1

Sending input0.05x1

Receiving delta0.008771355δ at h1

Connection gradient0.000438568x1 → h1

Now update this weight

Old weight0.15

New weight0.149780716

Subtract η × gradient = 0.000219284. This weight decreases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.

$g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1$ LaTeX: g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1

Apply the learning rate to this gradient and subtract the result from the old weight.

x2 → h1:

The calculation on x2 → h1

Sending input0.1x2

Receiving delta0.008771355δ at h1

Connection gradient0.000877135x2 → h1

Now update this weight

Old weight0.2

New weight0.199561432

Subtract η × gradient = 0.000438568. This weight decreases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.

$g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2$ LaTeX: g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2

Apply the learning rate to this gradient and subtract the result from the old weight.

x1 → h2:

The calculation on x1 → h2

Sending input0.05x1

Receiving delta0.009954255δ at h2

Connection gradient0.000497713x1 → h2

Now update this weight

Old weight0.25

New weight0.249751144

Subtract η × gradient = 0.000248856. This weight decreases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.

$g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1$ LaTeX: g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1

Apply the learning rate to this gradient and subtract the result from the old weight.

x2 → h2:

The calculation on x2 → h2

Sending input0.1x2

Receiving delta0.009954255δ at h2

Connection gradient0.000995425x2 → h2

Now update this weight

Old weight0.3

New weight0.299502287

Subtract η × gradient = 0.000497712. This weight decreases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.

$g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2$ LaTeX: g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2

Apply the learning rate to this gradient and subtract the result from the old weight.

Step 6 bias updates. A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.

These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.

Why the photos add two terms for h₁: dL/dh₁ = δo₁w₅ + δo₂w₇ = 0.138498562 × 0.4 + (−0.038098237) × 0.5 = 0.055399425−0.019049118 = 0.036350306. Now multiply by h₁(1−h₁) = 0.241300709 to obtain δh₁ = 0.008771355; multiply by x₁ = 0.05 to obtain dL/dw₁ = 0.000438568. Adding the two output paths is essential.

Memory sequence: error → output slope → outgoing weight → add paths → hidden slope → input. If you can explain why each factor appears, changed numbers or extra output neurons are much less confusing.

Bias convention: the photo draws one bias value per layer. If these are literally tied scalars rather than equal initial per-neuron biases, sum the relevant deltas: hidden shared b becomes 0.35−0.5(0.008771355+0.009954255) = 0.340637195; output shared b becomes 0.6−0.5(0.138498562−0.038098237) = 0.549799837. If only weights are requested, do not invent a bias update requirement.