Deep Learning · TEE · section

F22 Q2b

Final examination November 2022 fully solved

Question

Question (10 marks). For a three-stage sigmoid network, use samples [0.3,0.2,0.4] → 0 and [0.5,0.6,0.8] → 1. First-stage feature multipliers are [0.3,0.4,0.6]. Next receiving rows are [0.5,0.2,0.3], [0.3,0.3,0.3], and output weights [0.3,0.3]. Find both predictions, average MSE and output-weight updates after the first backward pass with η = 0.2. Assume zero biases.

Answer — step by step

Given

For a three-stage sigmoid network, use samples [0.3,0.2,0.4] → 0 and [0.5,0.6,0.8] → 1.

First-stage feature multipliers are [0.3,0.4,0.6].

Next receiving rows are [0.5,0.2,0.3], [0.3,0.3,0.3], and output weights [0.3,0.3].

Find both predictions, average MSE and output-weight updates after the first backward pass with η = 0.2.

Assume zero biases.

To find

Find all preactivations, hidden activations and final outputs. Calculate MSE and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.

Formula — which one and why

Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.

x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.

$z_h=W_1x+b_1$ LaTeX: z_h=W_1x+b_1

$h=f_1(z_h)$ LaTeX: h=f_1(z_h)

$z_o=W_2h+b_2$ LaTeX: z_o=W_2h+b_2

$o=f_2(z_o)$ LaTeX: o=f_2(z_o)

Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.

Mean squared loss. Average the squared scalar prediction errors. The derivative retains the factor 2/n; n must count the outputs included in this mean.

n = number of scalar prediction errors in the stated mean; oⱼ = prediction; yⱼ = target.

$L=\frac1n\sum_j(o_j-y_j)^2$ LaTeX: L=\frac1n\sum_j(o_j-y_j)^2

$\frac{\partial L}{\partial o_j}=\frac2n(o_j-y_j)$ LaTeX: \frac{\partial L}{\partial o_j}=\frac2n(o_j-y_j)

Yaad rakhne ka cue: MSE mein derivative 2/n times error; half SSE ka rule blindly mat lagao.

Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.

$p=\frac{1}{1+e^{-z}}$ LaTeX: p=\frac{1}{1+e^{-z}}

$\frac{dp}{dz}=p(1-p)$ LaTeX: \frac{dp}{dz}=p(1-p)

Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).

Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.

δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.

$\delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})$ LaTeX: \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})

$\delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}$ LaTeX: \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}

$\frac{\partial L}{\partial W_{ji}}=\delta_ja_i$ LaTeX: \frac{\partial L}{\partial W_{ji}}=\delta_ja_i

$\frac{\partial L}{\partial b_j}=\delta_j$ LaTeX: \frac{\partial L}{\partial b_j}=\delta_j

$W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}$ LaTeX: W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}

Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.

Solution — step by step

Interpretation. Follow the supplied diagram: each raw feature is multiplied by 0.3, 0.4 or 0.6, then passed through sigmoid before the next two-neuron layer. Unspecified biases are zero. MSE is averaged over the two samples. The phrase “first backward pass” does not specify batch versus single-sample updating, so both interpretations are identified below.

Step 1: apply the first-stage multipliers and sigmoid

For sample 1, the preactivations are [0.3 × 0.3, 0.2 × 0.4, 0.4 × 0.6] = [0.09, 0.08, 0.24]. Sigmoid gives [0.522484825, 0.519989340, 0.559713649].

For sample 2, they are [0.5 × 0.3, 0.6 × 0.4, 0.8 × 0.6] = [0.15, 0.24, 0.48]. Sigmoid gives [0.537429845, 0.559713649, 0.617747875].

Step 2: evaluate the two hidden neurons

For each first-stage output a, calculate z₁ = 0.5a₁ + 0.2a₂ + 0.3a₃ and z₂ = 0.3a₁ + 0.3a₂ + 0.3a₃, then apply sigmoid.

Sample	Hidden preactivation 1	Hidden preactivation 2	Hidden output 1	Hidden output 2
1	0.533154375	0.480656344	0.630218519	0.617902849
2	0.565982015	0.514467411	0.637835530	0.625853155

For example, sample 1's first hidden sum is 0.5(0.522484825) + 0.2(0.519989340) + 0.3(0.559713649) = 0.533154375.

Step 3: evaluate the output neuron

The output preactivation is 0.3h₁ + 0.3h₂; apply sigmoid to the complete sum.

Sample	Output preactivation	Prediction	Target	Squared error
1	0.374436410	0.592530535	0	0.351092435
2	0.379106605	0.593657609	1	0.165114139

Step 4: calculate the average MSE

MSE = (0.351092435 + 0.165114139)/2 = 0.258103287.

Step 5: differentiate each sample's squared error

For one sample, output delta = 2(o − y)o(1 − o); each output-weight gradient is this delta multiplied by its sending hidden activation. Calculate both samples at the original weights before taking their mean.

Sample	Output delta	Gradient for h₁ → output	Gradient for h₂ → output
1	0.286118893	0.180317425	0.176793679
2	−0.196042530	−0.125042891	−0.122693836

Step 6: average gradients and make one batch update

The mean gradients are [(0.180317425 − 0.125042891)/2, (0.176793679 − 0.122693836)/2] ≈ [0.027637267, 0.027049922].

First weight: 0.3 − 0.2(0.027637267) = 0.294472547.

Second weight: 0.3 − 0.2(0.027049922) = 0.294590016.

Final answer under batch MSE: predictions [0.592530535, 0.593657609], average MSE 0.258103287, updated output weights [0.294472547, 0.294590016]. Averaging the sample gradients is required because the stated objective is their mean loss.

Alternative: first-sample SGD

If “first backward pass” means an update after sample 1 alone, use its gradients without averaging: 0.3 − 0.2(0.180317425) = 0.263936515 and 0.3 − 0.2(0.176793679) = 0.264641264. This is a different update schedule. State the interpretation and do not combine a batch loss with an unexplained single-sample gradient.