Deep Learning · TEE · section

R25 Q3a · 4–3–2 forward pass and dropout

February 2025 re-exam · complete solved paper

Question

Question. For the printed network, inputs=[0.1,0.2,0.3,0.4], targets=[0.01,0.99], w1…w18=[0.10,0.15,…,0.95] and both layer biases=0.60. Find total forward error, then drop hidden neuron H12 and compare errors. (10 marks)

Answer — step by step

Given

Input→hidden rows (destinations H11,H12,H13):

$\begin{bmatrix}0.1&0.25&0.4&0.55\\0.15&0.3&0.45&0.6\\0.2&0.35&0.5&0.65\end{bmatrix}$ LaTeX: \begin{bmatrix}0.1&0.25&0.4&0.55\\0.15&0.3&0.45&0.6\\0.2&0.35&0.5&0.65\end{bmatrix}

Hidden→output rows (destinations O1,O2):

$\begin{bmatrix}0.7&0.8&0.9\\0.75&0.85&0.95\end{bmatrix}$ LaTeX: \begin{bmatrix}0.7&0.8&0.9\\0.75&0.85&0.95\end{bmatrix}

4inputs → 3sigmoid hidden units → 2sigmoid outputs → Half sum of squared errors

To find

Original and dropped forward outputs and loss.

Formula: which one and why

The paper omits activation and error convention. Declare sigmoid at both layers and half-SSE. For the given single dropped unit use an ordinary binary mask, no additional inverted-dropout scaling because no keep probability is supplied.

$h_j=\sigma(\sum_iW_{ji}x_i+0.6),\quad o_k=\sigma(\sum_jV_{kj}h_j+0.6),\quad E=\frac12\sum_k(o_k-t_k)^2$ LaTeX: h_j=\sigma(\sum_iW_{ji}x_i+0.6),\quad o_k=\sigma(\sum_jV_{kj}h_j+0.6),\quad E=\frac12\sum_k(o_k-t_k)^2

Step 1. Multiply and add each hidden neuron

Neuron	Products + bias	Net	Sigmoid
1	0.1×0.1 + 0.2×0.25 + 0.3×0.4 + 0.4×0.55 +0.6	1.000000000	0.731058578630
2	0.1×0.15 + 0.2×0.3 + 0.3×0.45 + 0.4×0.6 +0.6	1.050000000	0.740774899182
3	0.1×0.2 + 0.2×0.35 + 0.3×0.5 + 0.4×0.65 +0.6	1.100000000	0.750260105595

Step 2. Compute original output nets and activations

Output	Full substitution	Net	Output
1	0.731058578630×0.7 + 0.740774899182×0.8 + 0.750260105595×0.9 +0.6	2.379595019422	0.915258028946
2	0.731058578630×0.75 + 0.740774899182×0.85 + 0.750260105595×0.95 +0.6	2.490699698593	0.923487256983

Step 3. Square both output errors and halve their sum

Output	Prediction−target	Square	Half square
1	0.9152580289459775−0.01=0.905258028946	0.819492098971	0.409746049486
2	0.9234872569833742−0.99=-0.066512743017	0.004423944984	0.002211972492

Total=0.411958021977.

Step 4. Drop H12 only and recompute both outputs

Masked hidden activations=[0.7310585786300049, 0, 0.7502601055951177]. Other hidden activations and all weights are unchanged.

Output	Net with H12 term zero	Sigmoid
1	0.731058578630×0.7+0×0.8+0.750260105595×0.9+0.6=1.786975100077	0.856556014402
2	0.731058578630×0.75+0×0.85+0.750260105595×0.95+0.6=1.861041034288	0.865418242930

Step 5. Compute dropped loss and compare

Half squares: 0.5×(0.856556014402−0.01)² + 0.5×(0.865418242930−0.99)² = 0.366088849857. Difference dropped−original=-0.045869172120.

Final answer

Original E=0.411958021977; dropped E=0.366088849857. A lower single-example masked loss is not evidence that dropout always lowers training or test loss.

Source / convention note. Activation/loss/dropout scaling missing from original; result is conditional on explicit standard classroom assumptions.