# R25 Q3a · 4–3–2 forward pass and dropout

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s6

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s6/

## Question

**Question. For the printed network, inputs=[0.1,0.2,0.3,0.4], targets=[0.01,0.99], w1…w18=[0.10,0.15,…,0.95] and both layer biases=0.60. Find total forward error, then drop hidden neuron H12 and compare errors.** (10 marks)

## Answer — step by step

### Given

Input→hidden rows (destinations H11,H12,H13):



$$
\begin{bmatrix}0.1&0.25&0.4&0.55\\0.15&0.3&0.45&0.6\\0.2&0.35&0.5&0.65\end{bmatrix}
$$



Hidden→output rows (destinations O1,O2):



$$
\begin{bmatrix}0.7&0.8&0.9\\0.75&0.85&0.95\end{bmatrix}
$$



4inputs → 3sigmoid hidden units → 2sigmoid outputs → Half sum of squared errors

### To find

Original and dropped forward outputs and loss.

### Formula: which one and why

The paper omits activation and error convention. Declare sigmoid at both layers and half-SSE. For the given single dropped unit use an ordinary binary mask, no additional inverted-dropout scaling because no keep probability is supplied.



$$
h_j=\sigma(\sum_iW_{ji}x_i+0.6),\quad o_k=\sigma(\sum_jV_{kj}h_j+0.6),\quad E=\frac12\sum_k(o_k-t_k)^2
$$



### Step 1. Multiply and add each hidden neuron

| Neuron | Products + bias | Net | Sigmoid |
| --- | --- | --- | --- |
| 1 | 0.1×0.1 + 0.2×0.25 + 0.3×0.4 + 0.4×0.55 +0.6 | 1.000000000 | 0.731058578630 |
| 2 | 0.1×0.15 + 0.2×0.3 + 0.3×0.45 + 0.4×0.6 +0.6 | 1.050000000 | 0.740774899182 |
| 3 | 0.1×0.2 + 0.2×0.35 + 0.3×0.5 + 0.4×0.65 +0.6 | 1.100000000 | 0.750260105595 |

### Step 2. Compute original output nets and activations

| Output | Full substitution | Net | Output |
| --- | --- | --- | --- |
| 1 | 0.731058578630×0.7 + 0.740774899182×0.8 + 0.750260105595×0.9 +0.6 | 2.379595019422 | 0.915258028946 |
| 2 | 0.731058578630×0.75 + 0.740774899182×0.85 + 0.750260105595×0.95 +0.6 | 2.490699698593 | 0.923487256983 |

### Step 3. Square both output errors and halve their sum

| Output | Prediction−target | Square | Half square |
| --- | --- | --- | --- |
| 1 | 0.9152580289459775−0.01=0.905258028946 | 0.819492098971 | 0.409746049486 |
| 2 | 0.9234872569833742−0.99=-0.066512743017 | 0.004423944984 | 0.002211972492 |

Total=0.411958021977.

### Step 4. Drop H12 only and recompute both outputs

Masked hidden activations=[0.7310585786300049, 0, 0.7502601055951177]. Other hidden activations and all weights are unchanged.

| Output | Net with H12 term zero | Sigmoid |
| --- | --- | --- |
| 1 | 0.731058578630×0.7+0×0.8+0.750260105595×0.9+0.6=1.786975100077 | 0.856556014402 |
| 2 | 0.731058578630×0.75+0×0.85+0.750260105595×0.95+0.6=1.861041034288 | 0.865418242930 |

### Step 5. Compute dropped loss and compare

Half squares: 0.5×(0.856556014402−0.01)² + 0.5×(0.865418242930−0.99)² = 0.366088849857. Difference dropped−original=-0.045869172120.

### Final answer

**Original E=0.411958021977; dropped E=0.366088849857.** A lower single-example masked loss is not evidence that dropout always lowers training or test loss.

**Source / convention note.** Activation/loss/dropout scaling missing from original; result is conditional on explicit standard classroom assumptions.