Deep Learning · TEE · section

Rehearsal Q4b

Complete TEE paper rehearsal

Question (10 marks). Define vanishing gradients in RNNs and explain why they occur. Use “The dogs barked at the thief and … were appreciated” to discuss long context. Explain LSTM forgetting/retention, then compute the retained contribution for f = [0.1,0.9], Cprevious = [2,1].

Complete worked answer.

Given

Forget-gate outputs f = [0.1,0.9]; old cell = [2,1].

No input gate or candidate is supplied.

To find

Explain forgetting/retention and calculate the retained old-cell part only.

Formula — which one and why

LSTM retained old memory. Multiply each old cell value by its corresponding forget-gate value. This gives only the old contribution, not the complete new cell.

f = activated forget gate; C_old = previous cell state; ⊙ means multiply matching entries.

C_{\mathrm{retained}}=f\odot C_{\mathrm{old}}LaTeX: C_{\mathrm{retained}}=f\odot C_{\mathrm{old}}

Yaad rakhne ka cue: Forget gate × old cell = bacha hua purana hissa.

Solution — step by step

Backpropagation through time trains an RNN by treating its time-unrolled computations as a deep computational graph.

1. Initialize the hidden state and run the forward recurrence from the first input to the last.

2. Compute the specified output loss, at the final step or across several steps.

3. Start from the last loss and propagate derivatives backwards through output and hidden operations.

4. At each earlier state, add the direct output-loss contribution and the contribution arriving from future states.

5. Sum all uses of each shared weight before applying the optimizer update.

\delta_t=(W_{hy}^{T}\delta_t^{out}+W_{hh}^{T}\delta_{t+1})\odot(1-h_t^2),\quad \frac{\partial L}{\partial W_{hh}}=\sum_t\delta_t h_{t-1}^{T}LaTeX: \delta_t=(W_{hy}^{T}\delta_t^{out}+W_{hh}^{T}\delta_{t+1})\odot(1-h_t^2),\quad \frac{\partial L}{\partial W_{hh}}=\sum_t\delta_t h_{t-1}^{T}

The tanh derivative is 1−h². Repeated multiplication by recurrent weights and activation derivatives may shrink gradients toward zero or amplify them greatly. Early words then receive a weak learning signal or updates become unstable. Gated architectures help long-context learning; suitable initialization and learning rates help training; clipping limits exploding gradients. Truncated BPTT restricts the backward time window to reduce computation and memory. It does not mean creating independent weights for each time step.

BPTT follows this unrolled network backwards and sums gradients of shared weights Tap the diagram to enlarge.

The plural subject “dogs” must remain represented across intervening words. A vanilla RNN may fail to learn that dependency when early gradients are weak; it is not guaranteed to fail on every such sentence. In an LSTM, forget components close to 1 retain relevant subject information along the cell path.

Step 1: retain the first old-cell component.

0.1(2)=0.2LaTeX: 0.1(2)=0.2

Step 2: retain the second old-cell component.

0.9(1)=0.9LaTeX: 0.9(1)=0.9

Step 3: assemble the retained old-cell contribution.

f\odot C_{\mathrm{old}}=[0.2,0.9]LaTeX: f\odot C_{\mathrm{old}}=[0.2,0.9]

Final answer: [0.2,0.9] is only retained old content. The input gate and candidate are missing, so the complete updated cell cannot be calculated.