# F22 Q2a

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-s5

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s5/

## Question

**Question (10 marks).** Explain GRU advantages over a normal RNN. For “The student who did not submit the assignment was bound to loose marks,” show how memory and update gates can retain the subject before predicting “was”.

## Answer — step by step

A GRU uses an update gate and reset gate, with one hidden state and no separate LSTM-style cell state. Under the convention used here, z admits the new candidate:



$$
z_t=\sigma(W_zx_t+U_zh_{t-1}+b_z),\quad r_t=\sigma(W_rx_t+U_rh_{t-1}+b_r)
$$





$$
\widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h),\quad h_t=(1-z_t)\odot h_{t-1}+z_t\odot\widetilde h_t
$$



![Reset controls candidate context and update controls the old new mixture](https://deep-learning-study-room.vercel.app/assets/figures/gru.svg)

Diagram: Reset controls candidate context and update controls the old new mixture. Labels: h previous | Reset r × h | Candidate = tanh(...) | (1 − z) × h previous | z × candidate | + | New hidden state | Here z means the fraction of NEW content admitted.

Reset controls candidate context and update controls the old new mixture Tap the diagram to enlarge.

The reset gate controls how much prior hidden information enters the candidate. The update gate balances retention and replacement. A small z retains most of the old state under this formula; another textbook convention swaps z and 1−z. State the formula before interpreting the gate.

Compared with an LSTM's input, forget and output gates plus candidate transform, a standard GRU has two gates plus a candidate transform and usually fewer parameters at equal hidden width. This may reduce training cost and the amount of data needed to estimate parameters, but small-dataset superiority is not guaranteed. Validation should decide. To ignore an unhelpful token, a trained GRU can keep the prior state by admitting little candidate content; it still executes the recurrence rather than literally deleting the token. Both GRU and LSTM help long-range learning compared with an ungated vanilla RNN.

**Important data limit:**  no numerical weights or token vectors are supplied, so actual gate outputs cannot be uniquely calculated. The following is an explicitly idealized one-component illustration: candidate subject feature = 1 at “student”; use hnew = (1−z) hold+z candidate. Exact 0/1 gates are limiting idealizations of sigmoid values.

| Time and token | Illustrative z | Singular-subject memory after step |
| --- | --- | --- |
| 1 The | 0 while retaining this illustrative feature | 0 before subject is read |
| 2 student | 1 when admitting subject feature | 1 after subject is read |
| 3 who | 0 while retaining this illustrative feature | 1 after subject is read |
| 4 did | 0 while retaining this illustrative feature | 1 after subject is read |
| 5 not | 0 while retaining this illustrative feature | 1 after subject is read |
| 6 submit | 0 while retaining this illustrative feature | 1 after subject is read |
| 7 the | 0 while retaining this illustrative feature | 1 after subject is read |
| 8 assignment | 0 while retaining this illustrative feature | 1 after subject is read |

At “student”: hnew = (1−1) × 0+1 × 1 = 1. At each intervening token in this illustrative component: hnew = (1−0) × 1+0 × candidate = 1. The retained singular subject helps the output layer choose “was”. Other hidden components may still update to represent intervening words; this table is not a computed trace from an actual trained network.