Deep Learning · TEE · section

F22 Q2a

Final examination November 2022 fully solved

Question

Question (10 marks). Explain GRU advantages over a normal RNN. For “The student who did not submit the assignment was bound to loose marks,” show how memory and update gates can retain the subject before predicting “was”.

Answer — step by step

A GRU uses an update gate and reset gate, with one hidden state and no separate LSTM-style cell state. Under the convention used here, z admits the new candidate:

z_t=\sigma(W_zx_t+U_zh_{t-1}+b_z),\quad r_t=\sigma(W_rx_t+U_rh_{t-1}+b_r)LaTeX: z_t=\sigma(W_zx_t+U_zh_{t-1}+b_z),\quad r_t=\sigma(W_rx_t+U_rh_{t-1}+b_r)
\widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h),\quad h_t=(1-z_t)\odot h_{t-1}+z_t\odot\widetilde h_tLaTeX: \widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h),\quad h_t=(1-z_t)\odot h_{t-1}+z_t\odot\widetilde h_t
Reset controls candidate context and update controls the old new mixture Tap the diagram to enlarge.

The reset gate controls how much prior hidden information enters the candidate. The update gate balances retention and replacement. A small z retains most of the old state under this formula; another textbook convention swaps z and 1−z. State the formula before interpreting the gate.

Compared with an LSTM's input, forget and output gates plus candidate transform, a standard GRU has two gates plus a candidate transform and usually fewer parameters at equal hidden width. This may reduce training cost and the amount of data needed to estimate parameters, but small-dataset superiority is not guaranteed. Validation should decide. To ignore an unhelpful token, a trained GRU can keep the prior state by admitting little candidate content; it still executes the recurrence rather than literally deleting the token. Both GRU and LSTM help long-range learning compared with an ungated vanilla RNN.

Important data limit: no numerical weights or token vectors are supplied, so actual gate outputs cannot be uniquely calculated. The following is an explicitly idealized one-component illustration: candidate subject feature = 1 at “student”; use hnew = (1−z) hold+z candidate. Exact 0/1 gates are limiting idealizations of sigmoid values.

Time and tokenIllustrative zSingular-subject memory after step
1 The0 while retaining this illustrative feature0 before subject is read
2 student1 when admitting subject feature1 after subject is read
3 who0 while retaining this illustrative feature1 after subject is read
4 did0 while retaining this illustrative feature1 after subject is read
5 not0 while retaining this illustrative feature1 after subject is read
6 submit0 while retaining this illustrative feature1 after subject is read
7 the0 while retaining this illustrative feature1 after subject is read
8 assignment0 while retaining this illustrative feature1 after subject is read

At “student”: hnew = (1−1) × 0+1 × 1 = 1. At each intervening token in this illustrative component: hnew = (1−0) × 1+0 × candidate = 1. The retained singular subject helps the output layer choose “was”. Other hidden components may still update to represent intervening words; this table is not a computed trace from an actual trained network.