F23 Q6a

Answer — step by step

Terminology correction: the pictured LSTM uses forget, input and output gates plus a candidate. Its input/candidate path updates the cell. A reset gate belongs to GRU, so no genuine LSTM reset-gate equation exists. The complete consistent LSTM answer is:

An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.

LaTeX: f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i)

LaTeX: \widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c),\quad C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t

LaTeX: o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\quad h_t=o_t\odot\tanh(C_t)

Diagram: The cell path retains selected old content and adds selected new content. Labels: C previous | × | + | C new | tanh | × | h | new | σ: f | σ: i | tanh: C~ | σ: o | × | [h previous, x current] | Each gate has its own learned weights and bias.

The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.

In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.

For the explicitly named GRU terms in the question:

A GRU uses an update gate and reset gate, with one hidden state and no separate LSTM-style cell state. Under the convention used here, z admits the new candidate:

LaTeX: z_t=\sigma(W_zx_t+U_zh_{t-1}+b_z),\quad r_t=\sigma(W_rx_t+U_rh_{t-1}+b_r)

LaTeX: \widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h),\quad h_t=(1-z_t)\odot h_{t-1}+z_t\odot\widetilde h_t

Diagram: Reset controls candidate context and update controls the old new mixture. Labels: h previous | Reset r × h | Candidate = tanh(...) | (1 − z) × h previous | z × candidate | + | New hidden state | Here z means the fraction of NEW content admitted.

The reset gate controls how much prior hidden information enters the candidate. The update gate balances retention and replacement. A small z retains most of the old state under this formula; another textbook convention swaps z and 1−z. State the formula before interpreting the gate.

Compared with an LSTM's input, forget and output gates plus candidate transform, a standard GRU has two gates plus a candidate transform and usually fewer parameters at equal hidden width. This may reduce training cost and the amount of data needed to estimate parameters, but small-dataset superiority is not guaranteed. Validation should decide. To ignore an unhelpful token, a trained GRU can keep the prior state by admitting little candidate content; it still executes the recurrence rather than literally deleting the token. Both GRU and LSTM help long-range learning compared with an ungated vanilla RNN.

In an exam, state the mismatch in one sentence, label each architecture clearly and then give its correct equations. Do not memorize a fictitious reset gate for LSTM.