Deep Learning · TEE · section

Model answer: LSTM and GRU memory

Recurrent neural networks

Question

Question. Compare a vanilla RNN, an LSTM and a GRU. Explain the roles of the LSTM gates and state the GRU convention used in this course.

Answer — step by step

A vanilla RNN repeatedly transforms the current input and previous hidden state. LSTMs and GRUs add learned gates that control the retention and replacement of information. Gates usually use sigmoid, so their entries lie between 0 and 1; gate multiplication controls how much of a corresponding value is passed.

LSTM carries a cell and hidden state; GRU carries a hidden state with two gate roles. Tap the diagram to enlarge.

LSTM

An LSTM carries a cell state c and a hidden state h. Its forget gate f controls the old cell contribution. Its input gate i controls admission of a new candidate c-tilde. Its output gate o controls the exposed hidden state.

c_t=f_t\odot c_{t-1}+i_t\odot\widetilde c_tLaTeX: c_t=f_t\odot c_{t-1}+i_t\odot\widetilde c_t
h_t=o_t\odot\tanh(c_t)LaTeX: h_t=o_t\odot\tanh(c_t)

The candidate is commonly formed using tanh, so it can be negative. Gate multiplication is elementwise. The updated cell is not the same as the hidden output; the second equation is another required step.

GRU

A GRU carries one hidden state rather than a separate LSTM-style cell state. A reset gate r controls how previous hidden information contributes to candidate formation. An update gate z mixes the old state and candidate. This guide uses:

h_t=(1-z_t)\odot h_{t-1}+z_t\odot\widetilde h_tLaTeX: h_t=(1-z_t)\odot h_{t-1}+z_t\odot\widetilde h_t

Thus z near 1 favors the new candidate under this convention. Some sources exchange z and 1 − z; always follow the equation printed in the question.

ModelCarried stateMain gatesTypical parameter implication for the same dimensions
Vanilla RNNHidden stateNone of these memory gatesFewest affine transformations
LSTMCell and hidden statesForget, input and output gatesFour affine transformations including candidate
GRUHidden stateReset and update gatesThree affine transformations including candidate

Gated models improve control over information flow but do not guarantee perfect indefinite memory. Their benefit must be evaluated for the sequence task.