# Model answer: LSTM and GRU memory

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s16

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s16/

## Question

**Question.** Compare a vanilla RNN, an LSTM and a GRU. Explain the roles of the LSTM gates and state the GRU convention used in this course.

## Answer — step by step

A vanilla RNN repeatedly transforms the current input and previous hidden state. LSTMs and GRUs add learned gates that control the retention and replacement of information. Gates usually use sigmoid, so their entries lie between 0 and 1; gate multiplication controls how much of a corresponding value is passed.

![LSTM carries a cell and hidden state; GRU carries a hidden state with two gate roles.](https://deep-learning-study-room.vercel.app/assets/figures/v_rnn_comparison.svg)

Diagram: LSTM carries a cell and hidden state; GRU carries a hidden state with two gate roles.. Labels: What each recurrent cell carries forward | What each recurrent cell carries forward | Compare three different cell designs. These are alternatives, not consecutive | layers. | Vanilla RNN | One hidden state h | No memory gates | LSTM | Cell C and hidden h | Forget / input / | output gates | GRU | One hidden state h | Reset / update gates | At equal dimensions, a standard LSTM uses four affine transforms; a basic GRU | uses three.

LSTM carries a cell and hidden state; GRU carries a hidden state with two gate roles. Tap the diagram to enlarge.

### LSTM

An LSTM carries a cell state c and a hidden state h. Its forget gate f controls the old cell contribution. Its input gate i controls admission of a new candidate c-tilde. Its output gate o controls the exposed hidden state.



$$
c_t=f_t\odot c_{t-1}+i_t\odot\widetilde c_t
$$





$$
h_t=o_t\odot\tanh(c_t)
$$



The candidate is commonly formed using tanh, so it can be negative. Gate multiplication is elementwise. The updated cell is not the same as the hidden output; the second equation is another required step.

### GRU

A GRU carries one hidden state rather than a separate LSTM-style cell state. A reset gate r controls how previous hidden information contributes to candidate formation. An update gate z mixes the old state and candidate. This guide uses:



$$
h_t=(1-z_t)\odot h_{t-1}+z_t\odot\widetilde h_t
$$



Thus z near 1 favors the new candidate under this convention. Some sources exchange z and 1 − z; always follow the equation printed in the question.

| Model | Carried state | Main gates | Typical parameter implication for the same dimensions |
| --- | --- | --- | --- |
| Vanilla RNN | Hidden state | None of these memory gates | Fewest affine transformations |
| LSTM | Cell and hidden states | Forget, input and output gates | Four affine transformations including candidate |
| GRU | Hidden state | Reset and update gates | Three affine transformations including candidate |

Gated models improve control over information flow but do not guarantee perfect indefinite memory. Their benefit must be evaluated for the sequence task.