# Comparing the three recurrent models

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s15

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s15/

| Feature | Vanilla RNN | LSTM | GRU |
| --- | --- | --- | --- |
| Main state | h | h and C | h |
| Gates | None in vanilla cell | Forget input output | Reset update |
| Long dependencies | Often difficult to learn | Designed to help retain useful long context | Designed to help retain useful long context |
| Parameters at equal dimensions | Usually fewest | Usually most of these three | Usually between vanilla RNN and LSTM |
| Typical choice | Simple short sequences | Complex long-context sequences | Efficient gated sequence baseline |

With input size d and hidden size H, a basic single-bias vanilla recurrent layer has H(d+H+1) parameters; a standard LSTM has 4H(d+H+1); a basic GRU has 3H(d+H+1). These omit output layers; library versions with extra bias vectors differ. The factor four in LSTM counts three gates plus one candidate transform.

![Compare what is carried through time and how many gate or candidate transforms are used.](https://deep-learning-study-room.vercel.app/assets/figures/v_rnn_comparison.svg)

Diagram: Compare what is carried through time and how many gate or candidate transforms are used.. Labels: What each recurrent cell carries forward | What each recurrent cell carries forward | Compare three different cell designs. These are alternatives, not consecutive | layers. | Vanilla RNN | One hidden state h | No memory gates | LSTM | Cell C and hidden h | Forget / input / | output gates | GRU | One hidden state h | Reset / update gates | At equal dimensions, a standard LSTM uses four affine transforms; a basic GRU | uses three.

Compare what is carried through time and how many gate or candidate transforms are used. Tap the diagram to enlarge.