Deep Learning · TEE · section
Comparing the three recurrent models
Recurrent neural networks
| Feature | Vanilla RNN | LSTM | GRU |
|---|---|---|---|
| Main state | h | h and C | h |
| Gates | None in vanilla cell | Forget input output | Reset update |
| Long dependencies | Often difficult to learn | Designed to help retain useful long context | Designed to help retain useful long context |
| Parameters at equal dimensions | Usually fewest | Usually most of these three | Usually between vanilla RNN and LSTM |
| Typical choice | Simple short sequences | Complex long-context sequences | Efficient gated sequence baseline |
With input size d and hidden size H, a basic single-bias vanilla recurrent layer has H(d+H+1) parameters; a standard LSTM has 4H(d+H+1); a basic GRU has 3H(d+H+1). These omit output layers; library versions with extra bias vectors differ. The factor four in LSTM counts three gates plus one candidate transform.