Deep Learning · TEE · section

Recurrent networks: the complete relationship

Recurrent neural networks

A vanilla RNN repeatedly combines the new input and previous hidden state with shared weights. Backpropagation through time accumulates contributions from every use of those shared weights. LSTM adds an explicit cell state with forget, input and output gates. GRU mixes an old hidden state and a candidate using reset and update gates.

Diagram: Read from the ordinary recurrent calculation to the two gated memory designs.. Labels: What each recurrent cell carries forward | What each recurrent cell carries forward | Compare three different cell designs. These are alternatives, not consecutive | layers. | Vanilla RNN | One hidden state h | No memory gates | LSTM | Cell C and hidden h | Forget / input / | output gates | GRU | One hidden state h | Reset / update gates | At equal dimensions, a standard LSTM uses four affine transforms; a basic GRU | uses three.

For every recurrent calculation, compute the gate or hidden preactivation first, apply its stated activation, then combine the resulting values. Elementwise gate multiplication is different from matrix multiplication. The stated GRU update convention determines whether z retains the old state or selects the candidate.