Deep Learning · TEE · section

Reading an LSTM equation one operation at a time

Recurrent neural networks

1. Two memories on the page do not mean the same value

The cell state c carries the recurrent cell memory. The hidden state h is an exposed output computed from that memory. An LSTM passes both to the next step. A question may give the previous cell and hidden states separately; do not substitute one for the other.

Diagram: Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output.. Labels: Calculate cell memory before the hidden output | Calculate cell memory before the hidden output | Example: old cell 0.8, forget gate 0.25, input gate 0.5, candidate −0.4. | Retain old content | 0.25 × 0.8 = 0.2 | Admit new content | 0.5 × (−0.4) = −0.2 | New cell | 0.2 + (−0.2) = 0 | Hidden output: 0.7 × tanh(0) = 0 | The output gate is 0.7 here. Apply it to tanh of the NEW cell, after the cell | update.

2. Gate values are multipliers

For a previous cell value of 0.8, a forget gate of 0.25 keeps 0.25 × 0.8 = 0.2. A gate of 0 keeps none of that contribution; a gate of 1 keeps the whole contribution. Intermediate sigmoid values give partial retention.

An input gate does the same kind of multiplication to the new candidate. If the candidate is −0.4 and the input gate is 0.5, the admitted new contribution is −0.2. The negative sign belongs to the candidate; sigmoid gates themselves are positive.

3. Add old and new contributions

With those values, the new cell is 0.2 + (−0.2) = 0. A retained positive contribution and an admitted negative contribution can cancel.

LaTeX: c_t=f_tc_{t-1}+i_t\widetilde c_t

This example also explains why “input gate” does not mean simply multiply the raw input x by i. It multiplies the candidate calculated for the cell update.

4. Calculate the hidden state afterward

Apply tanh to the updated cell and multiply by the output gate. If o = 0.7 and c = 0, tanh(0) = 0 and h = 0.7 × 0 = 0.

LaTeX: h_t=o_t\tanh(c_t)

If c were 1 instead, tanh(1) ≈ 0.761594 and h ≈ 0.533116 for the same output gate. The output gate controls exposure of the transformed cell, not the earlier addition that produced c.

5. Supplied gates and calculated gates are different numerical types

When f, i and o are supplied, begin directly with their multiplications. When weights and inputs are supplied, first calculate each gate's affine input, apply sigmoid and retain those gate values. Calculate the candidate using tanh. Only then update c and h. The separate solved examples below cover both types and multi-step repetition.