R24 Q7a

Answer — step by step

An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.

LaTeX: f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i)

LaTeX: \widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c),\quad C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t

LaTeX: o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\quad h_t=o_t\odot\tanh(C_t)

Diagram: The cell path retains selected old content and adds selected new content. Labels: C previous | × | + | C new | tanh | × | h | new | σ: f | σ: i | tanh: C~ | σ: o | × | [h previous, x current] | Each gate has its own learned weights and bias.

The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.

In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.