R23 Q7a

Answer — step by step

An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.

LaTeX: f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i)

LaTeX: \widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c),\quad C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t

LaTeX: o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\quad h_t=o_t\odot\tanh(C_t)

Diagram: The cell path retains selected old content and adds selected new content. Labels: C previous | × | + | C new | tanh | × | h | new | σ: f | σ: i | tanh: C~ | σ: o | × | [h previous, x current] | Each gate has its own learned weights and bias.

The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.

In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.

The model can retain the earlier entity XYZ and the soldier context while updating intervening events. At the final prefix a vocabulary softmax predicts a suitable next token; “soldier” is a plausible completion. The exact output is not uniquely determined by the text or architecture alone. The forget gate retains useful entity content, the input gate adds the war/honor context, and the output gate exposes relevant state for prediction. Do not invent numerical gate values when no weights are provided.