# F22 Q7a

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-s15

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s15/

## Question

**Question (10 marks).** Explain with an example how LSTM handles long sequences in language modeling.

## Answer — step by step

An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.



$$
f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i)
$$





$$
\widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c),\quad C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t
$$





$$
o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\quad h_t=o_t\odot\tanh(C_t)
$$



![The cell path retains selected old content and adds selected new content](https://deep-learning-study-room.vercel.app/assets/figures/lstm.svg)

Diagram: The cell path retains selected old content and adds selected new content. Labels: C previous | × | + | C new | tanh | × | h | new | σ: f | σ: i | tanh: C~ | σ: o | × | [h previous, x current] | Each gate has its own learned weights and bias.

The cell path retains selected old content and adds selected new content Tap the diagram to enlarge.

The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.

In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.

Example: “The student, after several difficult assignments and a long discussion, was relieved.” Useful singular-subject information can be retained in selected cell components while intervening words update other components. At the final step a vocabulary softmax predicts the next token using h. Gates close to 1 on the retained path preserve context; numerical gate values require learned weights and cannot be inferred uniquely from the sentence.