# F23 Q4b

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/13/tee-c13-s9

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s9/

## Question

**Question (10 marks).** Design an LSTM autocomplete model for “Maharashtra is a state in the western part of India. The capital of Maharashtra is \_\_\_.” Draw the model and write conditional probabilities at each time step.

## Answer — step by step

An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.



$$
f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i)
$$





$$
\widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c),\quad C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t
$$





$$
o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\quad h_t=o_t\odot\tanh(C_t)
$$



![The cell path retains selected old content and adds selected new content](https://deep-learning-study-room.vercel.app/assets/figures/lstm.svg)

Diagram: The cell path retains selected old content and adds selected new content. Labels: C previous | × | + | C new | tanh | × | h | new | σ: f | σ: i | tanh: C~ | σ: o | × | [h previous, x current] | Each gate has its own learned weights and bias.

The cell path retains selected old content and adds selected new content Tap the diagram to enlarge.

The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.

In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.

Architecture: token → embedding → LSTM state → vocabulary softmax → next-token distribution. During training use the actual prefix and its next-word labels with cross-entropy. At inference, feed the available prefix and choose/sample the next token. “Mumbai” is the intended factual completion in this example; a model only predicts it if learned probabilities support it.

**One explicit tokenization ignoring punctuation:**

1. P(Maharashtra | START).

2. P(is | Maharashtra).

3. P(a | Maharashtra is).

4. P(state | Maharashtra is a).

5. P(in | Maharashtra is a state).

6. P(the | Maharashtra is a state in).

7. P(western | Maharashtra is a state in the).

8. P(part | Maharashtra is a state in the western).

9. P(of | Maharashtra is a state in the western part).

10. P(India | Maharashtra is a state in the western part of).

11. P(The | Maharashtra is a state in the western part of India).

12. P(capital | Maharashtra is a state in the western part of India The).

13. P(of | Maharashtra is a state in the western part of India The capital).

14. P(Maharashtra | Maharashtra is a state in the western part of India The capital of).

15. P(is | Maharashtra is a state in the western part of India The capital of Maharashtra).

16. P(Mumbai | Maharashtra is a state in the western part of India The capital of Maharashtra is).

The joint sequence probability is the product of these conditionals. Numerical probability values cannot be computed from the sentence alone because weights are not supplied.

![Replace the repeated recurrent cell with an LSTM and attach a vocabulary softmax at each step](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg)

Diagram: Replace the repeated recurrent cell with an LSTM and attach a vocabulary softmax at each step. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ←

Replace the repeated recurrent cell with an LSTM and attach a vocabulary softmax at each step Tap the diagram to enlarge.