Deep Learning · TEE · section
F23 Q4b
Final examination December 2023 fully solved
Question
Question (10 marks). Design an LSTM autocomplete model for “Maharashtra is a state in the western part of India. The capital of Maharashtra is ___.” Draw the model and write conditional probabilities at each time step.
Answer — step by step
An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.
![f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i)](https://deep-learning-study-room.vercel.app/assets/equations/0c991bf8824254bc.svg)
LaTeX: f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i)![\widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c),\quad C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t](https://deep-learning-study-room.vercel.app/assets/equations/31dbb5df72a8ba34.svg)
LaTeX: \widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c),\quad C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t![o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\quad h_t=o_t\odot\tanh(C_t)](https://deep-learning-study-room.vercel.app/assets/equations/932f49e3f8a32474.svg)
LaTeX: o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\quad h_t=o_t\odot\tanh(C_t)The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.
In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.
Architecture: token → embedding → LSTM state → vocabulary softmax → next-token distribution. During training use the actual prefix and its next-word labels with cross-entropy. At inference, feed the available prefix and choose/sample the next token. “Mumbai” is the intended factual completion in this example; a model only predicts it if learned probabilities support it.
One explicit tokenization ignoring punctuation:
1. P(Maharashtra | START).
2. P(is | Maharashtra).
3. P(a | Maharashtra is).
4. P(state | Maharashtra is a).
5. P(in | Maharashtra is a state).
6. P(the | Maharashtra is a state in).
7. P(western | Maharashtra is a state in the).
8. P(part | Maharashtra is a state in the western).
9. P(of | Maharashtra is a state in the western part).
10. P(India | Maharashtra is a state in the western part of).
11. P(The | Maharashtra is a state in the western part of India).
12. P(capital | Maharashtra is a state in the western part of India The).
13. P(of | Maharashtra is a state in the western part of India The capital).
14. P(Maharashtra | Maharashtra is a state in the western part of India The capital of).
15. P(is | Maharashtra is a state in the western part of India The capital of Maharashtra).
16. P(Mumbai | Maharashtra is a state in the western part of India The capital of Maharashtra is).
The joint sequence probability is the product of these conditionals. Numerical probability values cannot be computed from the sentence alone because weights are not supplied.