Deep Learning · TEE · section

R25 Q7a · Long-text sentiment architecture

February 2025 re-exam · complete solved paper

Question

Question. Choose a deep architecture for long social-media/customer-review sequences classified positive,negative or neutral. Explain its operation with equations. (10 marks)

Answer — step by step

An LSTM is a suitable syllabus-aligned choice for long dependencies. Tokenize and embed the text, process the sequence through LSTM states, form a sequence representation and use a three-class softmax. Padding/masking must prevent padded tokens from being treated as content.

Text tokens → Embedding sequence → LSTM hidden/cell recurrence → Final or pooled representation → Dense3 +softmax → Positive / negative / neutral

Definition. Long short-term memory (LSTM) is a gated recurrent architecture with a cell state and hidden state. Forget, input and output gates regulate state retention, incorporation of candidate information and exposure of the updated cell.

Explanation and working

Long short-term memory, or LSTM, keeps a cell state Cₜ and a hidden state hₜ. A sigmoid gate produces values from 0 to 1: approximately 0 blocks a component, approximately 1 passes it. Intermediate values pass a fraction. Gates are learned from the previous hidden state and current input, not manually assigned by the user during normal inference.

Follow the cell state from left to right then identify forget input and output gates Tap the diagram to enlarge.
f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i)LaTeX: f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i)
\widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c)LaTeX: \widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c)
C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_tLaTeX: C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t
o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\qquad h_t=o_t\odot\tanh(C_t)LaTeX: o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\qquad h_t=o_t\odot\tanh(C_t)

The brackets [h, x] concatenate two vectors: [a, b] joined with [c, d] gives [a, b, c, d]. Concatenation is not addition. f is the forget gate: how much old cell content remains. i is the input gate: how much new candidate enters. The candidate is proposed content. o is the output gate: how much of the activated cell becomes visible as the hidden state.

p=\operatorname{softmax}(Wh+b),\qquad L=-\sum_{k=1}^3y_k\log p_kLaTeX: p=\operatorname{softmax}(Wh+b),\qquad L=-\sum_{k=1}^3y_k\log p_k

Train on labelled examples with cross-entropy and BPTT; evaluate on held-out reviews with class-wise precision/recall/F1. LSTM gates reduce long-path gradient decay but do not guarantee perfect memory. The three outputs are mutually exclusive classes for this question.