Deep Learning · MT2 · section

Model answer: why recurrent networks use a hidden state

Recurrent neural networks

Question

Question. What is a recurrent neural network? Explain how a hidden state allows it to process a sequence and distinguish it from a feedforward network.

Answer — step by step

An RNN processes ordered inputs using a hidden state that is carried from one time step to the next. The current state combines information from the new input with information retained from earlier inputs. The same recurrent weights are reused across time steps.

The previous hidden state brings earlier information into the current calculation alongside the new input. Tap the diagram to enlarge.

For a simple tanh RNN:

$h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h)$ LaTeX: h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h)

Symbol	Meaning
xₜ	Input at the current time step
hₜ₋₁	Hidden state passed from the previous step
Wxh	Weights connecting the current input to the hidden state
Whh	Weights connecting the previous state to the current state
bₕ	Hidden-state bias
hₜ	New state after the activation

For words in a sentence, earlier words affect the state used when the next word arrives. Changing the order can therefore change the final representation. The state is a learned summary; it is not a guaranteed exact copy of every earlier input.

A feedforward network maps inputs through its layers without this recurrent state transition. An RNN can accept a sequence, reuse its parameters at each step and produce either an output at each step or an output from a final state, depending on the task. Examples include sentiment classification, next-word prediction and time-series modeling.

Unrolling an RNN draws its repeated use at t = 1, 2, 3 and so on. Those boxes share parameters; drawing three time steps does not create three independently learned copies of Whh.