Deep Learning · MT2 · section

What makes a sequence different

Recurrent neural networks

Definition. A recurrent neural network (RNN) is a neural architecture that models sequential data by updating a hidden state from the current input and the previous state, using shared parameters across time steps.

LaTeX: h_t=\phi(W_{xh}x_t+W_{hh}h_{t-1}+b_h)

The state hₜ is a learned representation of information available up to time t. It is not a guaranteed lossless record of the entire sequence. The initial state is supplied, initialized to zero or learned, according to the model or question.

Why order matters

In language, audio and time series, the meaning of an observation can depend on earlier observations and their order. A basic feedforward network treating each input independently has no recurrent state. An RNN allows the output at a time step to depend on previous processing through hₜ₋₁.

Diagram: Unrolling shows successive evaluations of one recurrent cell with shared parameters.. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ←

Parameter sharing

The unrolled cells use the same Wₓₕ, Wₕₕ and bₕ at every time step. A longer input sequence increases the number of evaluations, not the number of independently learned copies of these parameters. Each new hidden state must be calculated before it is used by the next step.

MT2 interpretation

The TEE route contains the state equation, time-step computations, shared parameters, sequence mappings, BPTT, vanishing/exploding gradients, LSTM and GRU. RNN is outside the current focused MT2 path based on the 12 October faculty update.