Deep Learning · MT2 · section

The vanilla RNN equations from scratch

Recurrent neural networks

Definition. A vanilla recurrent neural network updates a hidden state from the current sequence input and the previous hidden state, using the same trainable parameters at every time step. An output transformation converts that state into the task's prediction.

Hidden-state and output equations

For a tanh recurrent layer:

LaTeX: a_t=W_{xh}x_t+W_{hh}h_{t-1}+b_h

LaTeX: h_t=\tanh(a_t)

aₜ is the preactivation and hₜ is the activated hidden state. Wₓₕ transforms the new input; Wₕₕ transforms the previous state. Both contributions have the same hidden dimension before they are added. bₕ has one component per hidden unit.

For classification at time t, an output layer can produce logits and probabilities:

LaTeX: z_t=W_{hy}h_t+b_y,\qquad p_t=\operatorname{softmax}(z_t)

A regression output may instead be linear. A question asking only for hidden states does not require an output-layer prediction, and such a prediction cannot be calculated without the output transformation.

Diagram: The current input and previous activated state enter different weight transformations. Their sum is activated to obtain the next state.. Labels: Combine the current input and the previous state | Combine the current input and the previous state | The new state depends on both paths. | Previous state | h at time t − 1 | Current input | x at time t | Combine | Weighted sum + | bias | Activate | New state h at t | The same recurrent weight matrices are used again at the next time step.

Complete numerical: two recurrent time steps

Question. Let h₀ = 0, Wₓₕ = 0.5, Wₕₕ = 0.2, bₕ = 0, x₁ = 1 and x₂ = 2. Calculate h₁ and h₂ using tanh activation.

Given

h₀ = 0; Wxh = 0.5; Whh = 0.2; b = 0.

x₁ = 1; x₂ = 2; activation = tanh.

To find

Find h₁ and h₂ in order.

Formula — which one and why

Recurrent forward pass. At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.

x_t = current input; h_(t−1) = previous hidden state; W_xh = input weights; W_hh = recurrent weights; b = bias.

LaTeX: a_t=W_{xh}x_t+W_{hh}h_{t-1}+b

LaTeX: h_t=\tanh(a_t)

Yaad rakhne ka cue: Naya input + pichhla activated state; phir activation.

Solution — step by step

Step 1: Time step 1 — calculate the preactivation.

LaTeX: a_1=0.5(1)+0.2(0)+0=0.5

Apply the recurrent activation.

LaTeX: h_1=\tanh(0.5)\approx0.462117157

This activated value, not a₁ = 0.5, is passed to the next time step.

Step 2: Time step 2 — combine the new input and previous state.

LaTeX: a_2=0.5(2)+0.2h_1+0

LaTeX: a_2\approx1+0.2(0.462117157)\approx1.092423431

Apply tanh to the complete new preactivation.

LaTeX: h_2=\tanh(a_2)\approx0.797760954

Final answer: h₁ ≈ 0.462117 and h₂ ≈ 0.797761. Keep the full calculator value of h₁ while computing the second step. Although a₂ is greater than 1, h₂ is between −1 and 1 because tanh is applied after the affine sum. Both time steps use the same weights; this calculation contains no parameter update.