# The vanilla RNN equations from scratch

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s4

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s4/

**Definition.** A vanilla recurrent neural network updates a hidden state from the current sequence input and the previous hidden state, using the same trainable parameters at every time step. An output transformation converts that state into the task's prediction.

### Hidden-state and output equations

For a tanh recurrent layer:



$$
a_t=W_{xh}x_t+W_{hh}h_{t-1}+b_h
$$





$$
h_t=\tanh(a_t)
$$



aₜ is the preactivation and hₜ is the activated hidden state. Wₓₕ transforms the new input; Wₕₕ transforms the previous state. Both contributions have the same hidden dimension before they are added. bₕ has one component per hidden unit.

For classification at time t, an output layer can produce logits and probabilities:



$$
z_t=W_{hy}h_t+b_y,\qquad p_t=\operatorname{softmax}(z_t)
$$



A regression output may instead be linear. A question asking only for hidden states does not require an output-layer prediction, and such a prediction cannot be calculated without the output transformation.

![The current input and previous activated state enter different weight transformations. Their sum is activated to obtain the next state.](https://deep-learning-study-room.vercel.app/assets/figures/v_rnn_state.svg)

Diagram: The current input and previous activated state enter different weight transformations. Their sum is activated to obtain the next state.. Labels: Combine the current input and the previous state | Combine the current input and the previous state | The new state depends on both paths. | Previous state | h at time t − 1 | Current input | x at time t | Combine | Weighted sum + | bias | Activate | New state h at t | The same recurrent weight matrices are used again at the next time step.

The current input and previous activated state enter different weight transformations. Their sum is activated to obtain the next state. Tap the diagram to enlarge.

### Complete numerical: two recurrent time steps

**Question.** Let h₀ = 0, Wₓₕ = 0.5, Wₕₕ = 0.2, bₕ = 0, x₁ = 1 and x₂ = 2. Calculate h₁ and h₂ using tanh activation.

### Given

h₀ = 0; Wxh = 0.5; Whh = 0.2; b = 0.

x₁ = 1; x₂ = 2; activation = tanh.

### To find

Find h₁ and h₂ in order.

### Formula — which one and why

**Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.

x\_t = current input; h\_(t−1) = previous hidden state; W\_xh = input weights; W\_hh = recurrent weights; b = bias.



$$
a_t=W_{xh}x_t+W_{hh}h_{t-1}+b
$$





$$
h_t=\tanh(a_t)
$$



**Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation.

### Solution — step by step

**Step 1: Time step 1 — calculate the preactivation.**



$$
a_1=0.5(1)+0.2(0)+0=0.5
$$



**Apply the recurrent activation.**



$$
h_1=\tanh(0.5)\approx0.462117157
$$



This activated value, not a₁ = 0.5, is passed to the next time step.

**Step 2: Time step 2 — combine the new input and previous state.**



$$
a_2=0.5(2)+0.2h_1+0
$$





$$
a_2\approx1+0.2(0.462117157)\approx1.092423431
$$



**Apply tanh to the complete new preactivation.**



$$
h_2=\tanh(a_2)\approx0.797760954
$$



| Time step | Current input | Previous state | Preactivation | New state |
| --- | --- | --- | --- | --- |
| 1 | 1 | 0 | 0.5 | 0.462117157 |
| 2 | 2 | 0.462117157 | 1.092423431 | 0.797760954 |

**Final answer:** h₁ ≈ 0.462117 and h₂ ≈ 0.797761. Keep the full calculator value of h₁ while computing the second step. Although a₂ is greater than 1, h₂ is between −1 and 1 because tanh is applied after the affine sum. Both time steps use the same weights; this calculation contains no parameter update.