# Vanilla RNN

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/37

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/37/

A recurrent state uses two sources: the current input and the previous hidden state. Transform each with its own weight, add the contributions and bias, and apply tanh. The activated state—not its raw sum—is carried into the next time step.



$$
h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h)
$$





$$
o_t=W_{hy}h_t+b_y
$$



## The vanilla RNN equations from scratch

**Definition.** A vanilla recurrent neural network updates a hidden state from the current sequence input and the previous hidden state, using the same trainable parameters at every time step. An output transformation converts that state into the task's prediction.

### Hidden-state and output equations

For a tanh recurrent layer:



$$
a_t=W_{xh}x_t+W_{hh}h_{t-1}+b_h
$$





$$
h_t=\tanh(a_t)
$$



aₜ is the preactivation and hₜ is the activated hidden state. Wₓₕ transforms the new input; Wₕₕ transforms the previous state. Both contributions have the same hidden dimension before they are added. bₕ has one component per hidden unit.

For classification at time t, an output layer can produce logits and probabilities:



$$
z_t=W_{hy}h_t+b_y,\qquad p_t=\operatorname{softmax}(z_t)
$$



A regression output may instead be linear. A question asking only for hidden states does not require an output-layer prediction, and such a prediction cannot be calculated without the output transformation.

![The current input and previous activated state enter different weight transformations. Their sum is activated to obtain the next state.](https://deep-learning-study-room.vercel.app/assets/figures/v_rnn_state.svg)

Diagram: The current input and previous activated state enter different weight transformations. Their sum is activated to obtain the next state.. Labels: Combine the current input and the previous state | Combine the current input and the previous state | The new state depends on both paths. | Previous state | h at time t − 1 | Current input | x at time t | Combine | Weighted sum + | bias | Activate | New state h at t | The same recurrent weight matrices are used again at the next time step.

The current input and previous activated state enter different weight transformations. Their sum is activated to obtain the next state. Tap the diagram to enlarge.

### Complete numerical: two recurrent time steps

**Question.** Let h₀ = 0, Wₓₕ = 0.5, Wₕₕ = 0.2, bₕ = 0, x₁ = 1 and x₂ = 2. Calculate h₁ and h₂ using tanh activation.

### Given

h₀ = 0; Wxh = 0.5; Whh = 0.2; b = 0.

x₁ = 1; x₂ = 2; activation = tanh.

### To find

Find h₁ and h₂ in order.

### Formula — which one and why

**Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.

x\_t = current input; h\_(t−1) = previous hidden state; W\_xh = input weights; W\_hh = recurrent weights; b = bias.



$$
a_t=W_{xh}x_t+W_{hh}h_{t-1}+b
$$





$$
h_t=\tanh(a_t)
$$



**Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation.

### Solution — step by step

**Step 1: Time step 1 — calculate the preactivation.**



$$
a_1=0.5(1)+0.2(0)+0=0.5
$$



**Apply the recurrent activation.**



$$
h_1=\tanh(0.5)\approx0.462117157
$$



This activated value, not a₁ = 0.5, is passed to the next time step.

**Step 2: Time step 2 — combine the new input and previous state.**



$$
a_2=0.5(2)+0.2h_1+0
$$





$$
a_2\approx1+0.2(0.462117157)\approx1.092423431
$$



**Apply tanh to the complete new preactivation.**



$$
h_2=\tanh(a_2)\approx0.797760954
$$



| Time step | Current input | Previous state | Preactivation | New state |
| --- | --- | --- | --- | --- |
| 1 | 1 | 0 | 0.5 | 0.462117157 |
| 2 | 2 | 0.462117157 | 1.092423431 | 0.797760954 |

**Final answer:** h₁ ≈ 0.462117 and h₂ ≈ 0.797761. Keep the full calculator value of h₁ while computing the second step. Although a₂ is greater than 1, h₂ is between −1 and 1 because tanh is applied after the affine sum. Both time steps use the same weights; this calculation contains no parameter update.

## A complete vector RNN calculation

**Question from R26 Q5a.** xₜ = [0,1], hₜ₋₁ = [0,1], no bias is supplied, Wxh is shown below, Whh is shown below. Use tanh and rows as receiving neurons.

### Given

Input x = [0,1]ᵀ; previous h = [0,1]ᵀ.

Wxh =  $\begin{bmatrix}0.6 & 0.4\\0.3 & 0.7\end{bmatrix}$ .

Whh =  $\begin{bmatrix}0.5 & 0.3\\0.2 & 0.4\end{bmatrix}$ ; bias = 0.

Apply tanh separately to the two sums.

### To find

Find the new two-component hidden state; output-layer parameters are not supplied.

### Formula — which one and why

**Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.

x\_t = current input; h\_(t−1) = previous hidden state; W\_xh = input weights; W\_hh = recurrent weights; b = bias.



$$
a_t=W_{xh}x_t+W_{hh}h_{t-1}+b
$$





$$
h_t=\tanh(a_t)
$$



**Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation.

### Solution — step by step

![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_deac39f14c.svg)

Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | Wxh | 0.6 | 0.4 | 0.3 | 0.7 | Whh | 0.5 | 0.3 | 0.2 | 0.4

Given matrices for this question Tap the diagram to enlarge.

**Step 1 new input:**  Wxh x = [0.6 × 0 + 0.4 × 1, 0.3 × 0 + 0.7 × 1] = [0.4,0.7].

**Step 2 old memory:**  Whh hprevious = [0.5 × 0 + 0.3 × 1, 0.2 × 0 + 0.4 × 1] = [0.3,0.4].

**Step 3 add:**  a = [0.4+0.3, 0.7+0.4] = [0.7,1.1].

**Step 4 activate each element:**  h = [tanh(0.7), tanh(1.1)] = **[0.604368,0.800499]**. An output-layer prediction cannot be numerically calculated without its weights and bias.

## Drill 41 Two RNN steps

## Question

**Question.** h₀ = 0, Wxh = 0.5, Whh = 0.2, b = 0, x₁ = 1, x₂ = 2, tanh activation. Find h₁, h₂.

## Answer — step by step

### Given

Initial h₀ = 0; input weight Wxh = 0.5; recurrent weight Whh = 0.2; b = 0.

Inputs: x₁ = 1, x₂ = 2; activation: tanh.

### To find

Calculate h₁, then use it to calculate h₂.

### Formula — which one and why

**Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.

x\_t = current input; h\_(t−1) = previous hidden state; W\_xh = input weights; W\_hh = recurrent weights; b = bias.



$$
a_t=W_{xh}x_t+W_{hh}h_{t-1}+b
$$





$$
h_t=\tanh(a_t)
$$



**Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation.

### Solution — step by step

### Meaning and formulas

At time t, the RNN combines the current input xₜ and the previous hidden state hₜ₋₁. The same two weights are reused at both times.



$$
a_t=W_{xh}x_t+W_{hh}h_{t-1}+b,\qquad h_t=\tanh(a_t)
$$



The raw sum a is not the hidden state. Tanh converts that sum into the state that is carried forward.



$$
\tanh(a)=\frac{e^{2a}-1}{e^{2a}+1}
$$



### Step 1: calculate the first raw sum



$$
a_1=0.5(1)+0.2(0)+0=0.5
$$



### Step 2: activate it to obtain the first state



$$
h_1=\tanh(0.5)=\frac{e^1-1}{e^1+1}
$$





$$
h_1=\frac{2.718281828-1}{2.718281828+1}=0.462117157
$$



### Step 3: use that state in the second raw sum



$$
a_2=0.5(2)+0.2(0.462117157)+0
$$





$$
a_2=1+0.092423431=1.092423431
$$



### Step 4: activate the second sum



$$
h_2=\tanh(1.092423431)=0.797760954
$$



**Final answer:** h₁ ≈ 0.462117157; h₂ ≈ 0.797760954. Pass h₁ to the next step, not the unactivated value 0.5.