Deep Learning · TEE · section

R25 Q5b · ReLU RNN state with declared orientation

February 2025 re-exam · complete solved paper

Question

Question. Sketch a one-hidden-layer RNN and state its hidden-state formula. Compute hₜ for Wxh=[[0.2,0.4],[0.1,0.3]], Whh=[[0.5,0.6],[0.3,0.4]], b=[0.1,0.2], x=[1,0], previoush=[0,1], ReLU. (10 marks)

Answer — step by step

Given

Input-to-hidden weight matrix Wₓₕ: transforms the two current input values into contributions to the two hidden units.

W_{xh}=\begin{bmatrix}0.2&0.4\\0.1&0.3\end{bmatrix}LaTeX: W_{xh}=\begin{bmatrix}0.2&0.4\\0.1&0.3\end{bmatrix}

Hidden-to-hidden weight matrix Wₕₕ: transforms the two previous hidden activations into recurrent contributions.

W_{hh}=\begin{bmatrix}0.5&0.6\\0.3&0.4\end{bmatrix}LaTeX: W_{hh}=\begin{bmatrix}0.5&0.6\\0.3&0.4\end{bmatrix}
QuantityGiven valueMeaning
xₜ[1, 0]Current input row: two input values
hₜ₋₁[0, 1]Previous hidden-state row: two stored activations
b[0.1, 0.2]One bias for each hidden unit
ActivationReLU(z) = max(0, z)Apply separately to each hidden weighted sum

To find

Sketch the one-hidden-layer RNN, state its recurrence, and calculate the two components of hₜ.

Formula: which one and why

One-hidden-layer recurrent neural network with a one-step recurrent connection Input x at time t passes through input-to-hidden weights W x h. The hidden layer adds that input contribution, the previous hidden state through recurrent weights W h h, and bias b, then applies ReLU to produce two hidden activations h at time t. The recurrent arrow passes through a one-step delay: the current hidden state becomes the previous hidden state for the next input. An optional output head maps the hidden state through W h y, output bias and an output activation to prediction y hat. Output-head parameters are illustrative and are not supplied or needed for this question. Input xₜ[1, 0]two input valuesWₓₕ+ bias bOne hidden layerzₜ = input + history + biashₜ = ReLU(zₜ)two hidden activationsWₕᵧOutput headŷₜillustrative onlyWₕₕ One-step delayprevious state hₜ₋₁+ output biasthen output activationThe same hidden layer and recurrent weights are reused at each time step.
Sketch to draw in the exam: input → one hidden layer → output, with a recurrent arrow through a one-step delay back to the hidden layer. Wₓₕ connects input to hidden; Wₕₕ carries the previous hidden state. Wₕᵧ denotes a possible hidden-to-output mapping. The paper asks for hₜ only, so no output weights or output prediction are required.

The printed vectors are horizontal and the weight-index convention is not specified. Adopt row vectors multiplied on the right:

h_t=\operatorname{ReLU}(x_tW_{xh}+h_{t-1}W_{hh}+b)LaTeX: h_t=\operatorname{ReLU}(x_tW_{xh}+h_{t-1}W_{hh}+b)

Step 1. Multiply the input row

[1,0]Wxh=[1×0.2+0×0.1,1×0.4+0×0.3]=[0.2,0.4].

Step 2. Multiply the previous-state row

[0,1]Whh=[0×0.5+1×0.3,0×0.6+1×0.4]=[0.3,0.4].

Step 3. Add componentwise including bias

[0.2+0.3+0.1,0.4+0.4+0.2]=[0.6,1.0].

Step 4. Apply ReLU separately

[max(0,0.6),max(0,1.0)]=[0.6,1.0].

Final answer

Row-vector convention: hₜ=[0.6,1.0]. If the supplied matrices instead map column inputs by Wxh x + Whh h +b, the untransposed arrays give[0.9,0.7]. That is a different orientation; explicitly state the formula. Transposing the matrices when switching convention preserves the same network.

Source / convention note. Square matrices and printed horizontal vectors leave orientation ambiguous; both possible readings are explained, main answer follows row-vector form.