Deep Learning · TEE · section

GRU two gates and one state

Recurrent neural networks

Definition. A gated recurrent unit (GRU) is a recurrent architecture that uses reset and update gates to control its hidden-state transition. Unlike a standard LSTM, it does not carry a separate cell state alongside the hidden state.

Gate equations and the convention used here

The reset gate r controls how the previous hidden state enters candidate formation. The update gate z controls the mixture of the old state and the candidate. Both are calculated from the current input and previous hidden state:

$r_t=\sigma(W_rx_t+U_rh_{t-1}+b_r)$ LaTeX: r_t=\sigma(W_rx_t+U_rh_{t-1}+b_r)

$z_t=\sigma(W_zx_t+U_zh_{t-1}+b_z)$ LaTeX: z_t=\sigma(W_zx_t+U_zh_{t-1}+b_z)

The candidate equation used in these notes applies reset before the recurrent candidate transformation:

$\widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h)$ LaTeX: \widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h)

These notes define z as the fraction of new candidate admitted:

Thus z near 0 favors retention of the old hidden state, while z near 1 favors the candidate. The reset gate affects the candidate calculation; it does not directly multiply the retained old-state term in the final mixture.

First calculate the reset-dependent candidate. Then combine the old hidden state and the candidate using the stated update-gate convention. Tap the diagram to enlarge.

Complete numerical with supplied gate values

Question. A scalar GRU has hₜ₋₁ = 0.4, reset gate rₜ = 0.5, update gate zₜ = 0.25, xₜ = 1, Wₕ = 0.2, Uₕ = 0.5 and bₕ = 0. Calculate the candidate and new hidden state using hₜ = (1−zₜ)hₜ₋₁ + zₜh̃ₜ.

Given

Old hidden state = 0.4; r = 0.5; z = 0.25.

x = 1; W = 0.2; U = 0.5; b = 0.

Here z weights the candidate in the final mixture.

To find

Find candidate and new hidden state.

Formula — which one and why

GRU reset and state mixture. The reset gate controls old-state information entering the candidate. The stated update convention then mixes the original old state and the candidate.

r = reset gate; z = update gate; h_old = previous state; h_tilde = candidate. Here z weights the candidate; some questions instead define z to weight the old state.

$\widetilde h=\tanh(Wx+U(r\odot h_{\mathrm{old}})+b)$ LaTeX: \widetilde h=\tanh(Wx+U(r\odot h_{\mathrm{old}})+b)

$h=(1-z)\odot h_{\mathrm{old}}+z\odot\widetilde h$ LaTeX: h=(1-z)\odot h_{\mathrm{old}}+z\odot\widetilde h

Yaad rakhne ka cue: Reset candidate ko control; update gate old-new mixture ko control.

Solution — step by step

Step 1: apply the reset gate to the previous hidden state.

$r_th_{t-1}=0.5(0.4)=0.2$ LaTeX: r_th_{t-1}=0.5(0.4)=0.2

The gates are already supplied as outputs, so no further sigmoid calculation is needed.

Step 2: calculate the candidate preactivation.

Step 3: apply tanh to obtain the candidate.

Step 4: calculate the two mixture contributions.

Step 5: add the contributions.

Final answer: candidate ≈ 0.291313 and new hidden state ≈ 0.372828. The new scalar state lies between the old state 0.4 and the candidate 0.291313, as expected for this convex mixture.

If the question uses the other update-gate convention

Some texts define z as the fraction of the old state retained:

$h_t=z_th_{t-1}+(1-z_t)\widetilde h_t$ LaTeX: h_t=z_th_{t-1}+(1-z_t)\widetilde h_t

Under that explicitly different convention, the same numerical value z = 0.25 gives:

This is not the answer to the worked question above, which states the first convention. To represent an identical mixture while switching conventions, the gate values must be complemented as well. Always identify the final mixture equation before substituting values.

Relationship to LSTM

A basic GRU has three affine transformations: reset, update and candidate. A standard LSTM has four: forget, input, output and candidate. With equal input and hidden dimensions and the same bias convention, GRU therefore has fewer recurrent parameters. This is a parameter-count comparison, not a guarantee of higher accuracy or faster execution for every task.

Retain-old GRU with gate 0

Question. Use h_new = z h_old+(1−z) candidate. Given h_old = 0.4, z = 0, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.