# GRU gates, candidate and update

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/44

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/44/

A GRU has a reset gate and an update gate. Reset changes how earlier state enters the candidate calculation; update controls the final old/new mixture. The numerical below states its convention explicitly: z is the fraction of the new candidate admitted.



$$
z_t=\sigma(W_zx_t+U_zh_{t-1}+b_z)
$$





$$
r_t=\sigma(W_rx_t+U_rh_{t-1}+b_r)
$$



## GRU two gates and one state

**Definition.** A gated recurrent unit (GRU) is a recurrent architecture that uses reset and update gates to control its hidden-state transition. Unlike a standard LSTM, it does not carry a separate cell state alongside the hidden state.

### Gate equations and the convention used here

The reset gate r controls how the previous hidden state enters candidate formation. The update gate z controls the mixture of the old state and the candidate. Both are calculated from the current input and previous hidden state:



$$
r_t=\sigma(W_rx_t+U_rh_{t-1}+b_r)
$$





$$
z_t=\sigma(W_zx_t+U_zh_{t-1}+b_z)
$$



The candidate equation used in these notes applies reset before the recurrent candidate transformation:



$$
\widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h)
$$



These notes define z as the fraction of **new candidate** admitted:



$$
h_t=(1-z_t)\odot h_{t-1}+z_t\odot\widetilde h_t
$$



Thus z near 0 favors retention of the old hidden state, while z near 1 favors the candidate. The reset gate affects the candidate calculation; it does not directly multiply the retained old-state term in the final mixture.

![First calculate the reset-dependent candidate. Then combine the old hidden state and the candidate using the stated update-gate convention.](https://deep-learning-study-room.vercel.app/assets/figures/gru.svg)

Diagram: First calculate the reset-dependent candidate. Then combine the old hidden state and the candidate using the stated update-gate convention.. Labels: h previous | Reset r × h | Candidate = tanh(...) | (1 − z) × h previous | z × candidate | + | New hidden state | Here z means the fraction of NEW content admitted.

First calculate the reset-dependent candidate. Then combine the old hidden state and the candidate using the stated update-gate convention. Tap the diagram to enlarge.

### Complete numerical with supplied gate values

**Question.** A scalar GRU has hₜ₋₁ = 0.4, reset gate rₜ = 0.5, update gate zₜ = 0.25, xₜ = 1, Wₕ = 0.2, Uₕ = 0.5 and bₕ = 0. Calculate the candidate and new hidden state using hₜ = (1−zₜ)hₜ₋₁ + zₜh̃ₜ.

### Given

Old hidden state = 0.4; r = 0.5; z = 0.25.

x = 1; W = 0.2; U = 0.5; b = 0.

Here z weights the candidate in the final mixture.

### To find

Find candidate and new hidden state.

### Formula — which one and why

**GRU reset and state mixture.** The reset gate controls old-state information entering the candidate. The stated update convention then mixes the original old state and the candidate.

r = reset gate; z = update gate; h\_old = previous state; h\_tilde = candidate. Here z weights the candidate; some questions instead define z to weight the old state.



$$
\widetilde h=\tanh(Wx+U(r\odot h_{\mathrm{old}})+b)
$$





$$
h=(1-z)\odot h_{\mathrm{old}}+z\odot\widetilde h
$$



**Yaad rakhne ka cue:** Reset candidate ko control; update gate old-new mixture ko control.

### Solution — step by step

**Step 1: apply the reset gate to the previous hidden state.**



$$
r_th_{t-1}=0.5(0.4)=0.2
$$



The gates are already supplied as outputs, so no further sigmoid calculation is needed.

**Step 2: calculate the candidate preactivation.**



$$
a_{\mathrm{candidate}}=W_hx_t+U_h(r_th_{t-1})+b_h
$$





$$
a_{\mathrm{candidate}}=0.2(1)+0.5(0.2)+0=0.3
$$



**Step 3: apply tanh to obtain the candidate.**



$$
\widetilde h_t=\tanh(0.3)\approx0.291312612
$$



**Step 4: calculate the two mixture contributions.**



$$
 (1-z_t)h_{t-1}=(1-0.25)(0.4)=0.3
$$





$$
z_t\widetilde h_t=0.25\tanh(0.3)\approx0.072828153
$$



**Step 5: add the contributions.**



$$
h_t=0.3+0.25\tanh(0.3)\approx0.372828153
$$



**Final answer:** candidate ≈ 0.291313 and new hidden state ≈ 0.372828. The new scalar state lies between the old state 0.4 and the candidate 0.291313, as expected for this convex mixture.

### If the question uses the other update-gate convention

Some texts define z as the fraction of the **old state retained**:



$$
h_t=z_th_{t-1}+(1-z_t)\widetilde h_t
$$



Under that explicitly different convention, the same numerical value z = 0.25 gives:



$$
h_t=0.25(0.4)+0.75\tanh(0.3)\approx0.318484459
$$



This is not the answer to the worked question above, which states the first convention. To represent an identical mixture while switching conventions, the gate values must be complemented as well. Always identify the final mixture equation before substituting values.

### Relationship to LSTM

A basic GRU has three affine transformations: reset, update and candidate. A standard LSTM has four: forget, input, output and candidate. With equal input and hidden dimensions and the same bias convention, GRU therefore has fewer recurrent parameters. This is a parameter-count comparison, not a guarantee of higher accuracy or faster execution for every task.

### Retain-old GRU with gate 0

**Question. Use h\_new = z h\_old+(1−z) candidate. Given h\_old = 0.4, z = 0, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.**

### Given

h\_old = 0.4; z = 0; r = 0.5; x = 1.

W = 0.2; U = 0.5; b = 0; z is the retain-old gate.

### To find

Candidate and final hidden state.

### Formula — which one and why

The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate.

The question explicitly makes z multiply the old state. r and z do different jobs.



$$
\widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b)
$$





$$
h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde h
$$



**Yaad rakhne ka cue:** Reset sirf candidate mein; update mix mein original old state.

### Solution — step by step

**Step 1: reset the old state for candidate formation.**



$$
rh_{\mathrm{old}}=0.5(0.4)=0.2
$$



**Step 2: calculate the candidate raw sum.**



$$
a=0.2(1)+0.5(0.2)+0=0.3
$$



**Step 3: apply tanh.**



$$
\widetilde h=\tanh(0.3)\approx0.291312612
$$



**Step 4: retain the old-state fraction.**



$$
0(0.4)=0
$$



**Step 5: calculate the candidate fraction.**



$$
(1-0)(0.291312612)\approx0.291312612
$$



**Step 6: add the two fractions.**



$$
h_{\mathrm{new}}=0+(0.291312612)\approx0.291312612
$$



**Final answer:** Candidate ≈ 0.291312612; new state ≈ 0.291312612.

### Retain-old GRU with gate 0.8

**Question. Use h\_new = z h\_old+(1−z) candidate. Given h\_old = -0.2, z = 0.8, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.**

### Given

h\_old = -0.2; z = 0.8; r = 0.5; x = 1.

W = 0.2; U = 0.5; b = 0; z is the retain-old gate.

### To find

Candidate and final hidden state.

### Formula — which one and why

The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate.

The question explicitly makes z multiply the old state. r and z do different jobs.



$$
\widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b)
$$





$$
h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde h
$$



**Yaad rakhne ka cue:** Reset sirf candidate mein; update mix mein original old state.

### Solution — step by step

**Step 1: reset the old state for candidate formation.**



$$
rh_{\mathrm{old}}=0.5(-0.2)=-0.1
$$



**Step 2: calculate the candidate raw sum.**



$$
a=0.2(1)+0.5(-0.1)+0=0.15
$$



**Step 3: apply tanh.**



$$
\widetilde h=\tanh(0.15)\approx0.148885034
$$



**Step 4: retain the old-state fraction.**



$$
0.8(-0.2)=-0.16
$$



**Step 5: calculate the candidate fraction.**



$$
(1-0.8)(0.148885034)\approx0.029777007
$$



**Step 6: add the two fractions.**



$$
h_{\mathrm{new}}=-0.16+(0.029777007)\approx-0.130222993
$$



**Final answer:** Candidate ≈ 0.148885034; new state ≈ -0.130222993.

## Drill 44 GRU reset then mix

## Question

**Question.** hprevious = 0.4, r = 0.5, z = 0.25, x = 1, W = 0.2, U = 0.5, b = 0. Use candidate = tanh(Wx+U(r × hprevious)) and h = (1−z) hprevious+z candidate.

## Answer — step by step

### Given

Old hidden state = 0.4; reset r = 0.5; update z = 0.25.

x = 1; W = 0.2; U = 0.5; b = 0.

Use h = (1−z)h\_old + z×candidate.

### To find

Find reset contribution, candidate and new hidden state.

### Formula — which one and why

**GRU reset and state mixture.** The reset gate controls old-state information entering the candidate. The stated update convention then mixes the original old state and the candidate.

r = reset gate; z = update gate; h\_old = previous state; h\_tilde = candidate. Here z weights the candidate; some questions instead define z to weight the old state.



$$
\widetilde h=\tanh(Wx+U(r\odot h_{\mathrm{old}})+b)
$$





$$
h=(1-z)\odot h_{\mathrm{old}}+z\odot\widetilde h
$$



**Yaad rakhne ka cue:** Reset candidate ko control; update gate old-new mixture ko control.

### Solution — step by step

### Meaning and given convention

The reset gate r scales the old state while making the candidate. The update gate z controls the amount of candidate used in the final mixture **under this question's convention**. Some books reverse the roles of z and 1 − z, so copy the supplied formula.



$$
\widetilde h=\tanh\left(Wx+U(rh_{\mathrm{previous}})+b\right)
$$





$$
h=(1-z)h_{\mathrm{previous}}+z\widetilde h
$$



### Step 1: reset the old state for candidate construction



$$
rh_{\mathrm{previous}}=0.5\times0.4=0.2
$$



### Step 2: form the candidate's raw score



$$
Wx=0.2\times1=0.2
$$





$$
U(rh_{\mathrm{previous}})=0.5\times0.2=0.1
$$





$$
a=0.2+0.1+0=0.3
$$



### Step 3: apply tanh



$$
\widetilde h=\tanh(0.3)=0.291312612
$$



### Step 4: mix the original old state with the candidate



$$
1-z=1-0.25=0.75
$$





$$
h=0.75(0.4)+0.25(0.291312612)
$$





$$
h=0.3+0.072828153=0.372828153
$$



**Final answer: h ≈ 0.372828153.** The final old-state term uses 0.4, not the reset value 0.2.