Deep Learning · TEE · formulas

GRU state under the retain-old convention

Some GRU questions use the opposite naming convention: z is the fraction of the OLD state retained. The formula below uses that convention. Do not combine its formula with an explanation that says z retains the new candidate.

\widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h)LaTeX: \widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h)
h_t=z_t\odot h_{t-1}+(1-z_t)\odot\widetilde h_tLaTeX: h_t=z_t\odot h_{t-1}+(1-z_t)\odot\widetilde h_t

Question. Use the retain-old GRU convention. Given hprevious = 0.4, reset gate r = 0.5, update gate z = 0.25, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and the next hidden state.

Given

h_old = 0.4; r = 0.5; z = 0.25; x = 1.

W = 0.2; U = 0.5; b = 0; z retains the old state.

To find

Find the candidate and new hidden state.

Formula — which one and why

GRU with a retain-old update gate. Form the candidate using reset memory; then z retains the old state and 1−z admits the candidate. Follow this question-specific convention.

r = reset gate; z = retain-old gate; h_old = previous state; h̃ = candidate.

\widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b)LaTeX: \widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b)
h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde hLaTeX: h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde h

Yaad rakhne ka cue: Question mein z kisko multiply karta hai wahi meaning; yahan old ko retain.

Solution — step by step

Step 1: reset the old state for candidate formation.

r h_{\mathrm{previous}}=0.5(0.4)=0.2LaTeX: r h_{\mathrm{previous}}=0.5(0.4)=0.2

Step 2: calculate the candidate's raw input. Its new-input contribution is 0.2 × 1; its reset-memory contribution is 0.5 × 0.2.

a=0.2(1)+0.5(0.2)+0=0.2+0.1=0.3LaTeX: a=0.2(1)+0.5(0.2)+0=0.2+0.1=0.3

Step 3: apply tanh to this raw input.

\widetilde h=\tanh(0.3)\approx0.291312612LaTeX: \widetilde h=\tanh(0.3)\approx0.291312612

Step 4: calculate how much old state is retained. In this question, z = 0.25 multiplies the original old state 0.4.

z h_{\mathrm{previous}}=0.25(0.4)=0.1LaTeX: z h_{\mathrm{previous}}=0.25(0.4)=0.1

Step 5: calculate how much candidate is admitted. The other fraction is 1 − z = 0.75.

1-z=1-0.25=0.75LaTeX: 1-z=1-0.25=0.75
(1-z)\widetilde h=0.75(0.291312612)\approx0.218484459LaTeX: (1-z)\widetilde h=0.75(0.291312612)\approx0.218484459

Step 6: add the two final contributions.

h=0.1+0.218484459=0.318484459LaTeX: h=0.1+0.218484459=0.318484459

Final answer: h ≈ 0.318484459. The candidate was 0.291312612; the final hidden state is a mixture of that candidate and the original old state. Under the admit-new convention with the same numeric z, the result would instead be 0.372828153.

Retain-old GRU with gate 0

Question. Use h_new = z h_old+(1−z) candidate. Given h_old = 0.4, z = 0, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.

Given

h_old = 0.4; z = 0; r = 0.5; x = 1.

W = 0.2; U = 0.5; b = 0; z is the retain-old gate.

To find

Candidate and final hidden state.

Formula — which one and why

The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate.

The question explicitly makes z multiply the old state. r and z do different jobs.

\widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b)LaTeX: \widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b)
h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde hLaTeX: h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde h

Yaad rakhne ka cue: Reset sirf candidate mein; update mix mein original old state.

Solution — step by step

Step 1: reset the old state for candidate formation.

rh_{\mathrm{old}}=0.5(0.4)=0.2LaTeX: rh_{\mathrm{old}}=0.5(0.4)=0.2

Step 2: calculate the candidate raw sum.

a=0.2(1)+0.5(0.2)+0=0.3LaTeX: a=0.2(1)+0.5(0.2)+0=0.3

Step 3: apply tanh.

\widetilde h=\tanh(0.3)\approx0.291312612LaTeX: \widetilde h=\tanh(0.3)\approx0.291312612

Step 4: retain the old-state fraction.

0(0.4)=0LaTeX: 0(0.4)=0

Step 5: calculate the candidate fraction.

(1-0)(0.291312612)\approx0.291312612LaTeX: (1-0)(0.291312612)\approx0.291312612

Step 6: add the two fractions.

h_{\mathrm{new}}=0+(0.291312612)\approx0.291312612LaTeX: h_{\mathrm{new}}=0+(0.291312612)\approx0.291312612

Final answer: Candidate ≈ 0.291312612; new state ≈ 0.291312612.

Retain-old GRU with gate 0.8

Question. Use h_new = z h_old+(1−z) candidate. Given h_old = -0.2, z = 0.8, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.

Given

h_old = -0.2; z = 0.8; r = 0.5; x = 1.

W = 0.2; U = 0.5; b = 0; z is the retain-old gate.

To find

Candidate and final hidden state.

Formula — which one and why

The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate.

The question explicitly makes z multiply the old state. r and z do different jobs.

\widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b)LaTeX: \widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b)
h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde hLaTeX: h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde h

Yaad rakhne ka cue: Reset sirf candidate mein; update mix mein original old state.

Solution — step by step

Step 1: reset the old state for candidate formation.

rh_{\mathrm{old}}=0.5(-0.2)=-0.1LaTeX: rh_{\mathrm{old}}=0.5(-0.2)=-0.1

Step 2: calculate the candidate raw sum.

a=0.2(1)+0.5(-0.1)+0=0.15LaTeX: a=0.2(1)+0.5(-0.1)+0=0.15

Step 3: apply tanh.

\widetilde h=\tanh(0.15)\approx0.148885034LaTeX: \widetilde h=\tanh(0.15)\approx0.148885034

Step 4: retain the old-state fraction.

0.8(-0.2)=-0.16LaTeX: 0.8(-0.2)=-0.16

Step 5: calculate the candidate fraction.

(1-0.8)(0.148885034)\approx0.029777007LaTeX: (1-0.8)(0.148885034)\approx0.029777007

Step 6: add the two fractions.

h_{\mathrm{new}}=-0.16+(0.029777007)\approx-0.130222993LaTeX: h_{\mathrm{new}}=-0.16+(0.029777007)\approx-0.130222993

Final answer: Candidate ≈ 0.148885034; new state ≈ -0.130222993.