Deep Learning · TEE · section

Drill 50 Three step LSTM retention and replacement

Numerical practice from direct to mixed

Question

Question. C₀ = 0, output gate 0.7 at all steps. Supplied(f, i, candidate) are(1,.5,.8), (.9,0,−.2), (.2,.5,−.6). Find C and h at each step.

Answer — step by step

Given

Initial C₀ = 0; output gate o = 0.7 at all three steps.

Supplied (f,i,candidate): (1,0.5,0.8), (0.9,0,−0.2), (0.2,0.5,−0.6).

To find

Calculate C₁,h₁,C₂,h₂,C₃,h₃ in order.

Formula — which one and why

LSTM cell and hidden state. Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.

f,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.

C=f\odot C_{\mathrm{old}}+i\odot\widetilde CLaTeX: C=f\odot C_{\mathrm{old}}+i\odot\widetilde C
h=o\odot\tanh CLaTeX: h=o\odot\tanh C

Yaad rakhne ka cue: Purana rakho + naya lo; phir tanh aur output gate.

Solution — step by step

Meaning and formulas

At each time step use the previous cell state C, not the previous visible output h, in the memory equation. The output gate is 0.7 at every step.

C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t)LaTeX: C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t)

Step 1: Time 1: admit new content.

f_1C_0=1(0)=0,\qquad i_1\widetilde C_1=0.5(0.8)=0.4LaTeX: f_1C_0=1(0)=0,\qquad i_1\widetilde C_1=0.5(0.8)=0.4
C_1=0+0.4=0.4LaTeX: C_1=0+0.4=0.4
h_1=0.7\tanh(0.4)=0.7(0.379948962)=0.265964274LaTeX: h_1=0.7\tanh(0.4)=0.7(0.379948962)=0.265964274

Step 2: Time 2: retain memory and admit no new content.

f_2C_1=0.9(0.4)=0.36,\qquad i_2\widetilde C_2=0(-0.2)=0LaTeX: f_2C_1=0.9(0.4)=0.36,\qquad i_2\widetilde C_2=0(-0.2)=0
C_2=0.36+0=0.36LaTeX: C_2=0.36+0=0.36
h_2=0.7\tanh(0.36)=0.7(0.345214034)=0.241649824LaTeX: h_2=0.7\tanh(0.36)=0.7(0.345214034)=0.241649824

Step 3: Time 3: retain a small part and add negative content.

f_3C_2=0.2(0.36)=0.072,\qquad i_3\widetilde C_3=0.5(-0.6)=-0.3LaTeX: f_3C_2=0.2(0.36)=0.072,\qquad i_3\widetilde C_3=0.5(-0.6)=-0.3
C_3=0.072-0.3=-0.228LaTeX: C_3=0.072-0.3=-0.228
h_3=0.7\tanh(-0.228)=0.7(-0.224129674)=-0.156890772LaTeX: h_3=0.7\tanh(-0.228)=0.7(-0.224129674)=-0.156890772

Final cell states: 0.4, 0.36, −0.228. Final hidden states: approximately 0.265964274, 0.241649824, −0.156890772. The supplied gate value 1 is an idealized limit, used exactly as given.