Deep Learning · TEE · formulas

LSTM candidate and output gate

The candidate is proposed new cell content, so it may be negative. The output gate is a positive sigmoid multiplier that controls how much of the activated cell is exposed. Candidate content and gate values therefore have different roles.

\widetilde C_t=\tanh(W_C[h_{t-1},x_t]+b_C)LaTeX: \widetilde C_t=\tanh(W_C[h_{t-1},x_t]+b_C)
o_t=\sigma(W_o[h_{t-1},x_t]+b_o)LaTeX: o_t=\sigma(W_o[h_{t-1},x_t]+b_o)

LSTM numerical when gates must first be calculated

Question. At an LSTM time step, xₜ = 1, hₜ₋₁ = 0 and Cₜ₋₁ = 0.2. The affine preactivations of the forget, input and output gates are all 0. The candidate preactivation is 0.5. Calculate the activated gates, candidate cell content, new cell state and new hidden state.

Given

x = 1; old hidden state = 0; old cell = 0.2.

Forget/input/output preactivations are all 0.

Candidate preactivation = 0.5.

To find

Activate all gates and candidate, then find C and h.

Formula — which one and why

Activate raw gate scores first. A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations.

z_f,z_i,z_o = gate preactivations; z_c = candidate preactivation. An already activated gate must not be passed through sigmoid again.

f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o)LaTeX: f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o)
\widetilde C=\tanh(z_c)LaTeX: \widetilde C=\tanh(z_c)

Yaad rakhne ka cue: Raw score pehle activate; activated gate ko dobara sigmoid nahi.

LSTM cell and hidden state. Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.

f,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.

C=f\odot C_{\mathrm{old}}+i\odot\widetilde CLaTeX: C=f\odot C_{\mathrm{old}}+i\odot\widetilde C
h=o\odot\tanh CLaTeX: h=o\odot\tanh C

Yaad rakhne ka cue: Purana rakho + naya lo; phir tanh aur output gate.

Solution — step by step

Distinguish preactivations from gate outputs

The affine transformations have already been evaluated in this question. Their outputs are preactivations, so activation functions are still required. Sigmoid is used for f, i and o; tanh is used for the candidate.

\sigma(a)=\frac1{1+e^{-a}}LaTeX: \sigma(a)=\frac1{1+e^{-a}}
C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t)LaTeX: C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t)
A gate preactivation of zero becomes a sigmoid gate output of 0.5. Use that activated gate value in the cell update. Tap the diagram to enlarge.

Step 1: activate the three gates.

f_t=i_t=o_t=\sigma(0)=\frac1{1+e^0}=0.5LaTeX: f_t=i_t=o_t=\sigma(0)=\frac1{1+e^0}=0.5

Step 2: activate the candidate preactivation.

\widetilde C_t=\tanh(0.5)\approx0.462117157LaTeX: \widetilde C_t=\tanh(0.5)\approx0.462117157

Step 3: calculate the old and new cell contributions separately.

f_tC_{t-1}=0.5(0.2)=0.1LaTeX: f_tC_{t-1}=0.5(0.2)=0.1
i_t\widetilde C_t=0.5\tanh(0.5)\approx0.231058579LaTeX: i_t\widetilde C_t=0.5\tanh(0.5)\approx0.231058579

Step 4: calculate the updated cell.

C_t=0.1+0.5\tanh(0.5)\approx0.331058579LaTeX: C_t=0.1+0.5\tanh(0.5)\approx0.331058579

Step 5: calculate the hidden output from the updated cell.

h_t=0.5\tanh(C_t)\approx0.159735818LaTeX: h_t=0.5\tanh(C_t)\approx0.159735818

Final answer: fₜ = iₜ = oₜ = 0.5; C̃ₜ ≈ 0.462117157; Cₜ ≈ 0.331058579; hₜ ≈ 0.159735818. Retain unrounded calculator values between steps. A gate preactivation of 0 is different from a supplied gate output of 0: the former becomes 0.5 under sigmoid, while the latter would suppress its contribution completely.

LSTM gates from raw score 0

Question. An LSTM has forget, input and output gate raw scores all equal to 0, candidate raw score 0, and old cell 1. Calculate all gates, candidate, new cell and hidden state.

Given

Raw scores z_f = z_i = z_o = z_candidate = 0.

Old cell C_old = 1; gate activation sigmoid; candidate activation tanh.

To find

f,i,o,candidate,C_new,h_new.

Formula — which one and why

Raw gate scores require sigmoid first. The candidate uses tanh; do not apply sigmoid to it.

f retains old memory, i admits candidate, o controls exposed hidden output; C is cell memory.

f=i=o=\sigma(z)LaTeX: f=i=o=\sigma(z)
\widetilde C=\tanh(z)LaTeX: \widetilde C=\tanh(z)
C_{\mathrm{new}}=fC_{\mathrm{old}}+i\widetilde CLaTeX: C_{\mathrm{new}}=fC_{\mathrm{old}}+i\widetilde C
h_{\mathrm{new}}=o\tanh(C_{\mathrm{new}})LaTeX: h_{\mathrm{new}}=o\tanh(C_{\mathrm{new}})

Yaad rakhne ka cue: Gates sigmoid; candidate tanh; cell mein add; hidden pe output gate.

Solution — step by step

Step 1: calculate the shared gate value.

\sigma(0)=\frac1{1+e^{-0}}\approx0.500000000LaTeX: \sigma(0)=\frac1{1+e^{-0}}\approx0.500000000

Step 2: calculate the candidate.

\widetilde C=\tanh(0)\approx0.000000000LaTeX: \widetilde C=\tanh(0)\approx0.000000000

Step 3: retain the old-cell part.

fC_{\mathrm{old}}=(0.500000000)(1)\approx0.500000000LaTeX: fC_{\mathrm{old}}=(0.500000000)(1)\approx0.500000000

Step 4: admit the new content.

i\widetilde C=(0.500000000)(0.000000000)\approx0.000000000LaTeX: i\widetilde C=(0.500000000)(0.000000000)\approx0.000000000

Step 5: add the memory contributions.

C_{\mathrm{new}}=0.500000000+(0.000000000)\approx0.500000000LaTeX: C_{\mathrm{new}}=0.500000000+(0.000000000)\approx0.500000000

Step 6: apply tanh to the cell.

\tanh(C_{\mathrm{new}})\approx0.462117157LaTeX: \tanh(C_{\mathrm{new}})\approx0.462117157

Step 7: apply the output gate.

h_{\mathrm{new}}=(0.500000000)(0.462117157)\approx0.231058579LaTeX: h_{\mathrm{new}}=(0.500000000)(0.462117157)\approx0.231058579

Final answer: f=i=o≈0.500000000; candidate≈0.000000000; cell≈0.500000000; hidden≈0.231058579.

LSTM gates from raw score 1

Question. An LSTM has forget, input and output gate raw scores all equal to 1, candidate raw score 1, and old cell -0.5. Calculate all gates, candidate, new cell and hidden state.

Given

Raw scores z_f = z_i = z_o = z_candidate = 1.

Old cell C_old = -0.5; gate activation sigmoid; candidate activation tanh.

To find

f,i,o,candidate,C_new,h_new.

Formula — which one and why

Raw gate scores require sigmoid first. The candidate uses tanh; do not apply sigmoid to it.

f retains old memory, i admits candidate, o controls exposed hidden output; C is cell memory.

f=i=o=\sigma(z)LaTeX: f=i=o=\sigma(z)
\widetilde C=\tanh(z)LaTeX: \widetilde C=\tanh(z)
C_{\mathrm{new}}=fC_{\mathrm{old}}+i\widetilde CLaTeX: C_{\mathrm{new}}=fC_{\mathrm{old}}+i\widetilde C
h_{\mathrm{new}}=o\tanh(C_{\mathrm{new}})LaTeX: h_{\mathrm{new}}=o\tanh(C_{\mathrm{new}})

Yaad rakhne ka cue: Gates sigmoid; candidate tanh; cell mein add; hidden pe output gate.

Solution — step by step

Step 1: calculate the shared gate value.

\sigma(1)=\frac1{1+e^{-1}}\approx0.731058579LaTeX: \sigma(1)=\frac1{1+e^{-1}}\approx0.731058579

Step 2: calculate the candidate.

\widetilde C=\tanh(1)\approx0.761594156LaTeX: \widetilde C=\tanh(1)\approx0.761594156

Step 3: retain the old-cell part.

fC_{\mathrm{old}}=(0.731058579)(-0.5)\approx-0.365529289LaTeX: fC_{\mathrm{old}}=(0.731058579)(-0.5)\approx-0.365529289

Step 4: admit the new content.

i\widetilde C=(0.731058579)(0.761594156)\approx0.556769941LaTeX: i\widetilde C=(0.731058579)(0.761594156)\approx0.556769941

Step 5: add the memory contributions.

C_{\mathrm{new}}=-0.365529289+(0.556769941)\approx0.191240652LaTeX: C_{\mathrm{new}}=-0.365529289+(0.556769941)\approx0.191240652

Step 6: apply tanh to the cell.

\tanh(C_{\mathrm{new}})\approx0.188942847LaTeX: \tanh(C_{\mathrm{new}})\approx0.188942847

Step 7: apply the output gate.

h_{\mathrm{new}}=(0.731058579)(0.188942847)\approx0.138128289LaTeX: h_{\mathrm{new}}=(0.731058579)(0.188942847)\approx0.138128289

Final answer: f=i=o≈0.731058579; candidate≈0.761594156; cell≈0.191240652; hidden≈0.138128289.

Drill 43 Calculate gates before the cell

Question

Question. Gate preactivations zf = zi = zo = 0, candidate preactivation 0.5, Cprevious = 0.2. Find gates, C, h.

Answer — step by step

Given

Raw gate scores: zf = zi = zo = 0.

Raw candidate score = 0.5; old cell C = 0.2.

To find

Activate gates/candidate, then calculate the new cell and hidden state.

Formula — which one and why

Activate raw gate scores first. A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations.

z_f,z_i,z_o = gate preactivations; z_c = candidate preactivation. An already activated gate must not be passed through sigmoid again.

f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o)LaTeX: f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o)
\widetilde C=\tanh(z_c)LaTeX: \widetilde C=\tanh(z_c)

Yaad rakhne ka cue: Raw score pehle activate; activated gate ko dobara sigmoid nahi.

LSTM cell and hidden state. Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.

f,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.

C=f\odot C_{\mathrm{old}}+i\odot\widetilde CLaTeX: C=f\odot C_{\mathrm{old}}+i\odot\widetilde C
h=o\odot\tanh CLaTeX: h=o\odot\tanh C

Yaad rakhne ka cue: Purana rakho + naya lo; phir tanh aur output gate.

Solution — step by step

Meaning and formulas

The question supplies raw gate scores, called preactivations. Convert them to gates using sigmoid first. Convert the candidate's score using tanh.

f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o),\quad \widetilde C=\tanh(z_C)LaTeX: f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o),\quad \widetilde C=\tanh(z_C)
\sigma(z)=\frac1{1+e^{-z}},\quad C=fC_{\mathrm{previous}}+i\widetilde C,\quad h=o\tanh(C)LaTeX: \sigma(z)=\frac1{1+e^{-z}},\quad C=fC_{\mathrm{previous}}+i\widetilde C,\quad h=o\tanh(C)

Step 1: turn each zero score into a gate

\sigma(0)=\frac1{1+e^0}=\frac12=0.5LaTeX: \sigma(0)=\frac1{1+e^0}=\frac12=0.5

Thus f = i = o = 0.5. A zero score is not a zero gate.

Step 2: calculate candidate content

\widetilde C=\tanh(0.5)=0.462117157LaTeX: \widetilde C=\tanh(0.5)=0.462117157

Step 3: combine retained memory and new content

fC_{\mathrm{previous}}=0.5\times0.2=0.1LaTeX: fC_{\mathrm{previous}}=0.5\times0.2=0.1
i\widetilde C=0.5\times0.462117157=0.231058579LaTeX: i\widetilde C=0.5\times0.462117157=0.231058579
C=0.1+0.231058579=0.331058579LaTeX: C=0.1+0.231058579=0.331058579

Step 4: compute the visible hidden state

\tanh(C)=\tanh(0.331058579)\approx0.319471636LaTeX: \tanh(C)=\tanh(0.331058579)\approx0.319471636
h=0.5\times0.319471636=0.159735818LaTeX: h=0.5\times0.319471636=0.159735818

Final answer: gates 0.5 each; candidate ≈ 0.462117157; C ≈ 0.331058579; h ≈ 0.159735818.

Drill 50 Three step LSTM retention and replacement

Question

Question. C₀ = 0, output gate 0.7 at all steps. Supplied(f, i, candidate) are(1,.5,.8), (.9,0,−.2), (.2,.5,−.6). Find C and h at each step.

Answer — step by step

Given

Initial C₀ = 0; output gate o = 0.7 at all three steps.

Supplied (f,i,candidate): (1,0.5,0.8), (0.9,0,−0.2), (0.2,0.5,−0.6).

To find

Calculate C₁,h₁,C₂,h₂,C₃,h₃ in order.

Formula — which one and why

LSTM cell and hidden state. Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.

f,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.

C=f\odot C_{\mathrm{old}}+i\odot\widetilde CLaTeX: C=f\odot C_{\mathrm{old}}+i\odot\widetilde C
h=o\odot\tanh CLaTeX: h=o\odot\tanh C

Yaad rakhne ka cue: Purana rakho + naya lo; phir tanh aur output gate.

Solution — step by step

Meaning and formulas

At each time step use the previous cell state C, not the previous visible output h, in the memory equation. The output gate is 0.7 at every step.

C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t)LaTeX: C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t)

Step 1: Time 1: admit new content.

f_1C_0=1(0)=0,\qquad i_1\widetilde C_1=0.5(0.8)=0.4LaTeX: f_1C_0=1(0)=0,\qquad i_1\widetilde C_1=0.5(0.8)=0.4
C_1=0+0.4=0.4LaTeX: C_1=0+0.4=0.4
h_1=0.7\tanh(0.4)=0.7(0.379948962)=0.265964274LaTeX: h_1=0.7\tanh(0.4)=0.7(0.379948962)=0.265964274

Step 2: Time 2: retain memory and admit no new content.

f_2C_1=0.9(0.4)=0.36,\qquad i_2\widetilde C_2=0(-0.2)=0LaTeX: f_2C_1=0.9(0.4)=0.36,\qquad i_2\widetilde C_2=0(-0.2)=0
C_2=0.36+0=0.36LaTeX: C_2=0.36+0=0.36
h_2=0.7\tanh(0.36)=0.7(0.345214034)=0.241649824LaTeX: h_2=0.7\tanh(0.36)=0.7(0.345214034)=0.241649824

Step 3: Time 3: retain a small part and add negative content.

f_3C_2=0.2(0.36)=0.072,\qquad i_3\widetilde C_3=0.5(-0.6)=-0.3LaTeX: f_3C_2=0.2(0.36)=0.072,\qquad i_3\widetilde C_3=0.5(-0.6)=-0.3
C_3=0.072-0.3=-0.228LaTeX: C_3=0.072-0.3=-0.228
h_3=0.7\tanh(-0.228)=0.7(-0.224129674)=-0.156890772LaTeX: h_3=0.7\tanh(-0.228)=0.7(-0.224129674)=-0.156890772

Final cell states: 0.4, 0.36, −0.228. Final hidden states: approximately 0.265964274, 0.241649824, −0.156890772. The supplied gate value 1 is an idealized limit, used exactly as given.