Deep Learning · TEE · formulas

LSTM cell and hidden state

First calculate the retained old contribution. Separately calculate the admitted new contribution. Add them to obtain the cell state. Only afterward apply tanh and the output gate to obtain the hidden state.

C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_tLaTeX: C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t
h_t=o_t\odot\tanh(C_t)LaTeX: h_t=o_t\odot\tanh(C_t)

Reading an LSTM equation one operation at a time

1. Two memories on the page do not mean the same value

The cell state c carries the recurrent cell memory. The hidden state h is an exposed output computed from that memory. An LSTM passes both to the next step. A question may give the previous cell and hidden states separately; do not substitute one for the other.

Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output. Tap the diagram to enlarge.

2. Gate values are multipliers

For a previous cell value of 0.8, a forget gate of 0.25 keeps 0.25 × 0.8 = 0.2. A gate of 0 keeps none of that contribution; a gate of 1 keeps the whole contribution. Intermediate sigmoid values give partial retention.

An input gate does the same kind of multiplication to the new candidate. If the candidate is −0.4 and the input gate is 0.5, the admitted new contribution is −0.2. The negative sign belongs to the candidate; sigmoid gates themselves are positive.

3. Add old and new contributions

With those values, the new cell is 0.2 + (−0.2) = 0. A retained positive contribution and an admitted negative contribution can cancel.

c_t=f_tc_{t-1}+i_t\widetilde c_tLaTeX: c_t=f_tc_{t-1}+i_t\widetilde c_t

This example also explains why “input gate” does not mean simply multiply the raw input x by i. It multiplies the candidate calculated for the cell update.

4. Calculate the hidden state afterward

Apply tanh to the updated cell and multiply by the output gate. If o = 0.7 and c = 0, tanh(0) = 0 and h = 0.7 × 0 = 0.

h_t=o_t\tanh(c_t)LaTeX: h_t=o_t\tanh(c_t)

If c were 1 instead, tanh(1) ≈ 0.761594 and h ≈ 0.533116 for the same output gate. The output gate controls exposure of the transformed cell, not the earlier addition that produced c.

5. Supplied gates and calculated gates are different numerical types

When f, i and o are supplied, begin directly with their multiplications. When weights and inputs are supplied, first calculate each gate's affine input, apply sigmoid and retain those gate values. Calculate the candidate using tanh. Only then update c and h. The separate solved examples below cover both types and multi-step repetition.

Drill 42 LSTM with a negative candidate

Question

Question. f = 0.8, i = 0.2, o = 0.6, candidate = −0.5, Cprevious = 0.3. Find C, h.

Answer — step by step

Given

Activated gates: f = 0.8, i = 0.2, o = 0.6.

Candidate = −0.5; old cell C = 0.3.

To find

Find the new cell and hidden state.

Formula — which one and why

LSTM cell and hidden state. Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.

f,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.

C=f\odot C_{\mathrm{old}}+i\odot\widetilde CLaTeX: C=f\odot C_{\mathrm{old}}+i\odot\widetilde C
h=o\odot\tanh CLaTeX: h=o\odot\tanh C

Yaad rakhne ka cue: Purana rakho + naya lo; phir tanh aur output gate.

Solution — step by step

Meaning and formulas

An LSTM keeps a cell memory C and a visible hidden state h. The forget gate f controls old memory, the input gate i controls new candidate content, and the output gate o controls the visible state. These gate values are already supplied.

C=fC_{\mathrm{previous}}+i\widetilde C,\qquad h=o\tanh(C)LaTeX: C=fC_{\mathrm{previous}}+i\widetilde C,\qquad h=o\tanh(C)

The candidate, written C with a tilde, may be negative. A negative candidate subtracts content when admitted.

Step 1: retained old memory

fC_{\mathrm{previous}}=0.8\times0.3=0.24LaTeX: fC_{\mathrm{previous}}=0.8\times0.3=0.24

Step 2: admitted new content

i\widetilde C=0.2\times(-0.5)=-0.1LaTeX: i\widetilde C=0.2\times(-0.5)=-0.1

Step 3: add them to obtain the new cell memory

C=0.24+(-0.1)=0.14LaTeX: C=0.24+(-0.1)=0.14

Step 4: apply tanh to the cell memory

\tanh(0.14)=\frac{e^{0.28}-1}{e^{0.28}+1}=0.139092448LaTeX: \tanh(0.14)=\frac{e^{0.28}-1}{e^{0.28}+1}=0.139092448

Step 5: apply the output gate

h=0.6\times0.139092448=0.083455469LaTeX: h=0.6\times0.139092448=0.083455469

Final answer: cell memory 0.14; hidden state approximately 0.083455469. C and h are different quantities.

Drill 50 Three step LSTM retention and replacement

Question

Question. C₀ = 0, output gate 0.7 at all steps. Supplied(f, i, candidate) are(1,.5,.8), (.9,0,−.2), (.2,.5,−.6). Find C and h at each step.

Answer — step by step

Given

Initial C₀ = 0; output gate o = 0.7 at all three steps.

Supplied (f,i,candidate): (1,0.5,0.8), (0.9,0,−0.2), (0.2,0.5,−0.6).

To find

Calculate C₁,h₁,C₂,h₂,C₃,h₃ in order.

Formula — which one and why

LSTM cell and hidden state. Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.

f,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.

C=f\odot C_{\mathrm{old}}+i\odot\widetilde CLaTeX: C=f\odot C_{\mathrm{old}}+i\odot\widetilde C
h=o\odot\tanh CLaTeX: h=o\odot\tanh C

Yaad rakhne ka cue: Purana rakho + naya lo; phir tanh aur output gate.

Solution — step by step

Meaning and formulas

At each time step use the previous cell state C, not the previous visible output h, in the memory equation. The output gate is 0.7 at every step.

C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t)LaTeX: C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t)

Step 1: Time 1: admit new content.

f_1C_0=1(0)=0,\qquad i_1\widetilde C_1=0.5(0.8)=0.4LaTeX: f_1C_0=1(0)=0,\qquad i_1\widetilde C_1=0.5(0.8)=0.4
C_1=0+0.4=0.4LaTeX: C_1=0+0.4=0.4
h_1=0.7\tanh(0.4)=0.7(0.379948962)=0.265964274LaTeX: h_1=0.7\tanh(0.4)=0.7(0.379948962)=0.265964274

Step 2: Time 2: retain memory and admit no new content.

f_2C_1=0.9(0.4)=0.36,\qquad i_2\widetilde C_2=0(-0.2)=0LaTeX: f_2C_1=0.9(0.4)=0.36,\qquad i_2\widetilde C_2=0(-0.2)=0
C_2=0.36+0=0.36LaTeX: C_2=0.36+0=0.36
h_2=0.7\tanh(0.36)=0.7(0.345214034)=0.241649824LaTeX: h_2=0.7\tanh(0.36)=0.7(0.345214034)=0.241649824

Step 3: Time 3: retain a small part and add negative content.

f_3C_2=0.2(0.36)=0.072,\qquad i_3\widetilde C_3=0.5(-0.6)=-0.3LaTeX: f_3C_2=0.2(0.36)=0.072,\qquad i_3\widetilde C_3=0.5(-0.6)=-0.3
C_3=0.072-0.3=-0.228LaTeX: C_3=0.072-0.3=-0.228
h_3=0.7\tanh(-0.228)=0.7(-0.224129674)=-0.156890772LaTeX: h_3=0.7\tanh(-0.228)=0.7(-0.224129674)=-0.156890772

Final cell states: 0.4, 0.36, −0.228. Final hidden states: approximately 0.265964274, 0.241649824, −0.156890772. The supplied gate value 1 is an idealized limit, used exactly as given.

Drill 43 Calculate gates before the cell

Question

Question. Gate preactivations zf = zi = zo = 0, candidate preactivation 0.5, Cprevious = 0.2. Find gates, C, h.

Answer — step by step

Given

Raw gate scores: zf = zi = zo = 0.

Raw candidate score = 0.5; old cell C = 0.2.

To find

Activate gates/candidate, then calculate the new cell and hidden state.

Formula — which one and why

Activate raw gate scores first. A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations.

z_f,z_i,z_o = gate preactivations; z_c = candidate preactivation. An already activated gate must not be passed through sigmoid again.

f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o)LaTeX: f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o)
\widetilde C=\tanh(z_c)LaTeX: \widetilde C=\tanh(z_c)

Yaad rakhne ka cue: Raw score pehle activate; activated gate ko dobara sigmoid nahi.

LSTM cell and hidden state. Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.

f,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.

C=f\odot C_{\mathrm{old}}+i\odot\widetilde CLaTeX: C=f\odot C_{\mathrm{old}}+i\odot\widetilde C
h=o\odot\tanh CLaTeX: h=o\odot\tanh C

Yaad rakhne ka cue: Purana rakho + naya lo; phir tanh aur output gate.

Solution — step by step

Meaning and formulas

The question supplies raw gate scores, called preactivations. Convert them to gates using sigmoid first. Convert the candidate's score using tanh.

f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o),\quad \widetilde C=\tanh(z_C)LaTeX: f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o),\quad \widetilde C=\tanh(z_C)
\sigma(z)=\frac1{1+e^{-z}},\quad C=fC_{\mathrm{previous}}+i\widetilde C,\quad h=o\tanh(C)LaTeX: \sigma(z)=\frac1{1+e^{-z}},\quad C=fC_{\mathrm{previous}}+i\widetilde C,\quad h=o\tanh(C)

Step 1: turn each zero score into a gate

\sigma(0)=\frac1{1+e^0}=\frac12=0.5LaTeX: \sigma(0)=\frac1{1+e^0}=\frac12=0.5

Thus f = i = o = 0.5. A zero score is not a zero gate.

Step 2: calculate candidate content

\widetilde C=\tanh(0.5)=0.462117157LaTeX: \widetilde C=\tanh(0.5)=0.462117157

Step 3: combine retained memory and new content

fC_{\mathrm{previous}}=0.5\times0.2=0.1LaTeX: fC_{\mathrm{previous}}=0.5\times0.2=0.1
i\widetilde C=0.5\times0.462117157=0.231058579LaTeX: i\widetilde C=0.5\times0.462117157=0.231058579
C=0.1+0.231058579=0.331058579LaTeX: C=0.1+0.231058579=0.331058579

Step 4: compute the visible hidden state

\tanh(C)=\tanh(0.331058579)\approx0.319471636LaTeX: \tanh(C)=\tanh(0.331058579)\approx0.319471636
h=0.5\times0.319471636=0.159735818LaTeX: h=0.5\times0.319471636=0.159735818

Final answer: gates 0.5 each; candidate ≈ 0.462117157; C ≈ 0.331058579; h ≈ 0.159735818.