Drill 42 LSTM with a negative candidate

Answer — step by step

Given

Activated gates: f = 0.8, i = 0.2, o = 0.6.

Candidate = −0.5; old cell C = 0.3.

To find

Find the new cell and hidden state.

Formula — which one and why

LSTM cell and hidden state. Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.

f,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.

LaTeX: C=f\odot C_{\mathrm{old}}+i\odot\widetilde C

LaTeX: h=o\odot\tanh C

Yaad rakhne ka cue: Purana rakho + naya lo; phir tanh aur output gate.

Solution — step by step

Meaning and formulas

An LSTM keeps a cell memory C and a visible hidden state h. The forget gate f controls old memory, the input gate i controls new candidate content, and the output gate o controls the visible state. These gate values are already supplied.

LaTeX: C=fC_{\mathrm{previous}}+i\widetilde C,\qquad h=o\tanh(C)

The candidate, written C with a tilde, may be negative. A negative candidate subtracts content when admitted.

Step 1: retained old memory

LaTeX: fC_{\mathrm{previous}}=0.8\times0.3=0.24

Step 2: admitted new content

LaTeX: i\widetilde C=0.2\times(-0.5)=-0.1

Step 3: add them to obtain the new cell memory

LaTeX: C=0.24+(-0.1)=0.14

Step 4: apply tanh to the cell memory

LaTeX: \tanh(0.14)=\frac{e^{0.28}-1}{e^{0.28}+1}=0.139092448

Step 5: apply the output gate

LaTeX: h=0.6\times0.139092448=0.083455469

Final answer: cell memory 0.14; hidden state approximately 0.083455469. C and h are different quantities.