Deep Learning · TEE · section

LSTM complete numerical with supplied gates

Recurrent neural networks

Question. An LSTM cell has input gate iₜ = 0.7, forget gate fₜ = 0.4, output gate oₜ = 0.8, candidate cell content C̃ₜ = 0.9 and previous cell state Cₜ₋₁ = 0.5. Calculate the new cell state Cₜ and hidden state hₜ.

Given

Activated input gate i = 0.7; forget gate f = 0.4; output gate o = 0.8.

Candidate = 0.9; old cell = 0.5.

To find

Find the new cell C and hidden state h.

Formula — which one and why

LSTM cell and hidden state. Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.

f,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.

$C=f\odot C_{\mathrm{old}}+i\odot\widetilde C$ LaTeX: C=f\odot C_{\mathrm{old}}+i\odot\widetilde C

$h=o\odot\tanh C$ LaTeX: h=o\odot\tanh C

Yaad rakhne ka cue: Purana rakho + naya lo; phir tanh aur output gate.

Solution — step by step

Equations and supplied-value convention

The question supplies activated gate outputs and the candidate content. Do not apply sigmoid or tanh to those supplied values again. First update the cell, then calculate the hidden state:

$C_t=f_tC_{t-1}+i_t\widetilde C_t$ LaTeX: C_t=f_tC_{t-1}+i_t\widetilde C_t

$h_t=o_t\tanh(C_t)$ LaTeX: h_t=o_t\tanh(C_t)

The first equation combines retained old content and admitted candidate content. The second equation is a separate transformation of the updated cell.

The diagram uses the supplied gate values. The output gate is applied after the cell has been updated and passed through tanh. Tap the diagram to enlarge.

Step 1: calculate the retained old-cell contribution.

$f_tC_{t-1}=0.4(0.5)=0.2$ LaTeX: f_tC_{t-1}=0.4(0.5)=0.2

Step 2: calculate the admitted candidate contribution.

$i_t\widetilde C_t=0.7(0.9)=0.63$ LaTeX: i_t\widetilde C_t=0.7(0.9)=0.63

Step 3: add the two contributions to obtain the new cell.

$C_t=0.2+0.63=0.83$ LaTeX: C_t=0.2+0.63=0.83

Step 4: activate the updated cell.

Step 5: multiply by the output gate.

Final answer: Cₜ = 0.83 and hₜ ≈ 0.544381. Multiplying the output gate directly by Cₜ would give 0.664 and omit the required tanh operation.

Related vector calculation: retention alone

Question. Given forget gate f = [0.1, 0.9] and previous cell C = [2, 1], calculate the retained old-cell contribution.

Given

Forget gate = [0.1,0.9]; old cell = [2,1].

Input-gate and candidate values are not supplied.

To find

Find only the retained old-cell contribution.

Formula — which one and why

LSTM retained old memory. Multiply each old cell value by its corresponding forget-gate value. This gives only the old contribution, not the complete new cell.

f = activated forget gate; C_old = previous cell state; ⊙ means multiply matching entries.

$C_{\mathrm{retained}}=f\odot C_{\mathrm{old}}$ LaTeX: C_{\mathrm{retained}}=f\odot C_{\mathrm{old}}

Yaad rakhne ka cue: Forget gate × old cell = bacha hua purana hissa.

Solution — step by step

Step 1: retain the first old-cell component.

$0.1(2)=0.2$ LaTeX: 0.1(2)=0.2

Step 2: retain the second old-cell component.

$0.9(1)=0.9$ LaTeX: 0.9(1)=0.9

Step 3: assemble the two retained components.

Keep their original order. This gives the following column:

$f\odot C=\begin{bmatrix}0.1(2)\\0.9(1)\end{bmatrix}=\begin{bmatrix}0.2\\0.9\end{bmatrix}$ LaTeX: f\odot C=\begin{bmatrix}0.1(2)\\0.9(1)\end{bmatrix}=\begin{bmatrix}0.2\\0.9\end{bmatrix}

Answer: [0.2, 0.9]. This is the retained contribution, not necessarily the final new cell. The candidate contribution must also be known to calculate the full cell update.