Deep Learning · TEE · section

Drill 43 Calculate gates before the cell

Numerical practice from direct to mixed

Question

Question. Gate preactivations zf = zi = zo = 0, candidate preactivation 0.5, Cprevious = 0.2. Find gates, C, h.

Answer — step by step

Given

Raw gate scores: zf = zi = zo = 0.

Raw candidate score = 0.5; old cell C = 0.2.

To find

Activate gates/candidate, then calculate the new cell and hidden state.

Formula — which one and why

Activate raw gate scores first. A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations.

z_f,z_i,z_o = gate preactivations; z_c = candidate preactivation. An already activated gate must not be passed through sigmoid again.

f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o)LaTeX: f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o)
\widetilde C=\tanh(z_c)LaTeX: \widetilde C=\tanh(z_c)

Yaad rakhne ka cue: Raw score pehle activate; activated gate ko dobara sigmoid nahi.

LSTM cell and hidden state. Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.

f,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.

C=f\odot C_{\mathrm{old}}+i\odot\widetilde CLaTeX: C=f\odot C_{\mathrm{old}}+i\odot\widetilde C
h=o\odot\tanh CLaTeX: h=o\odot\tanh C

Yaad rakhne ka cue: Purana rakho + naya lo; phir tanh aur output gate.

Solution — step by step

Meaning and formulas

The question supplies raw gate scores, called preactivations. Convert them to gates using sigmoid first. Convert the candidate's score using tanh.

f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o),\quad \widetilde C=\tanh(z_C)LaTeX: f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o),\quad \widetilde C=\tanh(z_C)
\sigma(z)=\frac1{1+e^{-z}},\quad C=fC_{\mathrm{previous}}+i\widetilde C,\quad h=o\tanh(C)LaTeX: \sigma(z)=\frac1{1+e^{-z}},\quad C=fC_{\mathrm{previous}}+i\widetilde C,\quad h=o\tanh(C)

Step 1: turn each zero score into a gate

\sigma(0)=\frac1{1+e^0}=\frac12=0.5LaTeX: \sigma(0)=\frac1{1+e^0}=\frac12=0.5

Thus f = i = o = 0.5. A zero score is not a zero gate.

Step 2: calculate candidate content

\widetilde C=\tanh(0.5)=0.462117157LaTeX: \widetilde C=\tanh(0.5)=0.462117157

Step 3: combine retained memory and new content

fC_{\mathrm{previous}}=0.5\times0.2=0.1LaTeX: fC_{\mathrm{previous}}=0.5\times0.2=0.1
i\widetilde C=0.5\times0.462117157=0.231058579LaTeX: i\widetilde C=0.5\times0.462117157=0.231058579
C=0.1+0.231058579=0.331058579LaTeX: C=0.1+0.231058579=0.331058579

Step 4: compute the visible hidden state

\tanh(C)=\tanh(0.331058579)\approx0.319471636LaTeX: \tanh(C)=\tanh(0.331058579)\approx0.319471636
h=0.5\times0.319471636=0.159735818LaTeX: h=0.5\times0.319471636=0.159735818

Final answer: gates 0.5 each; candidate ≈ 0.462117157; C ≈ 0.331058579; h ≈ 0.159735818.