# Drill 43 Calculate gates before the cell

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s43

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s43/

## Question

**Question.** Gate preactivations zf = zi = zo = 0, candidate preactivation 0.5, Cprevious = 0.2. Find gates, C, h.

## Answer — step by step

### Given

Raw gate scores: zf = zi = zo = 0.

Raw candidate score = 0.5; old cell C = 0.2.

### To find

Activate gates/candidate, then calculate the new cell and hidden state.

### Formula — which one and why

**Activate raw gate scores first.** A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations.

z\_f,z\_i,z\_o = gate preactivations; z\_c = candidate preactivation. An already activated gate must not be passed through sigmoid again.



$$
f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o)
$$





$$
\widetilde C=\tanh(z_c)
$$



**Yaad rakhne ka cue:** Raw score pehle activate; activated gate ko dobara sigmoid nahi.

**LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.

f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.



$$
C=f\odot C_{\mathrm{old}}+i\odot\widetilde C
$$





$$
h=o\odot\tanh C
$$



**Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate.

### Solution — step by step

### Meaning and formulas

The question supplies raw gate scores, called preactivations. Convert them to gates using sigmoid first. Convert the candidate's score using tanh.



$$
f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o),\quad \widetilde C=\tanh(z_C)
$$





$$
\sigma(z)=\frac1{1+e^{-z}},\quad C=fC_{\mathrm{previous}}+i\widetilde C,\quad h=o\tanh(C)
$$



### Step 1: turn each zero score into a gate



$$
\sigma(0)=\frac1{1+e^0}=\frac12=0.5
$$



Thus f = i = o = 0.5. A zero score is not a zero gate.

### Step 2: calculate candidate content



$$
\widetilde C=\tanh(0.5)=0.462117157
$$



### Step 3: combine retained memory and new content



$$
fC_{\mathrm{previous}}=0.5\times0.2=0.1
$$





$$
i\widetilde C=0.5\times0.462117157=0.231058579
$$





$$
C=0.1+0.231058579=0.331058579
$$



### Step 4: compute the visible hidden state



$$
\tanh(C)=\tanh(0.331058579)\approx0.319471636
$$





$$
h=0.5\times0.319471636=0.159735818
$$



**Final answer:** gates 0.5 each; candidate ≈ 0.462117157; C ≈ 0.331058579; h ≈ 0.159735818.