# Drill 50 Three step LSTM retention and replacement

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s50

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s50/

## Question

**Question.** C₀ = 0, output gate 0.7 at all steps. Supplied(f, i, candidate) are(1,.5,.8), (.9,0,−.2), (.2,.5,−.6). Find C and h at each step.

## Answer — step by step

### Given

Initial C₀ = 0; output gate o = 0.7 at all three steps.

Supplied (f,i,candidate): (1,0.5,0.8), (0.9,0,−0.2), (0.2,0.5,−0.6).

### To find

Calculate C₁,h₁,C₂,h₂,C₃,h₃ in order.

### Formula — which one and why

**LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.

f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.



$$
C=f\odot C_{\mathrm{old}}+i\odot\widetilde C
$$





$$
h=o\odot\tanh C
$$



**Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate.

### Solution — step by step

### Meaning and formulas

At each time step use the previous **cell state C**, not the previous visible output h, in the memory equation. The output gate is 0.7 at every step.



$$
C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t)
$$



**Step 1: Time 1: admit new content.**



$$
f_1C_0=1(0)=0,\qquad i_1\widetilde C_1=0.5(0.8)=0.4
$$





$$
C_1=0+0.4=0.4
$$





$$
h_1=0.7\tanh(0.4)=0.7(0.379948962)=0.265964274
$$



**Step 2: Time 2: retain memory and admit no new content.**



$$
f_2C_1=0.9(0.4)=0.36,\qquad i_2\widetilde C_2=0(-0.2)=0
$$





$$
C_2=0.36+0=0.36
$$





$$
h_2=0.7\tanh(0.36)=0.7(0.345214034)=0.241649824
$$



**Step 3: Time 3: retain a small part and add negative content.**



$$
f_3C_2=0.2(0.36)=0.072,\qquad i_3\widetilde C_3=0.5(-0.6)=-0.3
$$





$$
C_3=0.072-0.3=-0.228
$$





$$
h_3=0.7\tanh(-0.228)=0.7(-0.224129674)=-0.156890772
$$



**Final cell states:** 0.4, 0.36, −0.228. **Final hidden states:** approximately 0.265964274, 0.241649824, −0.156890772. The supplied gate value 1 is an idealized limit, used exactly as given.