# LSTM gates and candidate

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/41

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/41/

An LSTM gate is a multiplier between zero and one. The forget gate chooses the fraction of old cell content retained; the input gate chooses the fraction of candidate content admitted. Each gate is calculated from its own weighted sum and sigmoid.



$$
f_t=\sigma(W_f[h_{t-1},x_t]+b_f)
$$





$$
i_t=\sigma(W_i[h_{t-1},x_t]+b_i)
$$



## LSTM a memory path with three gates

**Definition.** Long short-term memory (LSTM) is a gated recurrent architecture with a cell state and hidden state. Forget, input and output gates regulate state retention, incorporation of candidate information and exposure of the updated cell.

### Explanation and working

Long short-term memory, or LSTM, keeps a cell state Cₜ and a hidden state hₜ. A sigmoid gate produces values from 0 to 1: approximately 0 blocks a component, approximately 1 passes it. Intermediate values pass a fraction. Gates are learned from the previous hidden state and current input, not manually assigned by the user during normal inference.

![Follow the cell state from left to right then identify forget input and output gates](https://deep-learning-study-room.vercel.app/assets/figures/lstm.svg)

Diagram: Follow the cell state from left to right then identify forget input and output gates. Labels: C previous | × | + | C new | tanh | × | h | new | σ: f | σ: i | tanh: C~ | σ: o | × | [h previous, x current] | Each gate has its own learned weights and bias.

Follow the cell state from left to right then identify forget input and output gates Tap the diagram to enlarge.



$$
f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i)
$$





$$
\widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c)
$$





$$
C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t
$$





$$
o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\qquad h_t=o_t\odot\tanh(C_t)
$$



The brackets [h, x] concatenate two vectors: [a, b] joined with [c, d] gives [a, b, c, d]. Concatenation is not addition. f is the forget gate: how much old cell content remains. i is the input gate: how much new candidate enters. The candidate is proposed content. o is the output gate: how much of the activated cell becomes visible as the hidden state.

## Reading an LSTM equation one operation at a time

### 1. Two memories on the page do not mean the same value

The cell state c carries the recurrent cell memory. The hidden state h is an exposed output computed from that memory. An LSTM passes both to the next step. A question may give the previous cell and hidden states separately; do not substitute one for the other.

![Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output.](https://deep-learning-study-room.vercel.app/assets/figures/v_lstm_split.svg)

Diagram: Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output.. Labels: Calculate cell memory before the hidden output | Calculate cell memory before the hidden output | Example: old cell 0.8, forget gate 0.25, input gate 0.5, candidate −0.4. | Retain old content | 0.25 × 0.8 = 0.2 | Admit new content | 0.5 × (−0.4) = −0.2 | New cell | 0.2 + (−0.2) = 0 | Hidden output: 0.7 × tanh(0) = 0 | The output gate is 0.7 here. Apply it to tanh of the NEW cell, after the cell | update.

Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output. Tap the diagram to enlarge.

### 2. Gate values are multipliers

For a previous cell value of 0.8, a forget gate of 0.25 keeps 0.25 × 0.8 = 0.2. A gate of 0 keeps none of that contribution; a gate of 1 keeps the whole contribution. Intermediate sigmoid values give partial retention.

An input gate does the same kind of multiplication to the new candidate. If the candidate is −0.4 and the input gate is 0.5, the admitted new contribution is −0.2. The negative sign belongs to the candidate; sigmoid gates themselves are positive.

### 3. Add old and new contributions

With those values, the new cell is 0.2 + (−0.2) = 0. A retained positive contribution and an admitted negative contribution can cancel.



$$
c_t=f_tc_{t-1}+i_t\widetilde c_t
$$



This example also explains why “input gate” does not mean simply multiply the raw input x by i. It multiplies the candidate calculated for the cell update.

### 4. Calculate the hidden state afterward

Apply tanh to the updated cell and multiply by the output gate. If o = 0.7 and c = 0, tanh(0) = 0 and h = 0.7 × 0 = 0.



$$
h_t=o_t\tanh(c_t)
$$



If c were 1 instead, tanh(1) ≈ 0.761594 and h ≈ 0.533116 for the same output gate. The output gate controls exposure of the transformed cell, not the earlier addition that produced c.

### 5. Supplied gates and calculated gates are different numerical types

When f, i and o are supplied, begin directly with their multiplications. When weights and inputs are supplied, first calculate each gate's affine input, apply sigmoid and retain those gate values. Calculate the candidate using tanh. Only then update c and h. The separate solved examples below cover both types and multi-step repetition.

## LSTM numerical when gates must first be calculated

**Question.** At an LSTM time step, xₜ = 1, hₜ₋₁ = 0 and Cₜ₋₁ = 0.2. The affine preactivations of the forget, input and output gates are all 0. The candidate preactivation is 0.5. Calculate the activated gates, candidate cell content, new cell state and new hidden state.

### Given

x = 1; old hidden state = 0; old cell = 0.2.

Forget/input/output preactivations are all 0.

Candidate preactivation = 0.5.

### To find

Activate all gates and candidate, then find C and h.

### Formula — which one and why

**Activate raw gate scores first.** A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations.

z\_f,z\_i,z\_o = gate preactivations; z\_c = candidate preactivation. An already activated gate must not be passed through sigmoid again.



$$
f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o)
$$





$$
\widetilde C=\tanh(z_c)
$$



**Yaad rakhne ka cue:** Raw score pehle activate; activated gate ko dobara sigmoid nahi.

**LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.

f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.



$$
C=f\odot C_{\mathrm{old}}+i\odot\widetilde C
$$





$$
h=o\odot\tanh C
$$



**Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate.

### Solution — step by step

### Distinguish preactivations from gate outputs

The affine transformations have already been evaluated in this question. Their outputs are preactivations, so activation functions are still required. Sigmoid is used for f, i and o; tanh is used for the candidate.



$$
\sigma(a)=\frac1{1+e^{-a}}
$$





$$
C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t)
$$



![A gate preactivation of zero becomes a sigmoid gate output of 0.5. Use that activated gate value in the cell update.](https://deep-learning-study-room.vercel.app/assets/figures/v_lstm_gates_first.svg)

Diagram: A gate preactivation of zero becomes a sigmoid gate output of 0.5. Use that activated gate value in the cell update.. Labels: When the question supplies gate raw scores | When the question supplies gate raw scores | A gate raw score is not yet a gate value. | Gate scores | f, i, o scores | = 0 | Apply sigmoid | Each gate | becomes 0.5 | Candidate | score | 0.5 → tanh → | 0.462117 | Then update | Compute cell, | then hidden | output | Sigmoid(0) = 0.5. Do not replace a zero raw score with a closed gate.

A gate preactivation of zero becomes a sigmoid gate output of 0.5. Use that activated gate value in the cell update. Tap the diagram to enlarge.

**Step 1: activate the three gates.**



$$
f_t=i_t=o_t=\sigma(0)=\frac1{1+e^0}=0.5
$$



**Step 2: activate the candidate preactivation.**



$$
\widetilde C_t=\tanh(0.5)\approx0.462117157
$$



**Step 3: calculate the old and new cell contributions separately.**



$$
f_tC_{t-1}=0.5(0.2)=0.1
$$





$$
i_t\widetilde C_t=0.5\tanh(0.5)\approx0.231058579
$$



**Step 4: calculate the updated cell.**



$$
C_t=0.1+0.5\tanh(0.5)\approx0.331058579
$$



**Step 5: calculate the hidden output from the updated cell.**



$$
h_t=0.5\tanh(C_t)\approx0.159735818
$$



**Final answer:** fₜ = iₜ = oₜ = 0.5; C̃ₜ ≈ 0.462117157; Cₜ ≈ 0.331058579; hₜ ≈ 0.159735818. Retain unrounded calculator values between steps. A gate preactivation of 0 is different from a supplied gate output of 0: the former becomes 0.5 under sigmoid, while the latter would suppress its contribution completely.

### LSTM gates from raw score 0

**Question. An LSTM has forget, input and output gate raw scores all equal to 0, candidate raw score 0, and old cell 1. Calculate all gates, candidate, new cell and hidden state.**

### Given

Raw scores z\_f = z\_i = z\_o = z\_candidate = 0.

Old cell C\_old = 1; gate activation sigmoid; candidate activation tanh.

### To find

f,i,o,candidate,C\_new,h\_new.

### Formula — which one and why

Raw gate scores require sigmoid first. The candidate uses tanh; do not apply sigmoid to it.

f retains old memory, i admits candidate, o controls exposed hidden output; C is cell memory.



$$
f=i=o=\sigma(z)
$$





$$
\widetilde C=\tanh(z)
$$





$$
C_{\mathrm{new}}=fC_{\mathrm{old}}+i\widetilde C
$$





$$
h_{\mathrm{new}}=o\tanh(C_{\mathrm{new}})
$$



**Yaad rakhne ka cue:** Gates sigmoid; candidate tanh; cell mein add; hidden pe output gate.

### Solution — step by step

**Step 1: calculate the shared gate value.**



$$
\sigma(0)=\frac1{1+e^{-0}}\approx0.500000000
$$



**Step 2: calculate the candidate.**



$$
\widetilde C=\tanh(0)\approx0.000000000
$$



**Step 3: retain the old-cell part.**



$$
fC_{\mathrm{old}}=(0.500000000)(1)\approx0.500000000
$$



**Step 4: admit the new content.**



$$
i\widetilde C=(0.500000000)(0.000000000)\approx0.000000000
$$



**Step 5: add the memory contributions.**



$$
C_{\mathrm{new}}=0.500000000+(0.000000000)\approx0.500000000
$$



**Step 6: apply tanh to the cell.**



$$
\tanh(C_{\mathrm{new}})\approx0.462117157
$$



**Step 7: apply the output gate.**



$$
h_{\mathrm{new}}=(0.500000000)(0.462117157)\approx0.231058579
$$



**Final answer:** f=i=o≈0.500000000; candidate≈0.000000000; cell≈0.500000000; hidden≈0.231058579.

### LSTM gates from raw score 1

**Question. An LSTM has forget, input and output gate raw scores all equal to 1, candidate raw score 1, and old cell -0.5. Calculate all gates, candidate, new cell and hidden state.**

### Given

Raw scores z\_f = z\_i = z\_o = z\_candidate = 1.

Old cell C\_old = -0.5; gate activation sigmoid; candidate activation tanh.

### To find

f,i,o,candidate,C\_new,h\_new.

### Formula — which one and why

Raw gate scores require sigmoid first. The candidate uses tanh; do not apply sigmoid to it.

f retains old memory, i admits candidate, o controls exposed hidden output; C is cell memory.



$$
f=i=o=\sigma(z)
$$





$$
\widetilde C=\tanh(z)
$$





$$
C_{\mathrm{new}}=fC_{\mathrm{old}}+i\widetilde C
$$





$$
h_{\mathrm{new}}=o\tanh(C_{\mathrm{new}})
$$



**Yaad rakhne ka cue:** Gates sigmoid; candidate tanh; cell mein add; hidden pe output gate.

### Solution — step by step

**Step 1: calculate the shared gate value.**



$$
\sigma(1)=\frac1{1+e^{-1}}\approx0.731058579
$$



**Step 2: calculate the candidate.**



$$
\widetilde C=\tanh(1)\approx0.761594156
$$



**Step 3: retain the old-cell part.**



$$
fC_{\mathrm{old}}=(0.731058579)(-0.5)\approx-0.365529289
$$



**Step 4: admit the new content.**



$$
i\widetilde C=(0.731058579)(0.761594156)\approx0.556769941
$$



**Step 5: add the memory contributions.**



$$
C_{\mathrm{new}}=-0.365529289+(0.556769941)\approx0.191240652
$$



**Step 6: apply tanh to the cell.**



$$
\tanh(C_{\mathrm{new}})\approx0.188942847
$$



**Step 7: apply the output gate.**



$$
h_{\mathrm{new}}=(0.731058579)(0.188942847)\approx0.138128289
$$



**Final answer:** f=i=o≈0.731058579; candidate≈0.761594156; cell≈0.191240652; hidden≈0.138128289.

## Drill 43 Calculate gates before the cell

## Question

**Question.** Gate preactivations zf = zi = zo = 0, candidate preactivation 0.5, Cprevious = 0.2. Find gates, C, h.

## Answer — step by step

### Given

Raw gate scores: zf = zi = zo = 0.

Raw candidate score = 0.5; old cell C = 0.2.

### To find

Activate gates/candidate, then calculate the new cell and hidden state.

### Formula — which one and why

**Activate raw gate scores first.** A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations.

z\_f,z\_i,z\_o = gate preactivations; z\_c = candidate preactivation. An already activated gate must not be passed through sigmoid again.



$$
f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o)
$$





$$
\widetilde C=\tanh(z_c)
$$



**Yaad rakhne ka cue:** Raw score pehle activate; activated gate ko dobara sigmoid nahi.

**LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.

f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.



$$
C=f\odot C_{\mathrm{old}}+i\odot\widetilde C
$$





$$
h=o\odot\tanh C
$$



**Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate.

### Solution — step by step

### Meaning and formulas

The question supplies raw gate scores, called preactivations. Convert them to gates using sigmoid first. Convert the candidate's score using tanh.



$$
f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o),\quad \widetilde C=\tanh(z_C)
$$





$$
\sigma(z)=\frac1{1+e^{-z}},\quad C=fC_{\mathrm{previous}}+i\widetilde C,\quad h=o\tanh(C)
$$



### Step 1: turn each zero score into a gate



$$
\sigma(0)=\frac1{1+e^0}=\frac12=0.5
$$



Thus f = i = o = 0.5. A zero score is not a zero gate.

### Step 2: calculate candidate content



$$
\widetilde C=\tanh(0.5)=0.462117157
$$



### Step 3: combine retained memory and new content



$$
fC_{\mathrm{previous}}=0.5\times0.2=0.1
$$





$$
i\widetilde C=0.5\times0.462117157=0.231058579
$$





$$
C=0.1+0.231058579=0.331058579
$$



### Step 4: compute the visible hidden state



$$
\tanh(C)=\tanh(0.331058579)\approx0.319471636
$$





$$
h=0.5\times0.319471636=0.159735818
$$



**Final answer:** gates 0.5 each; candidate ≈ 0.462117157; C ≈ 0.331058579; h ≈ 0.159735818.