# Reading recurrent memory and gates

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s3

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s3/

### Time means position in the sequence

t = 1 is the first input; t = 2 is the second. The state h₀ is the starting memory, often given as zero. At each step the input changes and the state changes, while the learned weights are reused.

![A gate is a multiplier. Its value controls how much of a particular contribution gets through.](https://deep-learning-study-room.vercel.app/assets/figures/v_gate.svg)

Diagram: A gate is a multiplier. Its value controls how much of a particular contribution gets through.. Labels: A gate multiplies the value passing through it | A gate multiplies the value passing through it | Example old value = 8. Compare three gate settings. | Gate = 0 | 8 × 0 = 0 | Pass none. | Gate = 0.25 | 8 × 0.25 = 2 | Pass one quarter. | Gate = 1 | 8 × 1 = 8 | Pass all. | Sigmoid gates lie between 0 and 1. A negative candidate can still contribute a | negative value.

A gate is a multiplier. Its value controls how much of a particular contribution gets through. Tap the diagram to enlarge.

### Raw score and activated state

First multiply and add to form a raw score. Then apply tanh or the activation specified in the question. The activated result is the hidden state passed to the next time step.



$$
a_t=W_{xh}x_t+W_{hh}h_{t-1}+b_h,\qquad h_t=\tanh(a_t)
$$



Wxh means the weight mapping input x to hidden state h. Whh maps the previous hidden state to the new hidden calculation. These labels describe a connection; they do not mean multiply the letters together.

### What a gate does

A gate is a multiplier controlling how much of a value passes through. Gate 0 keeps none of that contribution; gate 1 keeps all; gate 0.25 keeps a quarter. For example, 0.25 × 8 = 2. Sigmoid-generated gates lie between 0 and 1 for finite raw scores.

### A gate is different from its raw score

If a gate's raw score is 0, sigmoid gives 1/(1 + 1) = 0.5. A supplied gate value 0, however, is already zero and must not be passed through sigmoid again. Read whether the question gives **preactivations** or **gate outputs**.

### Memory values can be negative

The gate may be nonnegative while the value being gated is negative. Multiplying 0.2 by −0.5 gives −0.1. This is normal: the gate chooses an amount, and the candidate content supplies its sign.