Deep Learning · TEE · section

Reading recurrent memory and gates

Recurrent neural networks

Time means position in the sequence

t = 1 is the first input; t = 2 is the second. The state h₀ is the starting memory, often given as zero. At each step the input changes and the state changes, while the learned weights are reused.

Diagram: A gate is a multiplier. Its value controls how much of a particular contribution gets through.. Labels: A gate multiplies the value passing through it | A gate multiplies the value passing through it | Example old value = 8. Compare three gate settings. | Gate = 0 | 8 × 0 = 0 | Pass none. | Gate = 0.25 | 8 × 0.25 = 2 | Pass one quarter. | Gate = 1 | 8 × 1 = 8 | Pass all. | Sigmoid gates lie between 0 and 1. A negative candidate can still contribute a | negative value.

Raw score and activated state

First multiply and add to form a raw score. Then apply tanh or the activation specified in the question. The activated result is the hidden state passed to the next time step.

LaTeX: a_t=W_{xh}x_t+W_{hh}h_{t-1}+b_h,\qquad h_t=\tanh(a_t)

Wxh means the weight mapping input x to hidden state h. Whh maps the previous hidden state to the new hidden calculation. These labels describe a connection; they do not mean multiply the letters together.

What a gate does

A gate is a multiplier controlling how much of a value passes through. Gate 0 keeps none of that contribution; gate 1 keeps all; gate 0.25 keeps a quarter. For example, 0.25 × 8 = 2. Sigmoid-generated gates lie between 0 and 1 for finite raw scores.

A gate is different from its raw score

If a gate's raw score is 0, sigmoid gives 1/(1 + 1) = 0.5. A supplied gate value 0, however, is already zero and must not be passed through sigmoid again. Read whether the question gives preactivations or gate outputs.

Memory values can be negative

The gate may be nonnegative while the value being gated is negative. Multiplying 0.2 by −0.5 gives −0.1. This is normal: the gate chooses an amount, and the candidate content supplies its sign.