Deep Learning · TEE · section

How an RNN is trained

Recurrent neural networks

Definition. Backpropagation through time (BPTT) differentiates the loss of an RNN by applying backpropagation to its temporally unrolled computational graph and summing contributions to shared parameters.

Explanation and working

Unroll the recurrent network for the sequence length. Run the forward calculation in time order. Calculate the specified loss. Differentiate backwards through the unrolled graph. Add contributions to each shared weight from every time it was used. Update each shared weight once for the selected sequence or batch. This is backpropagation through time, or BPTT.

Diagram: Unroll forward through time, then accumulate backward contributions for the shared parameters.. Labels: One shared weight receives contributions from many times | One shared weight receives contributions from many times | Unrolling draws repeated uses of one cell, not independent copies of its | weights. | Time 1 | Same weights W | Time 2 | Same weights W | Time 3 | Same weights W | Gradients flow backward through the state dependencies | Add shared-weight contributions | Total gradient = contribution 1 + contribution 2 + | contribution 3 | After summing the required contributions, update each shared parameter once.

For tanh, let δₜ mean the gradient with respect to the hidden preactivation. The following form allows losses at several time steps; a missing output loss contributes zero.

LaTeX: \delta_t=\left(W_{hy}^{T}\delta_t^{out}+W_{hh}^{T}\delta_{t+1}\right)\odot(1-h_t^2)

LaTeX: \frac{\partial L}{\partial W_{hh}}=\sum_t\delta_t h_{t-1}^{T},\quad \frac{\partial L}{\partial W_{xh}}=\sum_t\delta_t x_t^{T},\quad \frac{\partial L}{\partial b_h}=\sum_t\delta_t

The symbol ⊙ means pairwise multiplication. The transpose T turns a column into a row. δnext carries the effect of future time steps. The summation is essential because these are shared parameters.

One BPTT delta at h = 0.5

Question. In a scalar tanh RNN, h_t = 0.5, current output-path derivative = 0.2, next-step delta = 0.3 and recurrent weight = 0.4. Find the current preactivation delta.

Given

Activated h_t = 0.5; q_t = 0.2; δ_(t+1) = 0.3; recurrent weight u = 0.4.

To find

Current preactivation delta δ_t.

Formula — which one and why

Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh.

q_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation.

LaTeX: \delta_t=(q_t+u\delta_{t+1})(1-h_t^2)

Yaad rakhne ka cue: Future aur current dono add; tanh slope uske baad.

Solution — step by step

Step 1: calculate the future path contribution.

LaTeX: 0.4(0.3)=0.12

Step 2: add the current output contribution.

LaTeX: 0.2+0.12=0.32

Step 3: calculate the current tanh slope.

LaTeX: 1-(0.5)^2=0.75

Step 4: multiply by that slope.

LaTeX: \delta_t=(0.32)(0.75)=0.24

Final answer: δ_t = 0.24.

One BPTT delta at h = -0.2

Question. In a scalar tanh RNN, h_t = -0.2, current output-path derivative = 0.1, next-step delta = 0 and recurrent weight = 0.5. Find the current preactivation delta.

Given

Activated h_t = -0.2; q_t = 0.1; δ_(t+1) = 0; recurrent weight u = 0.5.

To find

Current preactivation delta δ_t.

Formula — which one and why

Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh.

q_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation.

LaTeX: \delta_t=(q_t+u\delta_{t+1})(1-h_t^2)

Yaad rakhne ka cue: Future aur current dono add; tanh slope uske baad.

Solution — step by step

Step 1: calculate the future path contribution.

LaTeX: 0.5(0)=0

Step 2: add the current output contribution.

LaTeX: 0.1+0=0.1

Step 3: calculate the current tanh slope.

LaTeX: 1-(-0.2)^2=0.96

Step 4: multiply by that slope.

LaTeX: \delta_t=(0.1)(0.96)=0.096

Final answer: δ_t = 0.096.