Deep Learning · TEE · formulas

BPTT hidden delta

At an RNN hidden state, a loss gradient can arrive from the current output and from the next time step. Add these two contributions, then multiply by the local activation derivative. The displayed output term p−y assumes softmax with categorical cross-entropy.

$\delta_t=\left[W_{hy}^T(p_t-y_t)+W_{hh}^T\delta_{t+1}\right]\odot(1-h_t^2)$ LaTeX: \delta_t=\left[W_{hy}^T(p_t-y_t)+W_{hh}^T\delta_{t+1}\right]\odot(1-h_t^2)

How an RNN is trained

Definition. Backpropagation through time (BPTT) differentiates the loss of an RNN by applying backpropagation to its temporally unrolled computational graph and summing contributions to shared parameters.

Explanation and working

Unroll the recurrent network for the sequence length. Run the forward calculation in time order. Calculate the specified loss. Differentiate backwards through the unrolled graph. Add contributions to each shared weight from every time it was used. Update each shared weight once for the selected sequence or batch. This is backpropagation through time, or BPTT.

Unroll forward through time, then accumulate backward contributions for the shared parameters. Tap the diagram to enlarge.

For tanh, let δₜ mean the gradient with respect to the hidden preactivation. The following form allows losses at several time steps; a missing output loss contributes zero.

$\delta_t=\left(W_{hy}^{T}\delta_t^{out}+W_{hh}^{T}\delta_{t+1}\right)\odot(1-h_t^2)$ LaTeX: \delta_t=\left(W_{hy}^{T}\delta_t^{out}+W_{hh}^{T}\delta_{t+1}\right)\odot(1-h_t^2)

The symbol ⊙ means pairwise multiplication. The transpose T turns a column into a row. δnext carries the effect of future time steps. The summation is essential because these are shared parameters.

One BPTT delta at h = 0.5

Question. In a scalar tanh RNN, h_t = 0.5, current output-path derivative = 0.2, next-step delta = 0.3 and recurrent weight = 0.4. Find the current preactivation delta.

Given

Activated h_t = 0.5; q_t = 0.2; δ_(t+1) = 0.3; recurrent weight u = 0.4.

To find

Current preactivation delta δ_t.

Formula — which one and why

Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh.

q_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation.

$\delta_t=(q_t+u\delta_{t+1})(1-h_t^2)$ LaTeX: \delta_t=(q_t+u\delta_{t+1})(1-h_t^2)

Yaad rakhne ka cue: Future aur current dono add; tanh slope uske baad.

Solution — step by step

Step 1: calculate the future path contribution.

$0.4(0.3)=0.12$ LaTeX: 0.4(0.3)=0.12

Step 2: add the current output contribution.

$0.2+0.12=0.32$ LaTeX: 0.2+0.12=0.32

Step 3: calculate the current tanh slope.

$1-(0.5)^2=0.75$ LaTeX: 1-(0.5)^2=0.75

Step 4: multiply by that slope.

$\delta_t=(0.32)(0.75)=0.24$ LaTeX: \delta_t=(0.32)(0.75)=0.24

Final answer: δ_t = 0.24.

One BPTT delta at h = -0.2

Question. In a scalar tanh RNN, h_t = -0.2, current output-path derivative = 0.1, next-step delta = 0 and recurrent weight = 0.5. Find the current preactivation delta.