Deep Learning · TEE · section

Drill 45 BPTT shared weights

Numerical practice from direct to mixed

Question

Question. Linear recurrence hₜ = uxₜ+vhₜ₋₁+b, output h₂, target 1, half squared loss. u = 0.2, v = 0.5, b = 0, h₀ = 0, x₁ = 1, x₂ = 2, η = 0.1. Find both states and one update.

Answer — step by step

Given

Linear recurrence hₜ = uxₜ+vhₜ₋₁+b.

u = 0.2; v = 0.5; b = 0; h₀ = 0.

x₁ = 1; x₂ = 2; target for h₂ = 1.

Half squared loss; learning rate η = 0.1.

To find

Find both states, all three shared-parameter gradients and one simultaneous update.

Formula — which one and why

Backpropagation through time. Unroll the recurrence, send the loss derivative backward through its links and add contributions for each shared parameter before updating it once.

For h_t=ux_t+vh_(t−1)+b with loss only at the last state: δ_t = loss derivative at that state. The linear recurrence has activation slope 1.

\delta_T=h_T-yLaTeX: \delta_T=h_T-y
\delta_t=v\delta_{t+1}LaTeX: \delta_t=v\delta_{t+1}
g_u=\sum_t\delta_tx_tLaTeX: g_u=\sum_t\delta_tx_t
g_v=\sum_t\delta_th_{t-1}LaTeX: g_v=\sum_t\delta_th_{t-1}
g_b=\sum_t\delta_tLaTeX: g_b=\sum_t\delta_t

Yaad rakhne ka cue: Time ke har use ka gradient jodo; shared weight ko ek baar update karo.

Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.

θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.

g_t=J\prime(\theta_{t-1})LaTeX: g_t=J\prime(\theta_{t-1})
\theta_t=\theta_{t-1}-\eta g_tLaTeX: \theta_t=\theta_{t-1}-\eta g_t

Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.

Solution — step by step

Meaning and formulas

The same u, v and b appear at both times. u multiplies the input, v multiplies the previous state, and b is a shared bias. The loss is only at the final state h₂. Because the recurrence is linear, its local activation derivative is 1.

h_t=ux_t+vh_{t-1}+b,\qquad L=\frac12(h_2-y)^2LaTeX: h_t=ux_t+vh_{t-1}+b,\qquad L=\frac12(h_2-y)^2
\delta_2=h_2-y,\qquad \delta_1=v\delta_2LaTeX: \delta_2=h_2-y,\qquad \delta_1=v\delta_2

Each shared parameter receives a contribution from both of its uses.

\frac{\partial L}{\partial u}=\delta_1x_1+\delta_2x_2LaTeX: \frac{\partial L}{\partial u}=\delta_1x_1+\delta_2x_2
\frac{\partial L}{\partial v}=\delta_1h_0+\delta_2h_1,\qquad \frac{\partial L}{\partial b}=\delta_1+\delta_2LaTeX: \frac{\partial L}{\partial v}=\delta_1h_0+\delta_2h_1,\qquad \frac{\partial L}{\partial b}=\delta_1+\delta_2

Step 1: forward through time 1

h_1=0.2(1)+0.5(0)+0=0.2LaTeX: h_1=0.2(1)+0.5(0)+0=0.2

Step 2: forward through time 2

h_2=0.2(2)+0.5(0.2)+0=0.4+0.1=0.5LaTeX: h_2=0.2(2)+0.5(0.2)+0=0.4+0.1=0.5
L=\frac12(0.5-1)^2=0.125LaTeX: L=\frac12(0.5-1)^2=0.125

Step 3: carry the gradient backwards

\delta_2=0.5-1=-0.5LaTeX: \delta_2=0.5-1=-0.5
\delta_1=0.5(-0.5)=-0.25LaTeX: \delta_1=0.5(-0.5)=-0.25

Step 4: add contributions for each shared parameter

\frac{\partial L}{\partial u}=(-0.25)(1)+(-0.5)(2)=-0.25-1=-1.25LaTeX: \frac{\partial L}{\partial u}=(-0.25)(1)+(-0.5)(2)=-0.25-1=-1.25
\frac{\partial L}{\partial v}=(-0.25)(0)+(-0.5)(0.2)=0-0.1=-0.1LaTeX: \frac{\partial L}{\partial v}=(-0.25)(0)+(-0.5)(0.2)=0-0.1=-0.1
\frac{\partial L}{\partial b}=-0.25-0.5=-0.75LaTeX: \frac{\partial L}{\partial b}=-0.25-0.5=-0.75

Step 5: update once after summing

u_{\mathrm{new}}=0.2-0.1(-1.25)=0.325LaTeX: u_{\mathrm{new}}=0.2-0.1(-1.25)=0.325
v_{\mathrm{new}}=0.5-0.1(-0.1)=0.51LaTeX: v_{\mathrm{new}}=0.5-0.1(-0.1)=0.51
b_{\mathrm{new}}=0-0.1(-0.75)=0.075LaTeX: b_{\mathrm{new}}=0-0.1(-0.75)=0.075

Final answer: states 0.2 and 0.5; new parameters u = 0.325, v = 0.51, b = 0.075.