# Drill 45 BPTT shared weights

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s45

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s45/

## Question

**Question.** Linear recurrence hₜ = uxₜ+vhₜ₋₁+b, output h₂, target 1, half squared loss. u = 0.2, v = 0.5, b = 0, h₀ = 0, x₁ = 1, x₂ = 2, η = 0.1. Find both states and one update.

## Answer — step by step

### Given

Linear recurrence hₜ = uxₜ+vhₜ₋₁+b.

u = 0.2; v = 0.5; b = 0; h₀ = 0.

x₁ = 1; x₂ = 2; target for h₂ = 1.

Half squared loss; learning rate η = 0.1.

### To find

Find both states, all three shared-parameter gradients and one simultaneous update.

### Formula — which one and why

**Backpropagation through time.** Unroll the recurrence, send the loss derivative backward through its links and add contributions for each shared parameter before updating it once.

For h\_t=ux\_t+vh\_(t−1)+b with loss only at the last state: δ\_t = loss derivative at that state. The linear recurrence has activation slope 1.



$$
\delta_T=h_T-y
$$





$$
\delta_t=v\delta_{t+1}
$$





$$
g_u=\sum_t\delta_tx_t
$$





$$
g_v=\sum_t\delta_th_{t-1}
$$





$$
g_b=\sum_t\delta_t
$$



**Yaad rakhne ka cue:** Time ke har use ka gradient jodo; shared weight ko ek baar update karo.

**Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.

θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.



$$
g_t=J\prime(\theta_{t-1})
$$





$$
\theta_t=\theta_{t-1}-\eta g_t
$$



**Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient.

### Solution — step by step

### Meaning and formulas

The same u, v and b appear at both times. u multiplies the input, v multiplies the previous state, and b is a shared bias. The loss is only at the final state h₂. Because the recurrence is linear, its local activation derivative is 1.



$$
h_t=ux_t+vh_{t-1}+b,\qquad L=\frac12(h_2-y)^2
$$





$$
\delta_2=h_2-y,\qquad \delta_1=v\delta_2
$$



Each shared parameter receives a contribution from both of its uses.



$$
\frac{\partial L}{\partial u}=\delta_1x_1+\delta_2x_2
$$





$$
\frac{\partial L}{\partial v}=\delta_1h_0+\delta_2h_1,\qquad \frac{\partial L}{\partial b}=\delta_1+\delta_2
$$



### Step 1: forward through time 1



$$
h_1=0.2(1)+0.5(0)+0=0.2
$$



### Step 2: forward through time 2



$$
h_2=0.2(2)+0.5(0.2)+0=0.4+0.1=0.5
$$





$$
L=\frac12(0.5-1)^2=0.125
$$



### Step 3: carry the gradient backwards



$$
\delta_2=0.5-1=-0.5
$$





$$
\delta_1=0.5(-0.5)=-0.25
$$



### Step 4: add contributions for each shared parameter



$$
\frac{\partial L}{\partial u}=(-0.25)(1)+(-0.5)(2)=-0.25-1=-1.25
$$





$$
\frac{\partial L}{\partial v}=(-0.25)(0)+(-0.5)(0.2)=0-0.1=-0.1
$$





$$
\frac{\partial L}{\partial b}=-0.25-0.5=-0.75
$$



### Step 5: update once after summing



$$
u_{\mathrm{new}}=0.2-0.1(-1.25)=0.325
$$





$$
v_{\mathrm{new}}=0.5-0.1(-0.1)=0.51
$$





$$
b_{\mathrm{new}}=0-0.1(-0.75)=0.075
$$



**Final answer:** states 0.2 and 0.5; new parameters u = 0.325, v = 0.51, b = 0.075.