Deep Learning · TEE · section

Vanishing and exploding gradients

Recurrent neural networks

Definition. Vanishing and exploding gradients describe the attenuation or amplification of backward derivatives through repeated Jacobian products. In recurrent networks, these products extend across time and affect long-range credit assignment.

Explanation and working

In BPTT, the gradient repeatedly multiplies recurrent weights and activation derivatives. Products of many small factors can approach zero; early information then receives very little learning signal. Large repeated factors can make the gradient explode. These are about training signals, not an instruction that every hidden-state value must become zero or infinity.

Follow the repeated products. Small factors can weaken the learning signal; large products can destabilize updates. Tap the diagram to enlarge.

Numerical intuition: 0.5¹⁰ = 0.0009765625, whereas 2¹⁰ = 1,024. Sigmoid's derivative is at most 0.25 and becomes even smaller in saturation. Tanh also saturates for large magnitude inputs.

Useful remedies include LSTM or GRU gates and cell paths, suitable initialization and learning rate, and gradient clipping for explosion. Clipping does not by itself solve vanishing gradients. LSTM makes long-range learning easier; it does not guarantee perfect unlimited memory.

Why the effect grows with sequence length

For a vanilla tanh RNN, each hidden-state transition contributes a Jacobian. The derivative through several transitions contains a product of these Jacobians:

$\frac{\partial h_t}{\partial h_{t-1}}=\operatorname{diag}(1-h_t^2)W_{hh}$ LaTeX: \frac{\partial h_t}{\partial h_{t-1}}=\operatorname{diag}(1-h_t^2)W_{hh}

$\frac{\partial h_T}{\partial h_t}=\prod_{k=T}^{t+1}\frac{\partial h_k}{\partial h_{k-1}}$ LaTeX: \frac{\partial h_T}{\partial h_t}=\prod_{k=T}^{t+1}\frac{\partial h_k}{\partial h_{k-1}}

The product is ordered from the latest transition toward the earliest. Small activation derivatives and recurrent transformations can attenuate the signal; repeated amplification can make it grow. A one-dimensional product illustrates the issue, but a vector RNN depends on the full matrix products, not one weight alone.

Gradient norm clipping: a complete numerical

Norm clipping rescales an oversized gradient while retaining its direction. It limits the update size caused by an exploding gradient.

$g_{\mathrm{clipped}}=g\min\left(1,\frac{\tau}{\lVert g\rVert_2}\right)$ LaTeX: g_{\mathrm{clipped}}=g\min\left(1,\frac{\tau}{\lVert g\rVert_2}\right)

If g is the zero vector, leave it unchanged. The threshold τ is positive. This is global norm clipping for the vector being considered; elementwise clipping is a different operation.

Question. A gradient is g = [6, 8]ᵀ. Clip its Euclidean norm to at most τ = 5, then update w = [1, 1]ᵀ with learning rate η = 0.1.

Given

Gradient g = [6,8]ᵀ; norm threshold τ = 5.

Old weights w = [1,1]ᵀ; η = 0.1.

To find

Find the clipped gradient and updated weights.

Formula — which one and why

Gradient norm clipping. If the vector norm exceeds the allowed threshold, multiply every component by the same scale. This changes magnitude while preserving direction.

g = gradient vector; τ (tau) = maximum norm; ||g||₂ = square root of the sum of squared components; η = learning rate.

$\lVert g\rVert_2=\sqrt{\sum_i g_i^2}$ LaTeX: \lVert g\rVert_2=\sqrt{\sum_i g_i^2}

$g_{\mathrm{clip}}=g\min\left(1,\frac\tau{\lVert g\rVert_2}\right)$ LaTeX: g_{\mathrm{clip}}=g\min\left(1,\frac\tau{\lVert g\rVert_2}\right)

$w_{\mathrm{new}}=w-\eta g_{\mathrm{clip}}$ LaTeX: w_{\mathrm{new}}=w-\eta g_{\mathrm{clip}}

Yaad rakhne ka cue: Bada gradient? Sab components ko ek hi factor se chhota karo.

Solution — step by step

Step 1: calculate the original norm.

Step 2: find the scale. Since 10 > 5, scale the vector by 5/10 = 0.5.

$g_{\mathrm{clipped}}=0.5\begin{bmatrix}6\\8\end{bmatrix}=\begin{bmatrix}3\\4\end{bmatrix}$ LaTeX: g_{\mathrm{clipped}}=0.5\begin{bmatrix}6\\8\end{bmatrix}=\begin{bmatrix}3\\4\end{bmatrix}

Step 3: verify the clipped norm.

$\sqrt{3^2+4^2}=\sqrt{9+16}=5$ LaTeX: \sqrt{3^2+4^2}=\sqrt{9+16}=5

Step 4: update each parameter with the clipped gradient.

$w_{1,\mathrm{new}}=1-0.1(3)=0.7$ LaTeX: w_{1,\mathrm{new}}=1-0.1(3)=0.7

$w_{2,\mathrm{new}}=1-0.1(4)=0.6$ LaTeX: w_{2,\mathrm{new}}=1-0.1(4)=0.6

Final answer: clipped gradient [3, 4]ᵀ and new weights [0.7, 0.6]ᵀ. Clipping does not increase a vanishing gradient and does not replace the need for an appropriate recurrent architecture.