Deep Learning · TEE · section

Drill 47 Mixed backpropagation and L2

Numerical practice from direct to mixed

Question

Question. Single linear neuron ŷ = wx, x = 2, w = 0.4, y = 1. Objective J = ½(ŷ−y)²+(λ/2)w², λ = 0.1, η = 0.2. Find loss, gradient, new weight.

Answer — step by step

Given

Prediction ŷ = wx; x = 2; w = 0.4; target y = 1.

Data loss = ½(ŷ−y)²; penalty = (λ/2)w².

λ = 0.1; η = 0.2.

To find

Find total objective, its weight gradient and the new weight.

Formula — which one and why

L2 penalty with a factor of one-half. Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw.

λ = penalty strength; w = old weight; J_data and g_data = data loss and its gradient.

$J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2$ LaTeX: J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2

$g_i=g_{\mathrm{data},i}+\lambda w_i$ LaTeX: g_i=g_{\mathrm{data},i}+\lambda w_i

$w_i^{\mathrm{new}}=w_i-\eta g_i$ LaTeX: w_i^{\mathrm{new}}=w_i-\eta g_i

Yaad rakhne ka cue: Half wala L2 derivative mein lambda × weight deta hai.

Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.

θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.

$g_t=J\prime(\theta_{t-1})$ LaTeX: g_t=J\prime(\theta_{t-1})

$\theta_t=\theta_{t-1}-\eta g_t$ LaTeX: \theta_t=\theta_{t-1}-\eta g_t

Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.

Solution — step by step

Meaning and formulas

This objective contains both prediction error and a weight penalty. Its gradient also has two parts. Compute both at the old weight, add them, then update once.