Deep Learning · TEE · section

Drill 31 L1 update on positive and negative weights

Numerical practice from direct to mixed

Question

Question. w = [−0.5,0.3], data gradients = [0.2,−0.1], λ = 0.1, η = 0.2. Use L1 and ordinary gradient descent away from zero.

Answer — step by step

Given

Old weights = [−0.5,0.3].

Data gradients = [0.2,−0.1]; λ = 0.1; η = 0.2.

To find

Find total gradients and the two updated weights.

Formula — which one and why

L1 penalty and gradient. Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.

λ (lambda) = penalty strength; J_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.

J=J_{\mathrm{data}}+\lambda\sum_i|w_i|LaTeX: J=J_{\mathrm{data}}+\lambda\sum_i|w_i|
g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i)LaTeX: g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i)

Yaad rakhne ka cue: L1 absolute values leta hai; gradient mein sign aata hai.

Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.

θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.

g_t=J\prime(\theta_{t-1})LaTeX: g_t=J\prime(\theta_{t-1})
\theta_t=\theta_{t-1}-\eta g_tLaTeX: \theta_t=\theta_{t-1}-\eta g_t

Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.

Solution — step by step

Meaning and formula

The supplied data gradient is only one part of the total gradient. L1 adds λ times the sign of the old weight. Sign is −1 for a negative weight and +1 for a positive weight.

g_{j,\mathrm{total}}=g_{j,\mathrm{data}}+\lambda\operatorname{sign}(w_j)LaTeX: g_{j,\mathrm{total}}=g_{j,\mathrm{data}}+\lambda\operatorname{sign}(w_j)
w_{j,\mathrm{new}}=w_j-\eta g_{j,\mathrm{total}}LaTeX: w_{j,\mathrm{new}}=w_j-\eta g_{j,\mathrm{total}}

Both weights here are nonzero, so the ambiguous derivative of absolute value at zero is not needed.

Step 1: Weight 1: old value −0.5.

\lambda\operatorname{sign}(-0.5)=0.1(-1)=-0.1LaTeX: \lambda\operatorname{sign}(-0.5)=0.1(-1)=-0.1
g_{1,\mathrm{total}}=0.2+(-0.1)=0.1LaTeX: g_{1,\mathrm{total}}=0.2+(-0.1)=0.1
w_{1,\mathrm{new}}=-0.5-0.2(0.1)=-0.5-0.02=-0.52LaTeX: w_{1,\mathrm{new}}=-0.5-0.2(0.1)=-0.5-0.02=-0.52

Step 2: Weight 2: old value 0.3.

\lambda\operatorname{sign}(0.3)=0.1(1)=0.1LaTeX: \lambda\operatorname{sign}(0.3)=0.1(1)=0.1
g_{2,\mathrm{total}}=-0.1+0.1=0LaTeX: g_{2,\mathrm{total}}=-0.1+0.1=0
w_{2,\mathrm{new}}=0.3-0.2(0)=0.3LaTeX: w_{2,\mathrm{new}}=0.3-0.2(0)=0.3

Final answer: [−0.52, 0.3]. L1's own contribution pulls toward zero, but the combined data-plus-penalty update can move a weight away from zero.