# Drill 29 L1 update on positive and negative weights

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s29

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s29/

## Question

**Question.** w = [−0.5,0.3], data gradients = [0.2,−0.1], λ = 0.1, η = 0.2. Use L1 and ordinary gradient descent away from zero.

## Answer — step by step

### Given

Old weights = [−0.5,0.3].

Data gradients = [0.2,−0.1]; λ = 0.1; η = 0.2.

### To find

Find total gradients and the two updated weights.

### Formula — which one and why

**L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.

λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.



$$
J=J_{\mathrm{data}}+\lambda\sum_i|w_i|
$$





$$
g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i)
$$



**Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai.

**Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.

θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.



$$
g_t=J\prime(\theta_{t-1})
$$





$$
\theta_t=\theta_{t-1}-\eta g_t
$$



**Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient.

### Solution — step by step

### Meaning and formula

The supplied data gradient is only one part of the total gradient. L1 adds λ times the sign of the old weight. Sign is −1 for a negative weight and +1 for a positive weight.



$$
g_{j,\mathrm{total}}=g_{j,\mathrm{data}}+\lambda\operatorname{sign}(w_j)
$$





$$
w_{j,\mathrm{new}}=w_j-\eta g_{j,\mathrm{total}}
$$



Both weights here are nonzero, so the ambiguous derivative of absolute value at zero is not needed.

**Step 1: Weight 1: old value −0.5.**



$$
\lambda\operatorname{sign}(-0.5)=0.1(-1)=-0.1
$$





$$
g_{1,\mathrm{total}}=0.2+(-0.1)=0.1
$$





$$
w_{1,\mathrm{new}}=-0.5-0.2(0.1)=-0.5-0.02=-0.52
$$



**Step 2: Weight 2: old value 0.3.**



$$
\lambda\operatorname{sign}(0.3)=0.1(1)=0.1
$$





$$
g_{2,\mathrm{total}}=-0.1+0.1=0
$$





$$
w_{2,\mathrm{new}}=0.3-0.2(0)=0.3
$$



**Final answer: [−0.52, 0.3].** L1's own contribution pulls toward zero, but the combined data-plus-penalty update can move a weight away from zero.