Deep Learning · TEE · section

Drill 30 Regularized objective

Numerical practice from direct to mixed

Question

Question. Data loss = 0.4, w = [−2,1,0.5], λ = 0.1. Calculate L1 objective and L2 objective using (λ/2)Σw².

Answer — step by step

Given

Data loss = 0.4; weights = [−2,1,0.5]; λ = 0.1.

For L2, the question includes the factor ½.

To find

Find each penalty and the total L1/L2 objective.

Formula — which one and why

L1 penalty and gradient. Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.

λ (lambda) = penalty strength; J_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.

J=J_{\mathrm{data}}+\lambda\sum_i|w_i|LaTeX: J=J_{\mathrm{data}}+\lambda\sum_i|w_i|
g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i)LaTeX: g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i)

Yaad rakhne ka cue: L1 absolute values leta hai; gradient mein sign aata hai.

L2 penalty with a factor of one-half. Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw.

λ = penalty strength; w = old weight; J_data and g_data = data loss and its gradient.

J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2LaTeX: J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2
g_i=g_{\mathrm{data},i}+\lambda w_iLaTeX: g_i=g_{\mathrm{data},i}+\lambda w_i
w_i^{\mathrm{new}}=w_i-\eta g_iLaTeX: w_i^{\mathrm{new}}=w_i-\eta g_i

Yaad rakhne ka cue: Half wala L2 derivative mein lambda × weight deta hai.

Solution — step by step

Meaning and formulas

The data loss measures prediction error. A regularization penalty adds a cost for large weights. The objective J is the total being minimized. λ sets the strength of the penalty.

J_{L1}=L_{\mathrm{data}}+\lambda\sum_i|w_i|LaTeX: J_{L1}=L_{\mathrm{data}}+\lambda\sum_i|w_i|
J_{L2}=L_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2LaTeX: J_{L2}=L_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2

Absolute value means ignore the sign. Squaring means multiply the number by itself. These are different operations.

Step 1: Part A: L1 objective.

|-2|=2,\quad |1|=1,\quad |0.5|=0.5LaTeX: |-2|=2,\quad |1|=1,\quad |0.5|=0.5
\sum_i|w_i|=2+1+0.5=3.5LaTeX: \sum_i|w_i|=2+1+0.5=3.5
\text{L1 penalty}=0.1\times3.5=0.35LaTeX: \text{L1 penalty}=0.1\times3.5=0.35
J_{L1}=0.4+0.35=0.75LaTeX: J_{L1}=0.4+0.35=0.75

Step 2: Part B: L2 objective with the given half.

(-2)^2=4,\quad 1^2=1,\quad 0.5^2=0.25LaTeX: (-2)^2=4,\quad 1^2=1,\quad 0.5^2=0.25
\sum_iw_i^2=4+1+0.25=5.25LaTeX: \sum_iw_i^2=4+1+0.25=5.25
\frac\lambda2=\frac{0.1}2=0.05LaTeX: \frac\lambda2=\frac{0.1}2=0.05
\text{L2 penalty}=0.05\times5.25=0.2625LaTeX: \text{L2 penalty}=0.05\times5.25=0.2625
J_{L2}=0.4+0.2625=0.6625LaTeX: J_{L2}=0.4+0.2625=0.6625

Final answer: L1 objective 0.75; L2 objective 0.6625. These are total objectives, not just penalty values.