Deep Learning · TEE · section

Model answer: regularization and the L1–L2 difference

Regularization and generalization

Question

Question. What is regularization? Compare L1 and L2 regularization, including the objective, gradient effect and influence of regularization strength.

Answer — step by step

Regularization restricts or discourages certain fitted solutions to improve generalization. Weight penalties add a cost for parameter size to the training objective. The optimizer must balance fitting the data against that extra cost.

Follow the stated penalty convention, including lambda and any half factor, when you form the objective. Tap the diagram to enlarge.

Objectives used in this course

$J_{L1}=J_{\mathrm{data}}+\lambda\sum_i|w_i|$ LaTeX: J_{L1}=J_{\mathrm{data}}+\lambda\sum_i|w_i|

$J_{L2}=J_{\mathrm{data}}+\lambda\sum_iw_i^2$ LaTeX: J_{L2}=J_{\mathrm{data}}+\lambda\sum_iw_i^2

Jdata measures prediction error. λ controls the strength of the additional penalty. The sums are over the weights included by the stated question.

Basis	L1	L2
Quantity penalized	Absolute weight magnitude	Squared weight magnitude
Penalty derivative at nonzero w	λ sign(w)	2λw under the displayed convention
Typical effect	Encourages sparse solutions	Smoothly shrinks weight magnitudes
Zero weights	Can promote exact zeros with suitable optimization	Usually leaves small nonzero weights

The total gradient is the data-loss gradient plus the penalty gradient. With w = 2, data gradient 0.3 and λ = 0.1, the displayed L2 penalty adds 2 × 0.1 × 2 = 0.4. The total gradient is 0.7; update using that total.

Increasing λ strengthens the preference for smaller weights. Excessive regularization can prevent learning useful patterns and cause underfitting. L1 is not differentiable at zero, where a subgradient or suitable optimization method is used. If the examination writes λ/2 before the L2 sum, its penalty gradient becomes λw. Derive the factor from the printed objective.