# Model answer: regularization and the L1–L2 difference

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/6/tee-c6-s2

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c6-s2/

## Question

**Question.** What is regularization? Compare L1 and L2 regularization, including the objective, gradient effect and influence of regularization strength.

## Answer — step by step

Regularization restricts or discourages certain fitted solutions to improve generalization. Weight penalties add a cost for parameter size to the training objective. The optimizer must balance fitting the data against that extra cost.

![Follow the stated penalty convention, including lambda and any half factor, when you form the objective.](https://deep-learning-study-room.vercel.app/assets/figures/v_penalties.svg)

Diagram: Follow the stated penalty convention, including lambda and any half factor, when you form the objective.. Labels: A penalty adds a preference for smaller weights | A penalty adds a preference for smaller weights | Example weights: −2 and 1. The loss convention sets any λ or ½ factors. | L1: absolute values | |−2| + |1| = 2 + 1 = 3 | Linear cost with magnitude | Can promote sparse weights. | L2: squared values | (−2)² + 1² = 4 + 1 = 5 | Bigger weights cost more | Smooth shrinkage toward zero. | The regularized objective adds the data loss and the stated penalty.

Follow the stated penalty convention, including lambda and any half factor, when you form the objective. Tap the diagram to enlarge.

### Objectives used in this course



$$
J_{L1}=J_{\mathrm{data}}+\lambda\sum_i|w_i|
$$





$$
J_{L2}=J_{\mathrm{data}}+\lambda\sum_iw_i^2
$$



Jdata measures prediction error. λ controls the strength of the additional penalty. The sums are over the weights included by the stated question.

| Basis | L1 | L2 |
| --- | --- | --- |
| Quantity penalized | Absolute weight magnitude | Squared weight magnitude |
| Penalty derivative at nonzero w | λ sign(w) | 2λw under the displayed convention |
| Typical effect | Encourages sparse solutions | Smoothly shrinks weight magnitudes |
| Zero weights | Can promote exact zeros with suitable optimization | Usually leaves small nonzero weights |

The total gradient is the data-loss gradient plus the penalty gradient. With w = 2, data gradient 0.3 and λ = 0.1, the displayed L2 penalty adds 2 × 0.1 × 2 = 0.4. The total gradient is 0.7; update using that total.

Increasing λ strengthens the preference for smaller weights. Excessive regularization can prevent learning useful patterns and cause underfitting. L1 is not differentiable at zero, where a subgradient or suitable optimization method is used. If the examination writes λ/2 before the L2 sum, its penalty gradient becomes λw. Derive the factor from the printed objective.