Deep Learning · MT2 · formulas

Squared and absolute losses

SSE, half SSE and MSE start with the same errors but apply different final divisions. First understand what an error and its square mean. The full question below keeps all three answers separate.

$\mathrm{SSE}=\sum_i(o_i-y_i)^2$ LaTeX: \mathrm{SSE}=\sum_i(o_i-y_i)^2

$L_{\mathrm{half}}=\frac{\mathrm{SSE}}2$ LaTeX: L_{\mathrm{half}}=\frac{\mathrm{SSE}}2

$L_{\mathrm{MSE}}=\frac{\mathrm{SSE}}n$ LaTeX: L_{\mathrm{MSE}}=\frac{\mathrm{SSE}}n

$L_{\mathrm{MAE}}=\frac1n\sum_i|o_i-y_i|$ LaTeX: L_{\mathrm{MAE}}=\frac1n\sum_i|o_i-y_i|

Loss: how one wrong prediction becomes a number

Definition. A residual is the signed prediction error. Squared-error objectives aggregate squared residuals; their scaling convention changes both the reported loss and its derivative.

e_i=\hat y_i-y_i,\qquad \mathrm{SSE}=\sum_{i=1}^{n}e_i^2,\qquad L_{\mathrm{half}}=\frac12\sum_{i=1}^{n}e_i^2,\qquad \mathrm{MSE}=\frac1n\sum_{i=1}^{n}e_i^2

For this definition of n, each i indexes one included scalar prediction. In a multi-output batch, state whether the mean is over examples, output coordinates, or both. Half SSE and MSE happen to agree when n = 2; they are not generally the same objective.

Derivatives and their scaling

When differentiating with respect to prediction j, all other prediction terms are constant. The derivative of the j-th square is twice its residual:

The factor 1/2 cancels the derivative's factor 2. The factor 1/n instead produces 2/n. A weight derivative must also multiply by the derivative of the prediction with respect to that weight.

Question. Predictions are [0.2, 0.7, 0.4] and targets are [0, 1, 1]. Find SSE, half SSE and MSE, then their gradients with respect to the prediction vector.

Given

Predictions = [0.2,0.7,0.4].

Targets = [0,1,1]; n = 3 scalar predictions.

To find

Find SSE, half SSE and MSE, then all three prediction-gradient lists.

Formula — which one and why

Squared-error objectives. Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions.

ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss.

$e_i=\hat y_i-y_i$ LaTeX: e_i=\hat y_i-y_i

$\mathrm{SSE}=\sum_i e_i^2$ LaTeX: \mathrm{SSE}=\sum_i e_i^2

$L_{\mathrm{half}}=\frac12\sum_i e_i^2$ LaTeX: L_{\mathrm{half}}=\frac12\sum_i e_i^2

$\mathrm{MSE}=\frac1n\sum_i e_i^2$ LaTeX: \mathrm{MSE}=\frac1n\sum_i e_i^2

Yaad rakhne ka cue: Pehle error square; phir question ke hisaab se sum, half ya mean.

Solution — step by step

Step 1: calculate the three signed prediction errors.

Step 2: square every error.

$e\odot e=[0.04,0.09,0.36]$ LaTeX: e\odot e=[0.04,0.09,0.36]

Step 3: add the three squared errors.

Step 4: apply the requested loss divisors.

Step 5: multiply every error by 2 for the SSE gradient.

$\nabla_{\hat y}\mathrm{SSE}=[0.4,-0.6,-1.2]$ LaTeX: \nabla_{\hat y}\mathrm{SSE}=[0.4,-0.6,-1.2]

Step 6: use each error itself for the half-SSE gradient.

$\nabla_{\hat y}L_{\mathrm{half}}=[0.2,-0.3,-0.6]$ LaTeX: \nabla_{\hat y}L_{\mathrm{half}}=[0.2,-0.3,-0.6]

Step 7: multiply every error by 2/3 for the MSE gradient.

The residual signs survive in the gradients even though the losses are nonnegative. Multiplying a loss by a positive constant leaves its unregularized minimizers unchanged, but scales its gradients and therefore its updates at a fixed learning rate.

Drill 2 Half sum versus mean

Question

Question. Predictions = [0.2,0.7,0.4], targets = [0,1,1]. Find SSE, half SSE and MSE.