# Loss scaling: derive the gradient instead of guessing a factor

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/3/tee-c3-s8

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c3-s8/

### 1. One error, three possible objectives

Let a be a prediction and y a fixed target. The error is a − y. The derivative of that error with respect to a is 1 because y stays fixed. Apply the chain rule to its square:

![Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged.](https://deep-learning-study-room.vercel.app/assets/figures/v_loss_gradient.svg)

Diagram: Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged.. Labels: Prediction error and its derivative | Prediction error and its derivative | Keep the target y fixed when differentiating. | Half squared loss | L = ½(ŷ − y)² | Derivative = ŷ − y | The 2 cancels the half. | Mean squared error | L = sum of squares ÷ n | Derivative per output | = (2/n)(ŷ − y) | The scale of the loss changes the scale of its gradient.

Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged. Tap the diagram to enlarge.



$$
\frac{\partial (a-y)^2}{\partial a}=2(a-y)\times1
$$



Multiplying the objective by a constant multiplies its derivative by the same constant. This gives the common loss conventions below.

| Stated loss over n outputs | Derivative with respect to one prediction aⱼ |
| --- | --- |
| SSE: sum of all squared errors | 2(aⱼ − yⱼ) |
| Half SSE: half of that sum | aⱼ − yⱼ |
| MSE: that sum divided by n | 2(aⱼ − yⱼ)/n |
| Half MSE: that sum divided by 2n | (aⱼ − yⱼ)/n |

### 2. Why only the matching output appears

Suppose three predictions are separate output variables a₁, a₂ and a₃. When differentiating the loss with respect to a₂, the a₁ and a₃ terms are constant with respect to that output variable, so their direct derivatives are zero. The remaining term is the error of output 2, with the correct scaling factor.

Later, a shared hidden weight may influence all three predictions. Backpropagation then combines those separate output contributions through the chain rule. “Differentiate with respect to one output” and “differentiate with respect to a shared weight” are different stages.

### 3. The activation contributes another factor

If a = sigmoid(z), the derivative of a with respect to its input is a(1 − a). For half SSE, the output delta is therefore:



$$
\delta=\frac{\partial L}{\partial z}=(a-y)a(1-a)
$$



For a linear output, the activation derivative is 1. For positive ReLU input it is also 1; for negative ReLU input it is 0. Always use the derivative of the activation actually given in the question.

### 4. A numerical comparison

For a = 0.7 and y = 1, the signed error is −0.3. For half SSE with sigmoid, the activation derivative is 0.7 × 0.3 = 0.21. The output delta is −0.3 × 0.21 = −0.063. With plain SSE it doubles to −0.126. With MSE over three outputs it is −0.126/3 = −0.042.

The forward prediction 0.7 did not change. The loss convention changed the gradient scale. This is why the first line of a backpropagation answer should state the loss being used.