Deep Learning · TEE · section

Loss scaling: derive the gradient instead of guessing a factor

The maths behind a network

1. One error, three possible objectives

Let a be a prediction and y a fixed target. The error is a − y. The derivative of that error with respect to a is 1 because y stays fixed. Apply the chain rule to its square:

Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged. Tap the diagram to enlarge.

$\frac{\partial (a-y)^2}{\partial a}=2(a-y)\times1$ LaTeX: \frac{\partial (a-y)^2}{\partial a}=2(a-y)\times1

Multiplying the objective by a constant multiplies its derivative by the same constant. This gives the common loss conventions below.

Stated loss over n outputs	Derivative with respect to one prediction aⱼ
SSE: sum of all squared errors	2(aⱼ − yⱼ)
Half SSE: half of that sum	aⱼ − yⱼ
MSE: that sum divided by n	2(aⱼ − yⱼ)/n
Half MSE: that sum divided by 2n	(aⱼ − yⱼ)/n

2. Why only the matching output appears

Suppose three predictions are separate output variables a₁, a₂ and a₃. When differentiating the loss with respect to a₂, the a₁ and a₃ terms are constant with respect to that output variable, so their direct derivatives are zero. The remaining term is the error of output 2, with the correct scaling factor.

Later, a shared hidden weight may influence all three predictions. Backpropagation then combines those separate output contributions through the chain rule. “Differentiate with respect to one output” and “differentiate with respect to a shared weight” are different stages.

3. The activation contributes another factor

If a = sigmoid(z), the derivative of a with respect to its input is a(1 − a). For half SSE, the output delta is therefore:

$\delta=\frac{\partial L}{\partial z}=(a-y)a(1-a)$ LaTeX: \delta=\frac{\partial L}{\partial z}=(a-y)a(1-a)

For a linear output, the activation derivative is 1. For positive ReLU input it is also 1; for negative ReLU input it is 0. Always use the derivative of the activation actually given in the question.

4. A numerical comparison

For a = 0.7 and y = 1, the signed error is −0.3. For half SSE with sigmoid, the activation derivative is 0.7 × 0.3 = 0.21. The output delta is −0.3 × 0.21 = −0.063. With plain SSE it doubles to −0.126. With MSE over three outputs it is −0.126/3 = −0.042.

The forward prediction 0.7 did not change. The loss convention changed the gradient scale. This is why the first line of a backpropagation answer should state the loss being used.