Deep Learning · TEE · formulas

Sigmoid plus BCE; softmax plus CCE

Some activation–loss pairs simplify together. Sigmoid with binary cross-entropy, and softmax with one-hot categorical cross-entropy, give prediction minus target at the raw-score level. This result already includes the activation’s effect.

$\delta^{out}=p-y$ LaTeX: \delta^{out}=p-y

Drill 7 Sigmoid with binary cross entropy

Question

Question. A binary classifier has logit z = 1, target 1. Find probability, BCE and the gradient with respect to z.

Answer — step by step

Given

Logit z = 1; binary target y = 1.

Output: sigmoid; loss: binary cross-entropy.

To find

Find probability, BCE and the loss derivative with respect to z.

Formula — which one and why

Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.

$p=\frac{1}{1+e^{-z}}$ LaTeX: p=\frac{1}{1+e^{-z}}

$\frac{dp}{dz}=p(1-p)$ LaTeX: \frac{dp}{dz}=p(1-p)

Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).

Binary cross-entropy. For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y.

y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative.

$L=-[y\ln p+(1-y)\ln(1-p)]$ LaTeX: L=-[y\ln p+(1-y)\ln(1-p)]

$\frac{\partial L}{\partial z}=p-y$ LaTeX: \frac{\partial L}{\partial z}=p-y

Yaad rakhne ka cue: Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p).

Solution — step by step

Meaning and formulas

A logit is the raw score before sigmoid. Sigmoid converts it to a probability p. The target y = 1 says the positive class is correct. Binary cross-entropy measures how little probability the model assigned to the correct binary outcome.

$p=\frac1{1+e^{-z}}$ LaTeX: p=\frac1{1+e^{-z}}

$L=-\left[y\ln p+(1-y)\ln(1-p)\right]$ LaTeX: L=-\left[y\ln p+(1-y)\ln(1-p)\right]

For sigmoid combined with this loss, the gradient with respect to the logit has the simplified form:

$\frac{\partial L}{\partial z}=p-y$ LaTeX: \frac{\partial L}{\partial z}=p-y

Step 1: convert logit 1 into probability

$e^{-1}\approx0.367879441$ LaTeX: e^{-1}\approx0.367879441

$1+e^{-1}\approx1.367879441$ LaTeX: 1+e^{-1}\approx1.367879441

Step 2: substitute the target in the full loss formula

$L=-\left[1\ln(0.731058579)+(1-1)\ln(1-0.731058579)\right]$ LaTeX: L=-\left[1\ln(0.731058579)+(1-1)\ln(1-0.731058579)\right]

The second term has multiplier zero, so it contributes zero.

Step 3: calculate the combined gradient

Final answer: probability ≈ 0.731059; BCE ≈ 0.313262; logit gradient ≈ −0.268941. The simplified gradient already includes sigmoid's effect. Multiplying by sigmoid's slope again would apply it twice.

The multiclass version, entry by entry

Question. Softmax probabilities are [0.1, 0.8, 0.1] and the one-hot target is [0, 1, 0]. For categorical cross-entropy on this one example, find the derivatives with respect to all three logits.

Given

Softmax probabilities = [0.1,0.8,0.1]; target = [0,1,0].

One-example categorical cross-entropy.

To find

Find all three logit derivatives.

Formula — which one and why

Softmax with categorical cross-entropy. For one example with one-hot target, differentiating softmax together with CCE simplifies the logit derivative to predicted probability minus target.

zᵢ = raw score before softmax; pᵢ = class probability; yᵢ = matching target entry.

$\frac{\partial L}{\partial z_i}=p_i-y_i$ LaTeX: \frac{\partial L}{\partial z_i}=p_i-y_i

Yaad rakhne ka cue: Softmax + CCE together: p minus y. Extra sigmoid slope nahi.

Solution — step by step

A logit is the raw score before softmax. Subtract each target entry from its matching probability entry.

$\frac{\partial L}{\partial z_1}=0.1-0=0.1$ LaTeX: \frac{\partial L}{\partial z_1}=0.1-0=0.1

$\frac{\partial L}{\partial z_2}=0.8-1=-0.2$ LaTeX: \frac{\partial L}{\partial z_2}=0.8-1=-0.2

$\frac{\partial L}{\partial z_3}=0.1-0=0.1$ LaTeX: \frac{\partial L}{\partial z_3}=0.1-0=0.1

Final answer: [0.1, −0.2, 0.1]. Do not multiply these by a separate sigmoid derivative or by p(1−p). A mean over several examples introduces the corresponding averaging factor.

Sigmoid+BCE logit gradient: p=0.8, y=1

Question. A sigmoid output is p = 0.8 and target y = 1. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.

Given

p = 0.8; y = 1; single-example BCE; p = sigmoid(z).

To find

dL/dz, not dL/dp.

Formula — which one and why

Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y.

p is an activated probability; z is its raw score; the requested derivative is with respect to z.

$\frac{dL}{dp}=\frac{p-y}{p(1-p)}$ LaTeX: \frac{dL}{dp}=\frac{p-y}{p(1-p)}

$\frac{dp}{dz}=p(1-p)$ LaTeX: \frac{dp}{dz}=p(1-p)

$\frac{dL}{dz}=p-y$ LaTeX: \frac{dL}{dz}=p-y

Yaad rakhne ka cue: BCE + sigmoid together ka shortcut p−y; chain already included.

Solution — step by step

Step 1: calculate the sigmoid slope.

$p(1-p)=0.8(0.2)=0.16$ LaTeX: p(1-p)=0.8(0.2)=0.16

Step 2: differentiate BCE with respect to p.

$\frac{dL}{dp}=\frac{0.8-1}{0.16}=-1.25$ LaTeX: \frac{dL}{dp}=\frac{0.8-1}{0.16}=-1.25

Step 3: multiply the two local derivatives.

Final answer: dL/dz = -0.2. No further sigmoid derivative is needed.

Sigmoid+BCE logit gradient: p=0.8, y=0

Question. A sigmoid output is p = 0.8 and target y = 0. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.

Given

p = 0.8; y = 0; single-example BCE; p = sigmoid(z).

To find

dL/dz, not dL/dp.

Formula — which one and why

Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y.

p is an activated probability; z is its raw score; the requested derivative is with respect to z.

$\frac{dL}{dp}=\frac{p-y}{p(1-p)}$ LaTeX: \frac{dL}{dp}=\frac{p-y}{p(1-p)}

$\frac{dp}{dz}=p(1-p)$ LaTeX: \frac{dp}{dz}=p(1-p)

$\frac{dL}{dz}=p-y$ LaTeX: \frac{dL}{dz}=p-y

Yaad rakhne ka cue: BCE + sigmoid together ka shortcut p−y; chain already included.

Solution — step by step

Step 1: calculate the sigmoid slope.

$p(1-p)=0.8(0.2)=0.16$ LaTeX: p(1-p)=0.8(0.2)=0.16

Step 2: differentiate BCE with respect to p.

$\frac{dL}{dp}=\frac{0.8-0}{0.16}=5$ LaTeX: \frac{dL}{dp}=\frac{0.8-0}{0.16}=5

Step 3: multiply the two local derivatives.

$\frac{dL}{dz}=(5)(0.16)=0.8$ LaTeX: \frac{dL}{dz}=(5)(0.16)=0.8

Final answer: dL/dz = 0.8. No further sigmoid derivative is needed.