Deep Learning · MT2 · formulas

Binary cross entropy

Binary cross-entropy measures how much probability was assigned to the correct binary answer. A target of 1 keeps the −ln(p) term; a target of 0 keeps the −ln(1−p) term. The factors y and 1−y select the appropriate term.

$L=-[y\ln p+(1-y)\ln(1-p)]$ LaTeX: L=-[y\ln p+(1-y)\ln(1-p)]

Drill 7 Sigmoid with binary cross entropy

Question

Question. A binary classifier has logit z = 1, target 1. Find probability, BCE and the gradient with respect to z.

Answer — step by step

Given

Logit z = 1; binary target y = 1.

Output: sigmoid; loss: binary cross-entropy.

To find

Find probability, BCE and the loss derivative with respect to z.

Formula — which one and why

Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.

$p=\frac{1}{1+e^{-z}}$ LaTeX: p=\frac{1}{1+e^{-z}}

$\frac{dp}{dz}=p(1-p)$ LaTeX: \frac{dp}{dz}=p(1-p)

Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).

Binary cross-entropy. For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y.

y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative.

$L=-[y\ln p+(1-y)\ln(1-p)]$ LaTeX: L=-[y\ln p+(1-y)\ln(1-p)]

$\frac{\partial L}{\partial z}=p-y$ LaTeX: \frac{\partial L}{\partial z}=p-y

Yaad rakhne ka cue: Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p).

Solution — step by step

Meaning and formulas

A logit is the raw score before sigmoid. Sigmoid converts it to a probability p. The target y = 1 says the positive class is correct. Binary cross-entropy measures how little probability the model assigned to the correct binary outcome.

$p=\frac1{1+e^{-z}}$ LaTeX: p=\frac1{1+e^{-z}}

$L=-\left[y\ln p+(1-y)\ln(1-p)\right]$ LaTeX: L=-\left[y\ln p+(1-y)\ln(1-p)\right]

For sigmoid combined with this loss, the gradient with respect to the logit has the simplified form:

$\frac{\partial L}{\partial z}=p-y$ LaTeX: \frac{\partial L}{\partial z}=p-y

Step 1: convert logit 1 into probability

$e^{-1}\approx0.367879441$ LaTeX: e^{-1}\approx0.367879441

$1+e^{-1}\approx1.367879441$ LaTeX: 1+e^{-1}\approx1.367879441

Step 2: substitute the target in the full loss formula

$L=-\left[1\ln(0.731058579)+(1-1)\ln(1-0.731058579)\right]$ LaTeX: L=-\left[1\ln(0.731058579)+(1-1)\ln(1-0.731058579)\right]

The second term has multiplier zero, so it contributes zero.

Step 3: calculate the combined gradient

Final answer: probability ≈ 0.731059; BCE ≈ 0.313262; logit gradient ≈ −0.268941. The simplified gradient already includes sigmoid's effect. Multiplying by sigmoid's slope again would apply it twice.

Binary cross-entropy: target 1, predicted positive-class probability 0.8

Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 1. Calculate binary cross-entropy using natural logarithms.

Given

Target y = 1; p = P(y=1) = 0.8; use natural log ln.

To find

The binary cross-entropy loss for this one example.

Formula — which one and why

The target selects the probability assigned to the correct class. Take its negative natural logarithm.

p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.

$L=-[y\ln p+(1-y)\ln(1-p)]$ LaTeX: L=-[y\ln p+(1-y)\ln(1-p)]

Yaad rakhne ka cue: True class ki probability lo, uska minus ln lo.

Solution — step by step

Step 1: identify the correct-class probability.

$p=0.8$ LaTeX: p=0.8

Step 2: evaluate its natural logarithm.

$\ln(0.8)\approx-0.223143551$ LaTeX: \ln(0.8)\approx-0.223143551

Step 3: apply the minus sign outside the logarithm.

Final answer: Loss ≈ 0.223143551 nats.

Binary cross-entropy: target 0, predicted positive-class probability 0.8

Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 0. Calculate binary cross-entropy using natural logarithms.

Given

Target y = 0; p = P(y=1) = 0.8; use natural log ln.

To find

The binary cross-entropy loss for this one example.

Formula — which one and why

The target selects the probability assigned to the correct class. Take its negative natural logarithm.

p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.

$L=-[y\ln p+(1-y)\ln(1-p)]$ LaTeX: L=-[y\ln p+(1-y)\ln(1-p)]

Yaad rakhne ka cue: True class ki probability lo, uska minus ln lo.

Solution — step by step

Step 1: identify the correct-class probability.

$1-p=1-0.8=0.2$ LaTeX: 1-p=1-0.8=0.2

Step 2: evaluate its natural logarithm.

$\ln(0.2)\approx-1.609437912$ LaTeX: \ln(0.2)\approx-1.609437912

Step 3: apply the minus sign outside the logarithm.

Final answer: Loss ≈ 1.609437912 nats.

Binary cross-entropy: target 1, predicted positive-class probability 0.5

Question. A binary classifier predicts P(y=1) = 0.5. The target is y = 1. Calculate binary cross-entropy using natural logarithms.

Given

Target y = 1; p = P(y=1) = 0.5; use natural log ln.

To find

The binary cross-entropy loss for this one example.

Formula — which one and why

The target selects the probability assigned to the correct class. Take its negative natural logarithm.

p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.

$L=-[y\ln p+(1-y)\ln(1-p)]$ LaTeX: L=-[y\ln p+(1-y)\ln(1-p)]

Yaad rakhne ka cue: True class ki probability lo, uska minus ln lo.

Solution — step by step

Step 1: identify the correct-class probability.

$p=0.5$ LaTeX: p=0.5

Step 2: evaluate its natural logarithm.

$\ln(0.5)\approx-0.693147181$ LaTeX: \ln(0.5)\approx-0.693147181

Step 3: apply the minus sign outside the logarithm.

Final answer: Loss ≈ 0.693147181 nats.