# Binary cross entropy

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/14

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/14/

Binary cross-entropy measures how much probability was assigned to the correct binary answer. A target of 1 keeps the −ln(p) term; a target of 0 keeps the −ln(1−p) term. The factors y and 1−y select the appropriate term.



$$
L=-[y\ln p+(1-y)\ln(1-p)]
$$



## Drill 7 Sigmoid with binary cross entropy

## Question

**Question.** A binary classifier has logit z = 1, target 1. Find probability, BCE and the gradient with respect to z.

## Answer — step by step

### Given

Logit z = 1; binary target y = 1.

Output: sigmoid; loss: binary cross-entropy.

### To find

Find probability, BCE and the loss derivative with respect to z.

### Formula — which one and why

**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.



$$
p=\frac{1}{1+e^{-z}}
$$





$$
\frac{dp}{dz}=p(1-p)
$$



**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).

**Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y.

y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative.



$$
L=-[y\ln p+(1-y)\ln(1-p)]
$$





$$
\frac{\partial L}{\partial z}=p-y
$$



**Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p).

### Solution — step by step

### Meaning and formulas

A logit is the raw score before sigmoid. Sigmoid converts it to a probability p. The target y = 1 says the positive class is correct. Binary cross-entropy measures how little probability the model assigned to the correct binary outcome.



$$
p=\frac1{1+e^{-z}}
$$





$$
L=-\left[y\ln p+(1-y)\ln(1-p)\right]
$$



For sigmoid combined with this loss, the gradient with respect to the logit has the simplified form:



$$
\frac{\partial L}{\partial z}=p-y
$$



### Step 1: convert logit 1 into probability



$$
e^{-1}\approx0.367879441
$$





$$
1+e^{-1}\approx1.367879441
$$





$$
p=\frac1{1.367879441}\approx0.731058579
$$



### Step 2: substitute the target in the full loss formula



$$
L=-\left[1\ln(0.731058579)+(1-1)\ln(1-0.731058579)\right]
$$



The second term has multiplier zero, so it contributes zero.



$$
L=-\ln(0.731058579)\approx0.313261688
$$



### Step 3: calculate the combined gradient



$$
\frac{\partial L}{\partial z}=0.731058579-1=-0.268941421
$$



**Final answer:** probability ≈ 0.731059; BCE ≈ 0.313262; logit gradient ≈ −0.268941. The simplified gradient already includes sigmoid's effect. Multiplying by sigmoid's slope again would apply it twice.

### Binary cross-entropy: target 1, predicted positive-class probability 0.8

**Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 1. Calculate binary cross-entropy using natural logarithms.**

### Given

Target y = 1; p = P(y=1) = 0.8; use natural log ln.

### To find

The binary cross-entropy loss for this one example.

### Formula — which one and why

The target selects the probability assigned to the correct class. Take its negative natural logarithm.

p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.



$$
L=-[y\ln p+(1-y)\ln(1-p)]
$$



**Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo.

### Solution — step by step

**Step 1: identify the correct-class probability.**



$$
p=0.8
$$



**Step 2: evaluate its natural logarithm.**



$$
\ln(0.8)\approx-0.223143551
$$



**Step 3: apply the minus sign outside the logarithm.**



$$
L=-(-0.223143551)=0.223143551
$$



**Final answer:** Loss ≈ 0.223143551 nats.

### Binary cross-entropy: target 0, predicted positive-class probability 0.8

**Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 0. Calculate binary cross-entropy using natural logarithms.**

### Given

Target y = 0; p = P(y=1) = 0.8; use natural log ln.

### To find

The binary cross-entropy loss for this one example.

### Formula — which one and why

The target selects the probability assigned to the correct class. Take its negative natural logarithm.

p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.



$$
L=-[y\ln p+(1-y)\ln(1-p)]
$$



**Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo.

### Solution — step by step

**Step 1: identify the correct-class probability.**



$$
1-p=1-0.8=0.2
$$



**Step 2: evaluate its natural logarithm.**



$$
\ln(0.2)\approx-1.609437912
$$



**Step 3: apply the minus sign outside the logarithm.**



$$
L=-(-1.609437912)=1.609437912
$$



**Final answer:** Loss ≈ 1.609437912 nats.

### Binary cross-entropy: target 1, predicted positive-class probability 0.5

**Question. A binary classifier predicts P(y=1) = 0.5. The target is y = 1. Calculate binary cross-entropy using natural logarithms.**

### Given

Target y = 1; p = P(y=1) = 0.5; use natural log ln.

### To find

The binary cross-entropy loss for this one example.

### Formula — which one and why

The target selects the probability assigned to the correct class. Take its negative natural logarithm.

p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.



$$
L=-[y\ln p+(1-y)\ln(1-p)]
$$



**Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo.

### Solution — step by step

**Step 1: identify the correct-class probability.**



$$
p=0.5
$$



**Step 2: evaluate its natural logarithm.**



$$
\ln(0.5)\approx-0.693147181
$$



**Step 3: apply the minus sign outside the logarithm.**



$$
L=-(-0.693147181)=0.693147181
$$



**Final answer:** Loss ≈ 0.693147181 nats.