# Sigmoid plus BCE; softmax plus CCE

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/17

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/17/

Some activation–loss pairs simplify together. Sigmoid with binary cross-entropy, and softmax with one-hot categorical cross-entropy, give prediction minus target at the raw-score level. This result already includes the activation’s effect.



$$
\delta^{out}=p-y
$$



## Drill 7 Sigmoid with binary cross entropy

## Question

**Question.** A binary classifier has logit z = 1, target 1. Find probability, BCE and the gradient with respect to z.

## Answer — step by step

### Given

Logit z = 1; binary target y = 1.

Output: sigmoid; loss: binary cross-entropy.

### To find

Find probability, BCE and the loss derivative with respect to z.

### Formula — which one and why

**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.



$$
p=\frac{1}{1+e^{-z}}
$$





$$
\frac{dp}{dz}=p(1-p)
$$



**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).

**Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y.

y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative.



$$
L=-[y\ln p+(1-y)\ln(1-p)]
$$





$$
\frac{\partial L}{\partial z}=p-y
$$



**Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p).

### Solution — step by step

### Meaning and formulas

A logit is the raw score before sigmoid. Sigmoid converts it to a probability p. The target y = 1 says the positive class is correct. Binary cross-entropy measures how little probability the model assigned to the correct binary outcome.



$$
p=\frac1{1+e^{-z}}
$$





$$
L=-\left[y\ln p+(1-y)\ln(1-p)\right]
$$



For sigmoid combined with this loss, the gradient with respect to the logit has the simplified form:



$$
\frac{\partial L}{\partial z}=p-y
$$



### Step 1: convert logit 1 into probability



$$
e^{-1}\approx0.367879441
$$





$$
1+e^{-1}\approx1.367879441
$$





$$
p=\frac1{1.367879441}\approx0.731058579
$$



### Step 2: substitute the target in the full loss formula



$$
L=-\left[1\ln(0.731058579)+(1-1)\ln(1-0.731058579)\right]
$$



The second term has multiplier zero, so it contributes zero.



$$
L=-\ln(0.731058579)\approx0.313261688
$$



### Step 3: calculate the combined gradient



$$
\frac{\partial L}{\partial z}=0.731058579-1=-0.268941421
$$



**Final answer:** probability ≈ 0.731059; BCE ≈ 0.313262; logit gradient ≈ −0.268941. The simplified gradient already includes sigmoid's effect. Multiplying by sigmoid's slope again would apply it twice.

### The multiclass version, entry by entry

**Question. Softmax probabilities are [0.1, 0.8, 0.1] and the one-hot target is [0, 1, 0]. For categorical cross-entropy on this one example, find the derivatives with respect to all three logits.**

### Given

Softmax probabilities = [0.1,0.8,0.1]; target = [0,1,0].

One-example categorical cross-entropy.

### To find

Find all three logit derivatives.

### Formula — which one and why

**Softmax with categorical cross-entropy.** For one example with one-hot target, differentiating softmax together with CCE simplifies the logit derivative to predicted probability minus target.

zᵢ = raw score before softmax; pᵢ = class probability; yᵢ = matching target entry.



$$
\frac{\partial L}{\partial z_i}=p_i-y_i
$$



**Yaad rakhne ka cue:** Softmax + CCE together: p minus y. Extra sigmoid slope nahi.

### Solution — step by step

A logit is the raw score before softmax. Subtract each target entry from its matching probability entry.



$$
\frac{\partial L}{\partial z_1}=0.1-0=0.1
$$





$$
\frac{\partial L}{\partial z_2}=0.8-1=-0.2
$$





$$
\frac{\partial L}{\partial z_3}=0.1-0=0.1
$$



**Final answer: [0.1, −0.2, 0.1].** Do not multiply these by a separate sigmoid derivative or by p(1−p). A mean over several examples introduces the corresponding averaging factor.

### Sigmoid+BCE logit gradient: p=0.8, y=1

**Question. A sigmoid output is p = 0.8 and target y = 1. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.**

### Given

p = 0.8; y = 1; single-example BCE; p = sigmoid(z).

### To find

dL/dz, not dL/dp.

### Formula — which one and why

Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y.

p is an activated probability; z is its raw score; the requested derivative is with respect to z.



$$
\frac{dL}{dp}=\frac{p-y}{p(1-p)}
$$





$$
\frac{dp}{dz}=p(1-p)
$$





$$
\frac{dL}{dz}=p-y
$$



**Yaad rakhne ka cue:** BCE + sigmoid together ka shortcut p−y; chain already included.

### Solution — step by step

**Step 1: calculate the sigmoid slope.**



$$
p(1-p)=0.8(0.2)=0.16
$$



**Step 2: differentiate BCE with respect to p.**



$$
\frac{dL}{dp}=\frac{0.8-1}{0.16}=-1.25
$$



**Step 3: multiply the two local derivatives.**



$$
\frac{dL}{dz}=(-1.25)(0.16)=-0.2
$$



**Final answer:** dL/dz = -0.2. No further sigmoid derivative is needed.

### Sigmoid+BCE logit gradient: p=0.8, y=0

**Question. A sigmoid output is p = 0.8 and target y = 0. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.**

### Given

p = 0.8; y = 0; single-example BCE; p = sigmoid(z).

### To find

dL/dz, not dL/dp.

### Formula — which one and why

Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y.

p is an activated probability; z is its raw score; the requested derivative is with respect to z.



$$
\frac{dL}{dp}=\frac{p-y}{p(1-p)}
$$





$$
\frac{dp}{dz}=p(1-p)
$$





$$
\frac{dL}{dz}=p-y
$$



**Yaad rakhne ka cue:** BCE + sigmoid together ka shortcut p−y; chain already included.

### Solution — step by step

**Step 1: calculate the sigmoid slope.**



$$
p(1-p)=0.8(0.2)=0.16
$$



**Step 2: differentiate BCE with respect to p.**



$$
\frac{dL}{dp}=\frac{0.8-0}{0.16}=5
$$



**Step 3: multiply the two local derivatives.**



$$
\frac{dL}{dz}=(5)(0.16)=0.8
$$



**Final answer:** dL/dz = 0.8. No further sigmoid derivative is needed.