Deep Learning · TEE · formulas

Categorical cross entropy

For one-hot multiclass targets, only one target entry is 1. The other entries are zero, so the loss becomes minus the logarithm of the true class probability. The worked question first calculates every softmax probability, then selects the correct one.

$H(p,q)=-\sum_i p_i\ln q_i$ LaTeX: H(p,q)=-\sum_i p_i\ln q_i

$\log_2q=\frac{\ln q}{\ln2}$ LaTeX: \log_2q=\frac{\ln q}{\ln2}

Cross-entropy for probability distributions

Definition. For a discrete target distribution p and a predicted distribution q over the same classes, cross-entropy is the target-weighted negative logarithm of the predicted probabilities.

$H(p,q)=-\sum_{i=1}^{K}p_i\log q_i$ LaTeX: H(p,q)=-\sum_{i=1}^{K}p_i\log q_i

Here pᵢ and qᵢ refer to the same class i, and each distribution sums to 1. The target supplies the weights outside the logarithm; the prediction goes inside it. If the target is one-hot, only the correct-class term remains. For a soft target with several nonzero entries, include all its nonzero terms.

Logarithm base and units

Natural logarithms produce nats; base-2 logarithms produce bits. These notes use ln unless a question states another base. On a calculator with ln, use the change-of-base rule for base 2:

$\log_2 q=\frac{\ln q}{\ln 2}$ LaTeX: \log_2 q=\frac{\ln q}{\ln 2}

For a target entry pᵢ = 0, its contribution is defined as zero. If pᵢ > 0 but qᵢ = 0, the loss is infinite. For numerical stability, software normally calculates cross-entropy from logits or carefully handles boundary probabilities.

A target distribution with two nonzero entries

Question. A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5]. Calculate cross-entropy using natural logarithms and using base-2 logarithms.

Given

A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5].

Calculate cross-entropy using natural logarithms and using base-2 logarithms.

To find

Find cross-entropy in the requested units.

Formula — which one and why

Cross-entropy between distributions. Weight each negative log prediction by the target probability. Unlike one-hot targets, several terms can contribute.

pᵢ = target-distribution probability; qᵢ = predicted probability. Natural logs give nats; base-2 logs give bits.

$H(p,q)=-\sum_i p_i\ln q_i$ LaTeX: H(p,q)=-\sum_i p_i\ln q_i

$H_2(p,q)=\frac{H(p,q)}{\ln2}$ LaTeX: H_2(p,q)=\frac{H(p,q)}{\ln2}

Yaad rakhne ka cue: Target probability batati hai kis log-loss ka kitna hissa lena hai.

Solution — step by step

Step 1: write the two terms. Both target entries are nonzero, so neither term can be discarded.

Step 2: calculate the logarithms.

$\ln(0.5)\approx-0.693147181$ LaTeX: \ln(0.5)\approx-0.693147181

Step 3: weight each negative logarithm by its target probability.

Step 4: add the contributions, retaining calculator precision.

Step 5: repeat with base 2. Since 2⁻¹ = 0.5, log₂(0.5) = −1.

Final answer: 0.693147 nats or 1 bit. The distributions are unchanged; the logarithm base changes the unit and the numerical scale.

Cross-entropy, entropy and relative entropy

These names are related but do not denote the same quantity. Entropy uses the target distribution in both positions. Relative entropy, also called KL divergence, measures their difference through a probability ratio.

$H(p)=-\sum_i p_i\ln p_i$ LaTeX: H(p)=-\sum_i p_i\ln p_i

For fixed p, H(p) is constant with respect to the model parameters, so minimizing cross-entropy also minimizes this KL divergence. KL is clarified here to distinguish the terminology; it is not a separately named unit in the official syllabus.

Question. For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats.