Deep Learning · TEE · section

Cross-entropy for probability distributions

Neural-network foundations

Definition. For a discrete target distribution p and a predicted distribution q over the same classes, cross-entropy is the target-weighted negative logarithm of the predicted probabilities.

H(p,q)=-\sum_{i=1}^{K}p_i\log q_iLaTeX: H(p,q)=-\sum_{i=1}^{K}p_i\log q_i

Here pᵢ and qᵢ refer to the same class i, and each distribution sums to 1. The target supplies the weights outside the logarithm; the prediction goes inside it. If the target is one-hot, only the correct-class term remains. For a soft target with several nonzero entries, include all its nonzero terms.

Logarithm base and units

Natural logarithms produce nats; base-2 logarithms produce bits. These notes use ln unless a question states another base. On a calculator with ln, use the change-of-base rule for base 2:

\log_2 q=\frac{\ln q}{\ln 2}LaTeX: \log_2 q=\frac{\ln q}{\ln 2}

For a target entry pᵢ = 0, its contribution is defined as zero. If pᵢ > 0 but qᵢ = 0, the loss is infinite. For numerical stability, software normally calculates cross-entropy from logits or carefully handles boundary probabilities.

A target distribution with two nonzero entries

Question. A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5]. Calculate cross-entropy using natural logarithms and using base-2 logarithms.

Given

A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5].

Calculate cross-entropy using natural logarithms and using base-2 logarithms.

To find

Find cross-entropy in the requested units.

Formula — which one and why

Cross-entropy between distributions. Weight each negative log prediction by the target probability. Unlike one-hot targets, several terms can contribute.

pᵢ = target-distribution probability; qᵢ = predicted probability. Natural logs give nats; base-2 logs give bits.

H(p,q)=-\sum_i p_i\ln q_iLaTeX: H(p,q)=-\sum_i p_i\ln q_i
H_2(p,q)=\frac{H(p,q)}{\ln2}LaTeX: H_2(p,q)=\frac{H(p,q)}{\ln2}

Yaad rakhne ka cue: Target probability batati hai kis log-loss ka kitna hissa lena hai.

Solution — step by step

Step 1: write the two terms. Both target entries are nonzero, so neither term can be discarded.

H(p,q)=-[0.75\ln(0.5)+0.25\ln(0.5)]LaTeX: H(p,q)=-[0.75\ln(0.5)+0.25\ln(0.5)]

Step 2: calculate the logarithms.

\ln(0.5)\approx-0.693147181LaTeX: \ln(0.5)\approx-0.693147181

Step 3: weight each negative logarithm by its target probability.

-0.75\ln(0.5)\approx0.519860385LaTeX: -0.75\ln(0.5)\approx0.519860385
-0.25\ln(0.5)\approx0.173286795LaTeX: -0.25\ln(0.5)\approx0.173286795

Step 4: add the contributions, retaining calculator precision.

H(p,q)\approx0.519860385+0.173286795\approx0.693147181\ \text{nats}LaTeX: H(p,q)\approx0.519860385+0.173286795\approx0.693147181\ \text{nats}

Step 5: repeat with base 2. Since 2⁻¹ = 0.5, log₂(0.5) = −1.

H_2(p,q)=-[0.75(-1)+0.25(-1)]=1\ \text{bit}LaTeX: H_2(p,q)=-[0.75(-1)+0.25(-1)]=1\ \text{bit}

Final answer: 0.693147 nats or 1 bit. The distributions are unchanged; the logarithm base changes the unit and the numerical scale.

Cross-entropy, entropy and relative entropy

These names are related but do not denote the same quantity. Entropy uses the target distribution in both positions. Relative entropy, also called KL divergence, measures their difference through a probability ratio.

H(p)=-\sum_i p_i\ln p_iLaTeX: H(p)=-\sum_i p_i\ln p_i
D_{\mathrm{KL}}(p\Vert q)=\sum_i p_i\ln\frac{p_i}{q_i}=H(p,q)-H(p)LaTeX: D_{\mathrm{KL}}(p\Vert q)=\sum_i p_i\ln\frac{p_i}{q_i}=H(p,q)-H(p)

For fixed p, H(p) is constant with respect to the model parameters, so minimizing cross-entropy also minimizes this KL divergence. KL is clarified here to distinguish the terminology; it is not a separately named unit in the official syllabus.

Question. For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats.

Given

For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats.

To find

Find entropy and KL divergence in nats.

Formula — which one and why

Entropy and relative entropy. Entropy measures the target distribution’s uncertainty. Relative entropy measures the extra cross-entropy when q is used instead of p.

p = target distribution; q = model distribution; H(p) = entropy; DKL = Kullback–Leibler divergence.

H(p)=-\sum_i p_i\ln p_iLaTeX: H(p)=-\sum_i p_i\ln p_i
D_{\mathrm{KL}}(p\Vert q)=H(p,q)-H(p)LaTeX: D_{\mathrm{KL}}(p\Vert q)=H(p,q)-H(p)

Yaad rakhne ka cue: KL = cross-entropy minus target ki own entropy.

Solution — step by step

Step 1: calculate the target entropy.

H(p)=-[0.75\ln(0.75)+0.25\ln(0.25)]LaTeX: H(p)=-[0.75\ln(0.75)+0.25\ln(0.25)]
\ln(0.75)\approx-0.287682072,\qquad\ln(0.25)\approx-1.386294361LaTeX: \ln(0.75)\approx-0.287682072,\qquad\ln(0.25)\approx-1.386294361
H(p)\approx0.215761554+0.346573590=0.562335145LaTeX: H(p)\approx0.215761554+0.346573590=0.562335145

Step 2: subtract target entropy from cross-entropy.

D_{\mathrm{KL}}(p\Vert q)\approx0.693147181-0.562335145=0.130812036LaTeX: D_{\mathrm{KL}}(p\Vert q)\approx0.693147181-0.562335145=0.130812036

Final answer: H(p) ≈ 0.562335 nats and DKL(p ∥ q) ≈ 0.130812 nats. For a one-hot target H(p) = 0, so cross-entropy and this KL divergence coincide; for this soft target they differ.