Natural Language Processing · TEE · papers

Q6b · Classification evaluation including ROC/AUC

2023-01-30 · re-exam · recovered AI paper

Previous examination · complete worked answers

Answer

Evaluation compares predicted results with known answers. A true positive is a correctly predicted positive; a false positive is a negative item predicted positive; a false negative is a missed positive; a true negative is a correctly rejected negative.

LaTeX: \text{Precision}=\frac{TP}{TP+FP}

LaTeX: \text{Recall}=\frac{TP}{TP+FN}

LaTeX: F_1=\frac{2PR}{P+R}=\frac{2TP}{2TP+FP+FN}

LaTeX: \text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}

Precision asks how many predicted positives were correct. Recall asks how many actual positives were found. F1 balances them by a harmonic mean. If a denominator is zero, state the convention rather than dividing by zero.

Example 1 · evaluate a classifier

A binary text classifier gives TP=6, FP=2, FN=4, TN=8. Compute precision, recall, F1 and accuracy.

Given

TP=6; FP=2; FN=4; TN=8.

To find

Four evaluation measures.

Formula and why it applies

LaTeX: P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN}

LaTeX: F_1=\frac{2TP}{2TP+FP+FN},\quad A=\frac{TP+TN}{TP+FP+FN+TN}

Use predicted positives in precision, actual positives in recall, and all examples in accuracy.

Solution

Step 1. Precision

LaTeX: P=\frac{6}{6+2}=0.750000

Step 2. Recall

LaTeX: R=\frac{6}{6+4}=0.600000

Step 3. F1

LaTeX: F_1=\frac{12}{12+2+4}=0.666667

Step 4. Accuracy

LaTeX: A=\frac{6+8}{6+2+4+8}=0.700000

Precision 0.7500; recall 0.6000; F1 0.6667; accuracy 0.7000.

Example 2 · evaluate a classifier

A binary text classifier gives TP=9, FP=3, FN=1, TN=7. Compute precision, recall, F1 and accuracy.

Given

TP=9; FP=3; FN=1; TN=7.

To find

Four evaluation measures.

Formula and why it applies

LaTeX: P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN}

LaTeX: F_1=\frac{2TP}{2TP+FP+FN},\quad A=\frac{TP+TN}{TP+FP+FN+TN}

Use predicted positives in precision, actual positives in recall, and all examples in accuracy.

Solution

Step 1. Precision

LaTeX: P=\frac{9}{9+3}=0.750000

Step 2. Recall

LaTeX: R=\frac{9}{9+1}=0.900000

Step 3. F1

LaTeX: F_1=\frac{18}{18+3+1}=0.818182

Step 4. Accuracy

LaTeX: A=\frac{9+7}{9+3+1+7}=0.800000

Precision 0.7500; recall 0.9000; F1 0.8182; accuracy 0.8000.

Example 3 · evaluate a classifier

A binary text classifier gives TP=4, FP=0, FN=4, TN=12. Compute precision, recall, F1 and accuracy.

Given

TP=4; FP=0; FN=4; TN=12.

To find

Four evaluation measures.

Formula and why it applies

LaTeX: P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN}

LaTeX: F_1=\frac{2TP}{2TP+FP+FN},\quad A=\frac{TP+TN}{TP+FP+FN+TN}

Use predicted positives in precision, actual positives in recall, and all examples in accuracy.

Solution

Step 1. Precision

LaTeX: P=\frac{4}{4+0}=1.000000

Step 2. Recall

LaTeX: R=\frac{4}{4+4}=0.500000

Step 3. F1

LaTeX: F_1=\frac{8}{8+0+4}=0.666667

Step 4. Accuracy

LaTeX: A=\frac{4+12}{4+0+4+12}=0.800000

Precision 1.0000; recall 0.5000; F1 0.6667; accuracy 0.8000.

ROC and AUC

LaTeX: \mathrm{TPR}=\frac{TP}{TP+FN},\qquad\mathrm{FPR}=\frac{FP}{FP+TN}

A classifier produces scores. Sweep the decision threshold; at each threshold calculate TPR and FPR. Plot FPR horizontally and TPR vertically. AUC is the area under that curve, equivalently the probability that a randomly chosen positive has a higher score than a randomly chosen negative (ties conventionally contribute half). AUC1 is perfect ranking;0.5 is chance ranking. This evaluates ranking over thresholds, not the accuracy at one threshold.

For straight segments through (0,0),(.2,.8),(1,1), trapezoid area=.2×(0+.8)/2+.8×(.8+1)/2=.08+.72=.80. These points illustrate integration; they are not supplied exam data. Precision–recall may be more informative for rare positives.