Natural Language Processing · TEE · formulas

Precision, recall, F1 and accuracy

Application calculations

Formulas explained through full worked examples

Evaluation compares predicted results with known answers. A true positive is a correctly predicted positive; a false positive is a negative item predicted positive; a false negative is a missed positive; a true negative is a correctly rejected negative.

\text{Precision}=\frac{TP}{TP+FP}LaTeX: \text{Precision}=\frac{TP}{TP+FP}
\text{Recall}=\frac{TP}{TP+FN}LaTeX: \text{Recall}=\frac{TP}{TP+FN}
F_1=\frac{2PR}{P+R}=\frac{2TP}{2TP+FP+FN}LaTeX: F_1=\frac{2PR}{P+R}=\frac{2TP}{2TP+FP+FN}
\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}LaTeX: \text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}

Precision asks how many predicted positives were correct. Recall asks how many actual positives were found. F1 balances them by a harmonic mean. If a denominator is zero, state the convention rather than dividing by zero.

Example 1 · evaluate a classifier

A binary text classifier gives TP=6, FP=2, FN=4, TN=8. Compute precision, recall, F1 and accuracy.

Given

TP=6; FP=2; FN=4; TN=8.

To find

Four evaluation measures.

Formula and why it applies

P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN}LaTeX: P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN}
F_1=\frac{2TP}{2TP+FP+FN},\quad A=\frac{TP+TN}{TP+FP+FN+TN}LaTeX: F_1=\frac{2TP}{2TP+FP+FN},\quad A=\frac{TP+TN}{TP+FP+FN+TN}

Use predicted positives in precision, actual positives in recall, and all examples in accuracy.

Solution

Step 1. Precision

P=\frac{6}{6+2}=0.750000LaTeX: P=\frac{6}{6+2}=0.750000

Step 2. Recall

R=\frac{6}{6+4}=0.600000LaTeX: R=\frac{6}{6+4}=0.600000

Step 3. F1

F_1=\frac{12}{12+2+4}=0.666667LaTeX: F_1=\frac{12}{12+2+4}=0.666667

Step 4. Accuracy

A=\frac{6+8}{6+2+4+8}=0.700000LaTeX: A=\frac{6+8}{6+2+4+8}=0.700000
Precision 0.7500; recall 0.6000; F1 0.6667; accuracy 0.7000.

Example 2 · evaluate a classifier

A binary text classifier gives TP=9, FP=3, FN=1, TN=7. Compute precision, recall, F1 and accuracy.

Given

TP=9; FP=3; FN=1; TN=7.

To find

Four evaluation measures.

Formula and why it applies

P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN}LaTeX: P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN}
F_1=\frac{2TP}{2TP+FP+FN},\quad A=\frac{TP+TN}{TP+FP+FN+TN}LaTeX: F_1=\frac{2TP}{2TP+FP+FN},\quad A=\frac{TP+TN}{TP+FP+FN+TN}

Use predicted positives in precision, actual positives in recall, and all examples in accuracy.

Solution

Step 1. Precision

P=\frac{9}{9+3}=0.750000LaTeX: P=\frac{9}{9+3}=0.750000

Step 2. Recall

R=\frac{9}{9+1}=0.900000LaTeX: R=\frac{9}{9+1}=0.900000

Step 3. F1

F_1=\frac{18}{18+3+1}=0.818182LaTeX: F_1=\frac{18}{18+3+1}=0.818182

Step 4. Accuracy

A=\frac{9+7}{9+3+1+7}=0.800000LaTeX: A=\frac{9+7}{9+3+1+7}=0.800000
Precision 0.7500; recall 0.9000; F1 0.8182; accuracy 0.8000.

Example 3 · evaluate a classifier

A binary text classifier gives TP=4, FP=0, FN=4, TN=12. Compute precision, recall, F1 and accuracy.

Given

TP=4; FP=0; FN=4; TN=12.

To find

Four evaluation measures.

Formula and why it applies

P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN}LaTeX: P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN}
F_1=\frac{2TP}{2TP+FP+FN},\quad A=\frac{TP+TN}{TP+FP+FN+TN}LaTeX: F_1=\frac{2TP}{2TP+FP+FN},\quad A=\frac{TP+TN}{TP+FP+FN+TN}

Use predicted positives in precision, actual positives in recall, and all examples in accuracy.

Solution

Step 1. Precision

P=\frac{4}{4+0}=1.000000LaTeX: P=\frac{4}{4+0}=1.000000

Step 2. Recall

R=\frac{4}{4+4}=0.500000LaTeX: R=\frac{4}{4+4}=0.500000

Step 3. F1

F_1=\frac{8}{8+0+4}=0.666667LaTeX: F_1=\frac{8}{8+0+4}=0.666667

Step 4. Accuracy

A=\frac{4+12}{4+0+4+12}=0.800000LaTeX: A=\frac{4+12}{4+0+4+12}=0.800000
Precision 1.0000; recall 0.5000; F1 0.6667; accuracy 0.8000.