Natural Language Processing · TEE · papers
Q2a · Normalize the Tanisha–Simran passage
2023-01-30 · re-exam · recovered AI paper
Previous examination · complete worked answers
Answer
No stoplist/stemmer is specified. For a reproducible answer choose stoplist {and,are,they,with,other,to,but,be,a,an} after lowercase; use the Porter stemmer. Other declared lists produce different valid counts.
| Stage | Result |
|---|---|
| Sentence segmentation | Tanisha and Simran are best friends. They play together with other friends. Tanisha likes to play badminton but Simran prefers to play online games. Tanisha wants to be a badminton player. Simran wants to become an online gamer. |
| Word tokens (punctuation omitted) | Tanisha, and, Simran, are, best, friends, They, play, together, with, other, friends, Tanisha, likes, to, play, badminton, but, Simran, prefers, to, play, online, games, Tanisha, wants, to, be, a, badminton, player, Simran, wants, to, become, an, online, gamer |
| Lowercase | tanisha, and, simran, are, best, friends, they, play, together, with, other, friends, tanisha, likes, to, play, badminton, but, simran, prefers, to, play, online, games, tanisha, wants, to, be, a, badminton, player, simran, wants, to, become, an, online, gamer |
| Declared stoplist | a, an, and, are, be, but, other, they, to, with |
| After exact stop removal | tanisha, simran, best, friends, play, together, friends, tanisha, likes, play, badminton, simran, prefers, play, online, games, tanisha, wants, badminton, player, simran, wants, become, online, gamer |
| Retained word | Porter stem |
|---|---|
| tanisha | tanisha |
| simran | simran |
| best | best |
| friends | friend |
| play | play |
| together | togeth |
| likes | like |
| badminton | badminton |
| prefers | prefer |
| online | onlin |
| games | game |
| wants | want |
| player | player |
| become | becom |
| gamer | gamer |
Full final token sequence: tanisha, simran, best, friend, play, togeth, friend, tanisha, like, play, badminton, simran, prefer, play, onlin, game, tanisha, want, badminton, player, simran, want, becom, onlin, gamer
Raw case-sensitive vocabulary (25): Simran, Tanisha, They, a, an, and, are, badminton, be, become, best, but, friends, gamer, games, likes, online, other, play, player, prefers, to, together, wants, with. After lowercase only: 25 distinct words. Final stem vocabulary (15): badminton, becom, best, friend, game, gamer, like, onlin, play, player, prefer, simran, tanisha, togeth, want.
Source / convention note. Stoplist and stemming algorithm are choices, not supplied numerical givens; both are stated.