Natural Language Processing · TEE · papers
Q5a · Corpora and structure
2022-11-22 · final · recovered AI paper
Previous examination · complete worked answers
Answer
A corpus is a collection of language data assembled for study or computational use. Its plural is corpora. A corpus can contain books, news, reviews or conversations. It need not be labelled. Annotation adds linguistic information such as POS tags, entities, parse structures or sense labels.
| Structure | Meaning | Use |
|---|---|---|
| Plain | Collection without category labels | Word counts, language modelling |
| Categorized | Documents grouped by genre/topic | Classification |
| Overlapping categories | One document has several categories | Multi-topic news |
| Temporal | Documents indexed by time | Language change |
| Annotated | Linguistic labels attached | Tagging, parsing, WSD |
| Parallel | Aligned texts in different languages | Translation |
A tagset is the allowed label inventory. A treebank is a corpus annotated with syntax trees. Metadata records source, date, genre, language, annotation method and limitations. A common benchmark supports comparison under shared data and evaluation conditions.
Raw example: “Riya works in Pune.” POS annotation: Riya/PROPN works/VERB in/ADP Pune/PROPN. NER annotation: Riya/PERSON, Pune/LOCATION. Each annotation answers a different task.
A good corpus is relevant to its domain, sufficiently representative, consistently annotated and appropriately split. More data alone do not repair duplicated examples, incorrect labels or a mismatch between training and real use.
Corpus=text collection. Annotation=text ke saath labels. Metadata=collection ki information.
| Resource | Contents | Typical use |
|---|---|---|
| Gutenberg selection | Literary/book texts | Long-form language and vocabulary |
| Web text | Informal domain-specific text | Study usage beyond edited books |
| Chat corpus | Conversational posts | Dialogue, informal forms |
| Brown | English texts across genres | Genre comparison |
| Reuters | News with topic labels | Text classification; overlapping topics |
| Inaugural | Speeches indexed by time | Vocabulary change across years |
| Wordlist | Words/names/stopwords | Lexical filtering |
| Pronouncing dictionary | Words and phonetic forms | Speech, stress, rhyme |
Gutenberg versus web/chat: book language does not represent every spelling or conversational pattern in messages. Brown versus Reuters: Brown emphasizes genres; Reuters news can have overlapping topic labels. Inaugural addresses: dates provide a time dimension.
Corpus sizes depend on version, subset and tokenization. Explain their purpose and structure rather than assuming a historical count is a permanent property.
Gutenberg=books; web/chat=informal; Brown=genres; Reuters=news topics; Inaugural=time.
Corpus readers expose several views. raw returns a character string, words returns token occurrences, and sents returns a list of sentences where each sentence contains tokens.
| Conceptual method | Returned view | Its length counts |
|---|---|---|
| fileids() | Document IDs | Files |
| raw(fileid) | Character string | Characters |
| words(fileid) | Token list | Token occurrences |
| sents(fileid) | Sentence lists | Sentences |
| categories() | Genre/topic labels | Categories |
| tagged_words(fileid) | Word–tag pairs | Annotated tokens |
Question. For sentence lists [the,cat,sleeps,.] and [the,dog,runs,.], count all tokens, sentences and lowercase alphabetic word types.
Given
Punctuation is included in token count but excluded from word types.
To find
For sentence lists [the,cat,sleeps,.] and [the,dog,runs,.], count all tokens, sentences and lowercase alphabetic word types.
Formula and why it applies
Count every token occurrence, then distinct alphabetic forms. Different counting rules give different totals.
Step 1. Token count
4+4=8.
Step 2. Sentence count
Two outer lists → 2.
Step 3. Types
{the,cat,sleeps,dog,runs} → 5.
Answer. 8 tokens, 2 sentences, 5 word types.
For “Cats run. Dogs sleep.”, a reader may return six tokens including two periods, but only four alphabetic word tokens. State the convention before calculating TTR or probabilities.
raw=characters, words=tokens, sents=sentences. Teen alag counts.