# Q5a · Corpora and structure

Subject: Natural Language Processing

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f22/q5a

Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q5a/

Previous examination · complete worked answers

**Question. Define a corpus. Explain corpus types, characteristics and structure.** [10 marks]

### Answer

**A corpus is a collection of language data assembled for study or computational use.** Its plural is corpora. A corpus can contain books, news, reviews or conversations. It need not be labelled. **Annotation** adds linguistic information such as POS tags, entities, parse structures or sense labels.

| Structure | Meaning | Use |
| --- | --- | --- |
| Plain | Collection without category labels | Word counts, language modelling |
| Categorized | Documents grouped by genre/topic | Classification |
| Overlapping categories | One document has several categories | Multi-topic news |
| Temporal | Documents indexed by time | Language change |
| Annotated | Linguistic labels attached | Tagging, parsing, WSD |
| Parallel | Aligned texts in different languages | Translation |

A **tagset** is the allowed label inventory. A **treebank** is a corpus annotated with syntax trees. **Metadata** records source, date, genre, language, annotation method and limitations. A common benchmark supports comparison under shared data and evaluation conditions.

**Raw example:** “Riya works in Pune.” **POS annotation:** Riya/PROPN works/VERB in/ADP Pune/PROPN. **NER annotation:** Riya/PERSON, Pune/LOCATION. Each annotation answers a different task.

A good corpus is relevant to its domain, sufficiently representative, consistently annotated and appropriately split. More data alone do not repair duplicated examples, incorrect labels or a mismatch between training and real use.

Corpus=text collection. Annotation=text ke saath labels. Metadata=collection ki information.

| Resource | Contents | Typical use |
| --- | --- | --- |
| Gutenberg selection | Literary/book texts | Long-form language and vocabulary |
| Web text | Informal domain-specific text | Study usage beyond edited books |
| Chat corpus | Conversational posts | Dialogue, informal forms |
| Brown | English texts across genres | Genre comparison |
| Reuters | News with topic labels | Text classification; overlapping topics |
| Inaugural | Speeches indexed by time | Vocabulary change across years |
| Wordlist | Words/names/stopwords | Lexical filtering |
| Pronouncing dictionary | Words and phonetic forms | Speech, stress, rhyme |

**Gutenberg versus web/chat:** book language does not represent every spelling or conversational pattern in messages. **Brown versus Reuters:** Brown emphasizes genres; Reuters news can have overlapping topic labels. **Inaugural addresses:** dates provide a time dimension.

Corpus sizes depend on version, subset and tokenization. Explain their purpose and structure rather than assuming a historical count is a permanent property.

Gutenberg=books; web/chat=informal; Brown=genres; Reuters=news topics; Inaugural=time.

Corpus readers expose several views. **raw** returns a character string, **words** returns token occurrences, and **sents** returns a list of sentences where each sentence contains tokens.

| Conceptual method | Returned view | Its length counts |
| --- | --- | --- |
| fileids() | Document IDs | Files |
| raw(fileid) | Character string | Characters |
| words(fileid) | Token list | Token occurrences |
| sents(fileid) | Sentence lists | Sentences |
| categories() | Genre/topic labels | Categories |
| tagged\_words(fileid) | Word–tag pairs | Annotated tokens |

**Question. For sentence lists [the,cat,sleeps,.] and [the,dog,runs,.], count all tokens, sentences and lowercase alphabetic word types.**

#### Given

Punctuation is included in token count but excluded from word types.

#### To find

For sentence lists [the,cat,sleeps,.] and [the,dog,runs,.], count all tokens, sentences and lowercase alphabetic word types.

#### Formula and why it applies

Count every token occurrence, then distinct alphabetic forms. Different counting rules give different totals.

#### Step 1. Token count

4+4=8.

#### Step 2. Sentence count

Two outer lists → 2.

#### Step 3. Types

{the,cat,sleeps,dog,runs} → 5.

**Answer.** 8 tokens, 2 sentences, 5 word types.

For “Cats run. Dogs sleep.”, a reader may return six tokens including two periods, but only four alphabetic word tokens. State the convention before calculating TTR or probabilities.

raw=characters, words=tokens, sents=sentences. Teen alag counts.