Natural Language Processing · TEE · papers
Q6a · Corpora and acquisition
2025-02-21 · re-exam · recovered AI paper
Previous examination · complete worked answers
Answer
A corpus is a collection of language data assembled for study or computational use. Its plural is corpora. A corpus can contain books, news, reviews or conversations. It need not be labelled. Annotation adds linguistic information such as POS tags, entities, parse structures or sense labels.
| Structure | Meaning | Use |
|---|---|---|
| Plain | Collection without category labels | Word counts, language modelling |
| Categorized | Documents grouped by genre/topic | Classification |
| Overlapping categories | One document has several categories | Multi-topic news |
| Temporal | Documents indexed by time | Language change |
| Annotated | Linguistic labels attached | Tagging, parsing, WSD |
| Parallel | Aligned texts in different languages | Translation |
A tagset is the allowed label inventory. A treebank is a corpus annotated with syntax trees. Metadata records source, date, genre, language, annotation method and limitations. A common benchmark supports comparison under shared data and evaluation conditions.
Raw example: “Riya works in Pune.” POS annotation: Riya/PROPN works/VERB in/ADP Pune/PROPN. NER annotation: Riya/PERSON, Pune/LOCATION. Each annotation answers a different task.
A good corpus is relevant to its domain, sufficiently representative, consistently annotated and appropriately split. More data alone do not repair duplicated examples, incorrect labels or a mismatch between training and real use.
Corpus=text collection. Annotation=text ke saath labels. Metadata=collection ki information.
| Resource | Contents | Typical use |
|---|---|---|
| Gutenberg selection | Literary/book texts | Long-form language and vocabulary |
| Web text | Informal domain-specific text | Study usage beyond edited books |
| Chat corpus | Conversational posts | Dialogue, informal forms |
| Brown | English texts across genres | Genre comparison |
| Reuters | News with topic labels | Text classification; overlapping topics |
| Inaugural | Speeches indexed by time | Vocabulary change across years |
| Wordlist | Words/names/stopwords | Lexical filtering |
| Pronouncing dictionary | Words and phonetic forms | Speech, stress, rhyme |
Gutenberg versus web/chat: book language does not represent every spelling or conversational pattern in messages. Brown versus Reuters: Brown emphasizes genres; Reuters news can have overlapping topic labels. Inaugural addresses: dates provide a time dimension.
Corpus sizes depend on version, subset and tokenization. Explain their purpose and structure rather than assuming a historical count is a permanent property.
Gutenberg=books; web/chat=informal; Brown=genres; Reuters=news topics; Inaugural=time.
Data acquisition obtains language examples relevant to the task from existing data, public resources or appropriate new collection. Product instrumentation can gather useful feedback. Supervised learning additionally needs an annotation process when labels are absent.
Data augmentation creates training variants intended to preserve the correct label. It is not simply changing words arbitrarily.
| Method | Procedure | Caution |
|---|---|---|
| Synonym replacement | Use suitable synonyms for selected words | Choose the right sense |
| Back translation | Translate to another language and back | May change meanings/entities |
| TF–IDF-informed replacement | Protect informative words while varying others | Do not remove label-bearing clues |
| Bigram flipping | Swap adjacent words | Can break grammar or meaning |
| Entity replacement | Replace by same-type entity | Must preserve the task label |
| Noise injection | Simulate typos/keyboard errors | Match expected noise |
| Active learning | Select informative cases for human labels | This acquires labels; it is not synthetic text |
Three label decisions
“The film is good”→“The film is excellent” may preserve a positive sentiment label. “The film is not good”→“The film is good” reverses the label and is not valid label-preserving augmentation. “Flights to Pune”→“Flights to Mumbai” may preserve a travel-topic label but changes the destination label.
Split train/test data before creating related augmented variants. Otherwise near-duplicates can leak between sets and produce misleading evaluation. Keep a representative evaluation set of genuine examples.
Sentence badal sakta hai; required correct label nahi. Label badla toh augmentation invalid.
Corpus structure diagram
Corpus ├─ Documents (plain collection) │ └─ Sentences → tokens ├─ Categories → document IDs (one or overlapping labels) ├─ Dates → document IDs (temporal collection) ├─ Annotations → token spans / tags / parse trees └─ Parallel pairs → aligned source and target sentences
An annotated corpus attaches linguistic labels to the raw text: e.g. Rita/PROPN runs/VERB. A wordlist is a lexical inventory without full discourse context; a Gutenberg corpus contains books; web/chat corpora contain informal online language. Their structures determine which evidence can be queried.