Natural Language Processing · TEE · papers
Q2a · Text corpus structure and applications
2023-12-01 · final · recovered AI paper
Previous examination · complete worked answers
Answer
A corpus is a collection of language data assembled for study or computational use. Its plural is corpora. A corpus can contain books, news, reviews or conversations. It need not be labelled. Annotation adds linguistic information such as POS tags, entities, parse structures or sense labels.
| Structure | Meaning | Use |
|---|---|---|
| Plain | Collection without category labels | Word counts, language modelling |
| Categorized | Documents grouped by genre/topic | Classification |
| Overlapping categories | One document has several categories | Multi-topic news |
| Temporal | Documents indexed by time | Language change |
| Annotated | Linguistic labels attached | Tagging, parsing, WSD |
| Parallel | Aligned texts in different languages | Translation |
A tagset is the allowed label inventory. A treebank is a corpus annotated with syntax trees. Metadata records source, date, genre, language, annotation method and limitations. A common benchmark supports comparison under shared data and evaluation conditions.
Raw example: “Riya works in Pune.” POS annotation: Riya/PROPN works/VERB in/ADP Pune/PROPN. NER annotation: Riya/PERSON, Pune/LOCATION. Each annotation answers a different task.
A good corpus is relevant to its domain, sufficiently representative, consistently annotated and appropriately split. More data alone do not repair duplicated examples, incorrect labels or a mismatch between training and real use.
Corpus=text collection. Annotation=text ke saath labels. Metadata=collection ki information.
| Resource | Contents | Typical use |
|---|---|---|
| Gutenberg selection | Literary/book texts | Long-form language and vocabulary |
| Web text | Informal domain-specific text | Study usage beyond edited books |
| Chat corpus | Conversational posts | Dialogue, informal forms |
| Brown | English texts across genres | Genre comparison |
| Reuters | News with topic labels | Text classification; overlapping topics |
| Inaugural | Speeches indexed by time | Vocabulary change across years |
| Wordlist | Words/names/stopwords | Lexical filtering |
| Pronouncing dictionary | Words and phonetic forms | Speech, stress, rhyme |
Gutenberg versus web/chat: book language does not represent every spelling or conversational pattern in messages. Brown versus Reuters: Brown emphasizes genres; Reuters news can have overlapping topic labels. Inaugural addresses: dates provide a time dimension.
Corpus sizes depend on version, subset and tokenization. Explain their purpose and structure rather than assuming a historical count is a permanent property.
Gutenberg=books; web/chat=informal; Brown=genres; Reuters=news topics; Inaugural=time.
Word-sequence counts support n-gram language models and smoothing. POS-annotated sentences support supervised taggers. Sense-labelled contexts support Naive Bayes WSD. Word-context counts support distributional semantics and GloVe. Parallel aligned sentences support translation.
For a question asking the importance of corpora, give three points: empirical language evidence, examples for training/evaluation, and reproducible comparison. For an annotated-corpus question, define the added labels and give a concrete example. For resource comparisons, distinguish running text, lexical lists, pronunciation entries and semantic relations.
Same text se task ke hisaab se counts, tags, senses ya context patterns nikaalte hain.