Corpus

A corpus is the complete body of text that a retrieval system searches over, treated as a single bounded collection regardless of how many files, databases, or sources contributed to it.

The word is inherited from linguistics and information retrieval, where it named a curated collection of texts assembled for study. The retrieval sense keeps the essential property — a defined boundary — and drops the curation. A corpus is whatever is in scope for search, and nothing else.

Boundedness is the whole point of naming it. Statements about a retrieval system are only meaningful relative to some collection: a query either has an answer in the corpus or it does not, and a measurement of how often answers are found is a measurement against one particular corpus. Two systems described with identical numbers may be operating over collections that differ by orders of magnitude in size and difficulty.

Plural is corpora, following the Latin. Corpuses appears and is not wrong, merely less common in technical writing.

The corpus is defined by content, not by storage. Text held in three databases, an object store, and a wiki is one corpus if one query can reach all of it, and several corpora if it cannot.

In practice

A corpus is rarely a static thing. In a working system it is described by a handful of properties that determine almost everything downstream:

Size — how many source documents, and how many chunks they became. The second number is the one that matters for index sizing; the ratio between them is a consequence of the chunking strategy.

Heterogeneity — whether the collection is one document type or twenty. A corpus of uniform support articles behaves very differently from one mixing contracts, spreadsheets, meeting transcripts, and scanned faxes, because each type wants different handling before it can be searched at all.

Churn — how much of the collection changes per day or week. A corpus that only grows is a simpler object than one where documents are revised and withdrawn.

Access boundaries — whether every reader may see every document. When they may not, the corpus is nominally one collection but effectively a different one per reader, and that difference has to be enforced somewhere in the retrieval path.

The term also appears in a narrower sense during development: an evaluation corpus or test corpus is a small collection assembled deliberately so that results over it are stable and comparable. It is a corpus in exactly the same sense, just chosen rather than inherited.

Commonly confused with

Knowledge base. A knowledge base is a corpus plus an implied claim about it — that its contents are authoritative, curated, and intended to answer questions. “Corpus” is neutral about quality and purpose; it is simply the set of text in scope. Every knowledge base is a corpus; most corpora are not knowledge bases.

Index. The corpus is the text; the index is the structure built over it to make search fast. One corpus may have several indexes built over it — a vector index and a keyword index, for instance — and rebuilding an index does not change the corpus. See vector index.

Dataset. Broadly interchangeable in casual use, but “dataset” carries a training connotation from machine learning, where it implies labelled examples used to fit a model. Text used for retrieval is not training data, and calling it a dataset invites the assumption that a model has learned it.

Document. A single item within the corpus. The confusion runs in one direction: people say “the documents” when they mean the corpus, which is harmless, and “the corpus” when they mean one document, which is not.

Context. What is actually placed in front of the model for a single request — a few chunks selected from the corpus. The corpus is the population; the context is the sample. See chunk.

Usage notes

Vendor documentation often avoids the word entirely, preferring product nouns: a collection, a namespace, a datastore, a knowledge source, a project. These are product boundaries and may or may not align with the corpus as defined here. When a service charges per collection, the number of collections is a billing decision, not a statement about how many corpora exist.

“Corpus” implies no ordering and no structure. It is a set. Relationships between documents — folder hierarchies, links, versions, supersession — exist in the source system and are not part of what the word conveys. Systems that depend on those relationships have to carry them as metadata explicitly.

The boundary is frequently unstated and frequently wrong. A system described as searching “our documentation” may in fact search a snapshot of part of it, taken on some date, excluding whatever failed to parse. The difference between the intended corpus and the actual one is a common and quiet source of missing answers.

In information-retrieval literature the word is used more strictly, often referring to a standard, published, fixed collection so that results are reproducible across research. That usage assumes stability; the engineering usage usually does not.

See also

Chunk · Vector index · Retriever · Upsert