Document

In retrieval, a document is whatever unit the system treats as one retrievable item, which may be an entire file, a section of one, or a single chunk of a few hundred words.

The word is inherited from information retrieval, where it named the thing a query returned and carried no commitment about size. A document could be a newspaper article, an abstract, or a whole book, and the theory was indifferent: what mattered was that the collection was a set of documents and the system ranked them. That indifference survives in the vocabulary and is the source of most of the confusion around the term today.

Three meanings are in simultaneous use. There is the source artefact — the PDF, the wiki page, the support ticket as it exists before any processing. There is the retrieval unit — the thing that gets scored and returned, which in a chunked system is a fragment of the artefact. And there is the record — a row in a store holding a vector and its metadata, which many libraries call a document regardless of what it contains.

Nothing in the word signals which meaning is intended. A sentence stating that a system holds forty thousand documents may describe forty thousand files or the chunks of four hundred of them, and a sentence stating that a retriever returned five documents almost always means five chunks.

The ambiguity has consequences because the three meanings have different cardinalities. Counts, hit rates, and deduplication all depend on which unit is being counted, and a figure computed over one unit is wrong when read as the other.

In practice

In most retrieval pipelines the source artefact and the retrieval unit are different objects, and the pipeline converts one into many of the other. The conversion is where the vocabulary breaks: the input to chunking is a document, the output is a list of chunks, and the store those chunks are written to calls each one a document.

Framework interfaces make this concrete. A loader produces objects named documents, one per file; a splitter consumes them and produces more objects of the same type, one per fragment. The type does not change, so nothing in the code distinguishes a whole file from a piece of one, and the word covers both stages of the pipeline. See document loader.

The source artefact usually survives as metadata rather than as a stored object. Each chunk records which document it came from and where in it, and that identifier is what allows results to be grouped, cited, or deduplicated back to the file level. Without it a retrieved passage cannot be located in its original context. See chunk.

Which unit a query is scored against decides what the ranking means. Scoring whole documents ranks by relevance of a whole file, which favours long files containing many topics; scoring chunks ranks by relevance of a passage, which can return five fragments of one file and nothing else. The behaviour that looks like a retrieval quality problem is often a choice of unit.

Access control attaches to the source artefact and applies to the retrieval unit, which is why the identifier linking them is load-bearing beyond citation. A chunk whose parent document is no longer readable by a requester is still a retrievable vector.

Commonly confused with

Chunk. The fragment produced by splitting. Every chunk is called a document by some part of the stack, and no document is necessarily a chunk. When a system’s counts and its user interface disagree, this pair is usually why. See chunk.

File. The artefact on disk. One file can yield several documents in the source sense — an archive, a spreadsheet with multiple sheets, an email thread — and one document can span several files. The two coincide often enough that the distinction is noticed only when it fails.

Record. The stored row: identifier, vector, metadata. Named a document by most vector-store interfaces, a point or an entry by others. It is a storage object and its contents are whatever was written, so calling it a document says nothing about the text it holds. See upsert.

Passage. A term used, mainly in research writing, precisely to avoid this ambiguity: a passage is explicitly a span of text shorter than a document. Material using both words consistently is distinguishing the retrieval unit from the artefact on purpose.

Corpus. The whole collection, not one member of it. A corpus is described as a number of documents, and which meaning of document that number uses is frequently unstated. See corpus.

Usage notes

The word is the field’s most overloaded and the overloading is rarely flagged. Documentation, interfaces, and papers each use it for a different unit, often within one paragraph, and no convention has emerged to separate them.

“Document” in classical information retrieval carries no size and should not be read as implying a whole file. Formulas expressed over documents — term frequencies, document frequencies, length normalisation — apply to whatever unit the index holds, which in a chunked system is the chunk. See BM25.

Counts stated in documents are ambiguous by default and the ratio between the meanings is commonly one to tens. A system described as indexing ten thousand documents may be holding a few hundred files.

Some writing uses “parent document” and “child document” for the artefact and its fragments, which resolves the ambiguity locally but is not standard and is easily mistaken for a hierarchy in the source material itself.

In multi-modal systems the word stretches further, covering images, audio transcripts, and table rows as retrievable items. The retrieval sense of the term survives the stretch; the intuition of a document as prose does not.

See also

Chunk · Corpus · Document loader · Upsert · Retriever