Chunk
A chunk is a unit of text that a retrieval system stores, embeds, and returns as a single item — the smallest thing a search can hand back.
Documents are usually too long to be useful as retrieval units. A hundred-page manual embedded as one vector produces a representation so averaged-out that it is close to nothing in particular, and returning the whole manual to answer one question wastes the model’s context. So documents are split, and each piece becomes a chunk.
There is no standard chunk. It might be a paragraph, a section under one heading, a fixed count of tokens, or an arbitrary span with an overlap into its neighbours. The choice is called a chunking strategy, and it is a per-corpus decision rather than a solved one.
The defining property is granularity: a chunk should be small enough to be specifically about something, and large enough to make sense when read alone. Those two pressures oppose each other, and every chunking decision is a position between them.
In practice
Chunking happens once, at ingestion, before embedding. Each chunk is stored with:
- its text
- its embedding
- a reference to the source document
- usually its position, headings, and metadata such as date or permissions
Common strategies, roughly in order of how often they appear:
Fixed-size with overlap. Split every N tokens, with the last few dozen tokens of each chunk repeated at the start of the next. Simple, predictable, indifferent to document structure. The overlap exists so that a sentence spanning a boundary still appears intact somewhere.
Structural. Split on headings, paragraphs, list items, or Markdown/HTML boundaries. Produces chunks that align with how the document was written, at the cost of highly variable size.
Recursive. Try successively finer separators — sections, then paragraphs, then sentences — until pieces are under a size limit. A compromise between the first two, and a common default.
Semantic. Split where the topic shifts, detected by comparing embeddings of adjacent sentences. More expensive at ingestion, sometimes better boundaries.
Whatever the strategy, chunk boundaries determine what can be retrieved. If a question’s answer requires a condition at the end of one chunk and its exception at the start of the next, no retrieval method can return a chunk that answers it. That failure is invisible at query time and only shows up by inspecting the stored text.
Commonly confused with
Document. A document is the source item — a file, a page, a database record. A chunk is a piece of one. The confusion is entrenched because many libraries and vector databases call their stored unit a “document” regardless of whether it is a whole document or a fragment. When reading tooling documentation, assume “document” means “the thing you stored” and check what size that actually is.
Node. Some frameworks use “node” for what this entry calls a chunk, usually to signal that the unit carries relationships to other units — parent document, previous and next sibling. Effectively a chunk with graph metadata.
Passage. Largely a synonym, inherited from information-retrieval literature, where passage retrieval named the task of returning a relevant span rather than a whole document. “Passage” tends to appear in research writing and “chunk” in engineering writing.
Token. A token is a model-level unit of text, typically a word or word-fragment. Chunks are measured in tokens; they are not made of them in any structural sense. A chunk size of 512 means 512 tokens.
Context window. The total amount the model can process in one request. Several chunks are placed inside a context window. See context window.
Usage notes
No agreed default size exists, despite the frequency with which round numbers like 512 or 1,000 tokens appear in tutorials. Appropriate size depends on document type, query type, and the embedding model’s own trained input length — chunks longer than what the model was trained on are truncated or degraded, often silently.
Chunk size interacts with everything downstream. Smaller chunks are more precise but need higher retrieval counts to cover an answer; larger chunks carry more context but dilute the embedding and consume more of the generation prompt. It is not an isolated tuning knob.
“Chunking” occasionally refers to something else entirely — HTTP chunked transfer encoding, or the psychological sense of grouping information. In retrieval contexts the meaning here is unambiguous, but the word is common enough elsewhere to produce mismatched search results.
Parent-document and small-to-big patterns blur the definition deliberately: the system embeds and searches over small chunks but returns a larger surrounding span to the model. In such systems the retrieval unit and the returned unit differ, and “chunk” may refer to either. Worth clarifying which is meant when the distinction matters.