Blog
-
Document
In retrieval, a document is whatever unit the system treats as one retrievable item — which may be a file, a section, or a single chunk.
-
Document Loader
The component that reads a source file or system and returns its text plus metadata, before any chunking or embedding happens.
-
Faithfulness
The property of a generated answer whose every claim is supported by the retrieved context — regardless of whether those claims are true of the world.
-
Vector Normalisation
Rescaling a vector to unit length while keeping its direction, which makes dot product and cosine similarity produce the same ranking.
-
HNSW
Hierarchical Navigable Small World: a layered graph index for approximate nearest-neighbour search, walked from a sparse top layer down to a dense one.
-
Inverted Index
A structure mapping each term to the list of documents containing it, so a keyword query examines only documents that hold at least one query term.
-
Query Rewriting
Transforming the query the user typed into one or more different queries before searching, because the original is often a poor search input.
-
Context Window
The maximum amount of text, measured in tokens, that a language model can process in a single request — input and output together.
-
Reranking
A second scoring pass that reorders an already-retrieved candidate set using a slower, more accurate model that reads query and document together.
-
Vector Index
A data structure that stores embeddings and finds the ones nearest to a query vector without comparing against every stored vector.
-
Chunk
A unit of text stored and retrieved as a single item in a retrieval system — the smallest thing a search can return. No standard size exists.
-
Embedding
A fixed-length list of numbers representing a piece of text, produced so that texts with similar meanings land near one another in the space.
-
Hallucination
Model output that is presented as fact but is not supported by the supplied sources or by anything else — fluent, confident, and unfounded.
-
Grounding
Tying a model's output to specific supplied source material, so each claim in the answer traces back to text the system provided rather than to training.
-
Cross-Encoder
A model that takes a query and a document as one joined input and outputs a relevance score, rather than encoding each into a comparable vector.
-
Quantisation
Storing each value of a vector in fewer bits than the model produced, cutting memory and comparison cost while keeping the number of values the same.
-
Upsert
A write that inserts a record if its identifier is new and replaces the existing record if it is not — the usual way vectors enter an index.
-
Retriever
The component that takes a query and returns the passages a system should read — an interface, not a single algorithm or data structure.
-
Bi-Encoder
A model arrangement that encodes query and document separately into vectors that are compared afterwards, so document vectors can be computed in advance.
-
Approximate Nearest Neighbour
The problem of finding the items closest to a query point while allowing the answer to be slightly wrong, and the index structures that solve it.
-
Top-k
The setting that fixes how many results a retrieval step returns: the k highest-scoring items, and nothing about whether any of them are good.
-
Hybrid Search
Retrieval that runs a keyword method and a vector method over the same collection and merges their two ranked lists into one result set.
-
BM25
A ranking function that scores a document against a query from term frequency, term rarity, and document length — the standard keyword-search baseline.
-
Sparse Retrieval
Retrieval that scores text by the terms it shares with the query, using a representation with one position per vocabulary term, almost all of them empty.
-
Dense Retrieval
Retrieval that compares learned vectors in which every value is used, matching query to text by meaning rather than by shared terms.
-
Cosine Similarity
A measure of how closely two vectors point in the same direction, ignoring their length — the usual way embedding similarity is scored.
-
Semantic Search
Search that matches on meaning rather than on shared words, so a query can retrieve a relevant passage with which it has no vocabulary in common.
-
Dimensionality
The number of values in an embedding vector — fixed by the model that produced it, and the same for every input that model ever sees.
-
Token
The unit of text a language model actually processes — a word, word-fragment, or character sequence produced by that model's tokeniser.
-
Corpus
The complete body of text a retrieval system searches over, treated as one bounded collection regardless of how many files or sources it came from.