BM25

BM25 is a ranking function that scores how well a document matches a query using three inputs: how often each query term appears in the document, how rare that term is across the collection, and how long the document is relative to the average.

It is a formula, not a system and not an index. Given those counts it produces a number; everything about storing text and finding candidates happens elsewhere. This is why BM25 appears as a selectable scoring option inside search engines rather than as a product.

The name abbreviates “Best Matching,” and the 25 identifies one formulation within a numbered family developed in the probabilistic-retrieval tradition. It is sometimes written Okapi BM25 after the early system it was implemented in.

Its durability comes from getting three behaviours right at once. Repeated terms increase the score but with diminishing returns, so a document mentioning a word fifty times does not overwhelm one mentioning it five. Rare terms count for more than common ones. Long documents are penalised for the extra opportunities their length gives them to contain a term by accident.

BM25 is the standard baseline against which retrieval methods are compared, and it is a demanding one.

In practice

The formula sums a contribution per query term. Each contribution combines the term’s rarity in the collection with a saturating function of its frequency in the document, adjusted by the document’s length.

Two parameters control the shape, and their conventional names are near-universal across implementations:

one governing how quickly repeated occurrences stop adding value — set low, a second occurrence adds almost nothing; set high, frequency counts more nearly linearly
one governing how strongly document length is normalised — at zero, length is ignored entirely; at one, it is fully corrected for

Both have widely used default values that most systems ship and few systems change. Tuning them matters mainly on collections with unusual length distributions.

BM25 is computed at query time over candidates gathered from an inverted index, so its cost scales with how many documents contain the query’s terms rather than with collection size. It requires collection-wide statistics — how many documents exist and in how many each term appears — which means scores shift as the corpus grows, and identical query-document pairs can score differently before and after an ingestion run.

In retrieval-augmented systems it usually appears as the keyword half of a hybrid setup, contributing exact-match capability that embeddings lack.

Commonly confused with

TF-IDF. The earlier and simpler weighting that BM25 refines. TF-IDF multiplies term frequency by inverse document frequency with no saturation and no principled length normalisation, so a heavily repeated term can dominate. BM25 keeps the same two ingredients and bounds their effect. Not interchangeable, though both are described as “term weighting.”

Keyword search. BM25 is one way to rank keyword search results. Keyword search also involves tokenising, normalising, and matching, none of which BM25 performs. A system can do keyword search with a different scorer, or with none.

Inverted index. The data structure listing where each term occurs. It finds candidates; BM25 orders them. The two are always used together and are frequently described as one thing.

Sparse retrieval. The category BM25 belongs to. Sparse retrieval names the family of term-based methods; BM25 names a specific scoring function within it. See sparse retrieval.

Relevance score. BM25 outputs an unbounded, collection-relative number. It is not a probability, not a percentage, and not comparable across queries or corpora, despite being displayed alongside scores that are bounded. See cosine similarity.

Usage notes

Variants differ and are all called BM25. Implementations vary in how they handle the term-rarity component for very common terms, whether they support per-field weighting, and how they treat terms absent from the collection. Two systems can return different scores for the same query and document while both being correct.

BM25F is a distinct extension, not a rename: it scores across multiple weighted fields such as title and body rather than treating a document as one block of text. Documentation sometimes says BM25 while configuring BM25F.

Scores are not portable and thresholds do not travel. Because the formula uses collection statistics, a cut-off tuned on one index is meaningless on another, and meaningless on the same index after substantial growth.

It is often used as shorthand for “the keyword side.” In discussions of hybrid retrieval, “BM25” frequently stands in for lexical matching in general, whether or not that specific function is the one configured. See hybrid search.

In practice

Commonly confused with

Usage notes

See also