Quantisation

Quantisation is the practice of storing each value of a vector in fewer bits than the model originally produced, reducing memory and comparison cost while leaving the number of values unchanged.

An embedding model emits its values at some precision — commonly a 32-bit floating-point number per dimension. Nothing about a retrieval system requires that precision to be preserved. Quantisation maps each value onto a coarser scale, so the vector still has the same length and the same meaning, expressed less exactly.

The operation is lossy and the loss is the point. What is given up is the ability to distinguish two very close values; what is gained is a smaller vector, and smaller vectors are cheaper to hold in memory and faster to compare. Because approximate search is already tolerant of small ranking errors, the two concessions compound tolerably rather than catastrophically.

The word also has a wider machine-learning meaning: reducing the precision of a model’s own weights so that the model itself is smaller and faster to run. That is a different object being quantised, and the two senses appear in the same documentation sets.

Spelled quantization in American usage. The two forms are identical in meaning.

In practice

Several distinct schemes carry the name, and they differ in what the stored representation is.

Scalar quantisation maps each value independently onto a smaller numeric type — a byte, for instance, instead of four. The vector keeps its shape and one value still corresponds to one dimension, which makes the transformation simple to reason about and to undo approximately.

Binary quantisation takes it to one bit per dimension, usually recording only whether each value is above or below a threshold. The reduction is severe, and the comparison becomes a bit operation rather than arithmetic.

Product quantisation does something structurally different: it splits the vector into segments and replaces each segment with the identifier of the nearest entry in a small learned codebook. The stored record is a list of codes, not a shortened vector, so its values no longer correspond to dimensions at all. The codebook has to be fitted to a sample of the collection, which means this family has a training step the others do not.

Two properties follow for any scheme. Compressed vectors are only comparable with vectors compressed the same way, using the same parameters or codebook, so the choice is a property of the index rather than of an individual record. And a query vector must be handled consistently with the stored ones, whether by compressing it too or by comparing at full precision against decompressed candidates.

A pattern common enough to name: search the compressed vectors to gather a wider candidate set, then rescore those candidates against full-precision vectors kept elsewhere. The cheap representation decides what is considered and the exact one decides the order.

Commonly confused with

Dimensionality reduction and truncation. These cut the number of values while keeping the precision of each. Quantisation keeps the count and cuts the precision. Both shrink a vector, they are independent operations, and they can be applied together. See dimensionality.

Model quantisation. Reducing the precision of a neural network’s weights, which changes what the model computes and how fast it runs. Quantising an embedding model’s weights alters the vectors it produces; quantising its output alters only how those vectors are stored. Both are correctly called quantisation.

Compression, generally. Lossless compression recovers the original exactly and does not permit comparison in the compressed form. Quantisation is lossy and its whole purpose is to be compared without decompressing.

Approximate nearest neighbour search. The problem quantisation is usually deployed in service of, not a form of it. Skipping most of the collection and storing what remains coarsely are separate savings, and they are commonly combined. See approximate nearest neighbour.

Normalisation. Rescaling a vector to unit length, which is exact and reversible in its effect on ranking. It changes magnitudes, not precision. See cosine similarity.

Usage notes

The name is used for the setting as well as the operation. A configuration field called quantization typically selects a scheme, and the schemes it offers differ enough that the field’s value matters more than the fact that it is set.

Accuracy cost is collection-dependent and not predictable from the scheme alone. How much recall a given scheme gives up depends on the embedding model, the dimension count, and how the vectors are distributed, so a figure measured on one collection is not a figure about the scheme. Published numbers in vendor material describe particular tests.

Some models are trained to survive it. Where a model is documented as supporting reduced-precision or binary output, the loss is smaller than the same scheme applied to a model that was not. This is a property of the model, not of the quantiser.

“Quantised index” says nothing about whether full-precision vectors were kept. Some systems discard the originals, making the compression irreversible; others retain them for rescoring or rebuilds. The distinction determines whether the choice can be revisited. See vector index.

See also

Dimensionality · Vector index · Approximate nearest neighbour · Embedding