Token
A token is the unit of text a language model actually processes: a word, a word-fragment, a punctuation mark, or a run of characters, produced by splitting text with the model’s own tokeniser.
Models do not read characters and do not read words. Before any text reaches a model it is converted into a sequence of integers, each standing for one entry in a fixed vocabulary. Those entries are the tokens, and the vocabulary is decided once, when the model is built.
Vocabulary entries are chosen so that common sequences get their own token and rare ones are assembled from smaller pieces. Frequent English words are usually a single token. Unusual words, names, and misspellings break into several. A leading space is normally part of the token, which is why the same word can have two different representations depending on whether it starts a sentence.
The practical consequence is that the number of tokens in a passage cannot be derived from its length by any reliable rule. It has to be counted with the tokeniser that the target model uses.
Tokens are also the unit of billing and the unit of limits. Context windows are measured in tokens, API prices are quoted per token, and rate limits are often expressed in tokens per minute.
In practice
Token counts vary substantially by content type, and the variation is systematic rather than random.
Ordinary English prose is the case tokenisers are most efficient at, because common English words dominate most vocabularies. Text drifts away from that best case in predictable directions:
- Code tokenises less efficiently — identifiers split, indentation and punctuation consume tokens.
- Tables and structured data spend tokens on delimiters and repeated field names rather than content.
- Languages written in other scripts may use several tokens per character where the vocabulary was built mostly from English text.
- Identifiers, hashes, and long numbers fragment heavily, since no part of them is frequent enough to earn a vocabulary entry.
A rough figure sometimes quoted for English prose is on the order of three-quarters of a word per token, but it is an average over one kind of text and should not be applied to a mixed corpus without checking.
In a retrieval pipeline, token counts appear at three points: measuring chunk size at ingestion, budgeting the assembled prompt against the context window, and accounting for cost. All three use the same unit, and all three are model-specific — a chunk sized to 512 tokens under one tokeniser is not 512 tokens under another.
Commonly confused with
Word. A word is a linguistic unit; a token is an engineering one. They coincide often enough in English to be mistaken for the same thing, and diverge exactly where it matters — on names, technical terms, and anything not in the vocabulary. Counting words gives an estimate of tokens, never a figure.
Character. Some models operate at or near character level, but most do not. Character counts are the cheapest proxy for token counts and the least accurate; the error is largest on the content types listed above.
Chunk. A chunk is a retrieval unit, measured in tokens but not composed of them in any structural sense. Chunks are chosen by a chunking strategy for the purpose of retrieval; tokens are chosen by a tokeniser for the purpose of feeding a model. See chunk.
Embedding. A passage of many tokens produces one embedding vector, not one per token. Internally the model computes a representation per token and pools them, which is why the misconception is easy to arrive at. See embedding.
Token, the security sense. An API token or access token is an unrelated credential. In documentation that covers both authentication and model usage, the same word appears with both meanings on the same page.
Usage notes
Tokenisers are model-specific and versioned. Models from the same family may share one, models from different families almost never do. Any token count is a count under a particular tokeniser, and the number should be treated as attached to that model.
“Tokens” in a price or a limit may mean input, output, or both. Input and output tokens are frequently priced differently and counted separately, so a single figure quoted without saying which is ambiguous. See context window.
Tokenisation is lossless and reversible, unlike embedding. The token sequence can be decoded back to the original text exactly. This distinguishes it from every other representation change in a retrieval pipeline.
“Tokenisation” also names a step in classical keyword search, where text is split into terms before indexing. That process serves a different purpose, uses different rules, and produces different units from a model’s tokeniser, despite the shared name. See BM25.