Faithfulness

Faithfulness is the property of a generated answer in which every claim it makes is supported by the text supplied to the model, judged without reference to whether those claims are true of the world.

The restriction is the whole point of the term. Faithfulness compares an answer to its sources and to nothing else. An answer that repeats an error present in a retrieved document is faithful. An answer that states a correct fact the retrieved documents do not contain is unfaithful. The property is about the relationship between two texts, not about the state of the world.

That makes it measurable in a way correctness is not. Establishing whether a claim is true requires knowing the truth; establishing whether a claim is supported requires only the passages the model was given. Faithfulness can therefore be checked by a process that has no knowledge of the subject, which is why it became the standard property to assess in retrieval systems.

The unit of judgement is the claim rather than the answer. An answer is decomposed into the individual assertions it makes, each assertion is checked against the context, and the result is expressed as the proportion supported. An answer can consequently be partly faithful, and usually is: a summary drawn from three passages with one unsupported sentence appended is not simply unfaithful.

Support is not the same as verbatim presence. A claim entailed by the context — implied by it, inferable from it without additional information — is supported. Where the line falls between entailment and inference that goes beyond the source is the term’s principal ambiguity, and different assessments draw it differently.

In practice

Faithfulness is the property a retrieval system is built to obtain and the one its failures are described against. An unfaithful answer is the visible symptom of most things going wrong downstream of retrieval, and of some things going wrong in retrieval itself: given context that does not contain the answer, a model asked to answer anyway produces claims from elsewhere.

The property is assessed on the pair of the answer and the context actually assembled into the prompt, not the whole corpus and not the retrieved candidate list. A claim supported by a document that was retrieved but truncated out of the prompt is unfaithful, because the model did not receive the support. This makes the assembled prompt the reference text, and it means a change to prompt assembly can change an answer’s faithfulness without changing the answer. See context window.

Three distinct failures fall under unfaithfulness and have different causes. A claim contradicting the context is one; a claim absent from the context but consistent with it is another; and an answer attributing a claim to the wrong source is a third. Only the first is detectable by looking for contradictions, which is why enumerating the categories precedes assessing the property. See hallucination.

Faithfulness says nothing about whether the answer addresses the question. A response that accurately paraphrases an irrelevant passage is fully faithful and useless, so the property is always reported alongside something that measures relevance to the query.

Commonly confused with

Grounding. Grounding is the arrangement — supplying retrieved text and instructing the model to answer from it. Faithfulness is whether the resulting answer actually stayed within that text. A system can be grounded and produce an unfaithful answer, which is the case that matters; the terms are not interchangeable, though they are frequently used as if they were. See grounding.

Correctness. Whether the answer is true. Faithfulness and correctness come apart in both directions, and the pair of an unfaithful correct answer and a faithful incorrect one is the standard illustration. A system with a wrong document in its corpus can be perfectly faithful and consistently wrong.

Hallucination. The failure, and a broader one. Unfaithfulness is hallucination measured against a supplied context; hallucination is also used for fabrication by a model with no context at all, where there is nothing to be faithful to.

Answer relevance. Whether the answer responds to the question. Orthogonal to faithfulness: the two are commonly assessed as separate quantities precisely because an answer can satisfy either without the other.

Attribution. Whether the citations an answer carries point at the passages that support its claims. An answer can be faithful with wrong citations — every claim supported somewhere in the context, but by different passages than the ones named.

Usage notes

“Faithfulness” and “groundedness” are used for the same property by different tooling and different writing, with no consistent distinction between them. Some material reserves groundedness for the presence of supporting text and faithfulness for the absence of contradiction; most uses both for the whole property. The distinction should be read from context rather than assumed.

A faithfulness score is a number produced by a particular judging procedure, and procedures differ in how they split an answer into claims and in what they count as support. Scores from two procedures are not comparable, and the same answer can receive different scores from each.

The term is sometimes used loosely for answer quality overall, especially in vendor material, where “more faithful” means “better”. The precise sense excludes relevance, fluency, and completeness entirely.

In older writing on summarisation, faithfulness means the same thing — a summary consistent with its source document. The retrieval sense is that sense with the retrieved context in place of the source, and material from the summarisation literature transfers with the vocabulary intact.

Perfect faithfulness is achievable trivially. An answer that quotes the context and adds nothing cannot be unfaithful, which is why the property is never optimised alone. See reranking.

See also

Grounding · Hallucination · Context window · Retriever · Chunk