Document Loader
A document loader is the component that reads a source — a file, a database, a web page, an API — and returns its text content together with metadata about where the text came from, before any splitting or embedding takes place.
It is the first stage of an ingestion pipeline and the only stage that touches the original format. Everything after it operates on plain text, so the loader’s job is to remove format from the problem: a PDF, a spreadsheet, and an HTML page arrive as three different things and leave as the same kind of object.
The term is a framework term rather than a concept from information retrieval. It names a position in a library’s pipeline, and libraries that structure ingestion differently do not have loaders at all — they have parsers, extractors, readers, or connectors. The word describes an architecture, not a necessary component.
Two outputs matter equally. The text is the obvious one. The metadata — source path or URL, page or row number, modification time, whatever the format exposes — is what allows a retrieved fragment to be attributed later, and it can only be captured here, because the format that carries it is discarded immediately afterwards.
A loader’s fidelity is bounded by what the source format makes recoverable. Reading order in a multi-column PDF, the association between a table cell and its header, whether a heading is a heading: these are questions about the format, and a loader either answers them or silently flattens them.
In practice
A pipeline commonly holds many loaders and one dispatch step that chooses among them by file type. The loaders are interchangeable in interface and not in behaviour, so which one handled a file determines what its text looks like downstream.
The output is usually one object per source and not one per unit of meaning. A hundred-page PDF may arrive as a single object with all its text concatenated, or as a hundred objects one per page, depending on the loader. Page-per-object loaders impose a boundary that the splitter then has to work within, so the loader’s granularity constrains chunking whether or not that was intended. See chunk.
Formats without extractable text require a step the loader either includes or delegates. A scanned page contains an image, and the character recognition that turns it into text is a separate process with its own error behaviour; where it sits — inside the loader or before it — varies by implementation.
Loaders for systems rather than files handle pagination, authentication, and rate limits, which makes them closer to API clients than to parsers. These are commonly called connectors to mark the difference, and their failure modes are network failure modes rather than parsing ones.
What the loader emits is what the rest of the pipeline believes the document to be. Text lost at this stage is not recoverable later and produces no error: a document whose body sat in an unread element is indexed as an empty or near-empty record and simply never retrieved. See corpus.
Commonly confused with
Parser. The narrower operation of interpreting one format’s bytes into structure. A loader typically wraps a parser and adds source handling, metadata capture, and the output type the pipeline expects. Material that uses the words interchangeably is usually describing a loader that does nothing but parse.
Splitter. The next stage: it divides loaded text into chunks. Loading and splitting are distinct operations that libraries present in sequence, and the confusion is encouraged by both producing objects of the same type. A loader that returns one object per page has not chunked the document, though the result resembles it. See document.
Indexing. The whole ingestion process, of which loading is the first step. A statement that documents were indexed does not indicate that they loaded correctly, and an empty extraction indexes without complaint. See upsert.
Embedding model. A different component entirely, applied to chunks after splitting. Some library entry points accept a file path and perform loading, splitting, and embedding in one call, which hides the three stages behind one name. See embedding.
Connector. Frequently a synonym, sometimes reserved for loaders that read from live systems rather than files. Where both terms appear in one library the distinction is usually source type rather than behaviour.
Usage notes
The term belongs to specific frameworks and is not universal. Pipelines built without those frameworks perform the same work under other names, and material describing a system as having “no document loader” may mean it has no such abstraction rather than that it does not read files.
Loader names imply capability more strongly than they deliver it. A loader named for a format handles that format to the extent its underlying library does, and two loaders for the same format commonly produce different text from the same file. Which one ran is part of what a corpus contains.
“Loader” sometimes covers the whole ingestion call in high-level interfaces that accept a directory and return a populated index. The word then names the pipeline rather than its first stage.
Metadata schemas are not standardised. The field names a loader attaches vary by loader, so metadata filtering applied across a corpus assembled by several loaders can silently match nothing for some sources.
The stage is where character encoding and whitespace decisions are made, mostly by default and mostly invisibly. Normalisation choices taken here propagate to every chunk, every vector, and every keyword index built from the text. See token.