chunker.BaseChunker
Abstract base class for chunkers.
Usage
chunker.BaseChunker()Chunking is the step that splits a document into smaller, retrievable passages. Good chunks are large enough to stand on their own but small enough that a search can return just the relevant part of a document rather than the whole thing. A chunker turns a Document into a ChunkedDocument whose chunks are ready to embed, index, and retrieve.
BaseChunker only defines the interface; it is not used directly. A concrete chunker implements a strategy by overriding chunk() and chunk_text(). raghilda ships MarkdownChunker, which splits Markdown at semantic boundaries (headings, paragraphs, sentences). Any object implementing this interface (including third-party chunkers such as chonkie’s) can be used wherever raghilda expects a chunker.
Methods
| Name | Description |
|---|---|
| chunk() | Split a document into a ChunkedDocument. |
| chunk_text() | Split raw text into a sequence of Chunk objects. |
chunk()
Split a document into a ChunkedDocument.
Usage
chunk(document)Parameters
document: Document- The document to chunk.
Returns
ChunkedDocument-
The same document with its
chunksattached.
chunk_text()
Split raw text into a sequence of Chunk objects.
Usage
chunk_text(text)Use this when you have a bare string rather than a Document. Most callers use chunk() instead, which preserves document metadata.
Parameters
text: str- The text to chunk.
Returns
Sequence[Chunk]- The resulting chunks, each with positional information.