Reference

Store

Vector storage backends for storing and retrieving chunks

store.BaseStore: Abstract base class for vector stores.
store.DuckDBStore: A vector store backed by DuckDB.
store.ChromaDBStore: A vector store backed by ChromaDB.
store.OpenAIStore: A vector store backed by OpenAI’s Vector Store API.
store.PostgreSQLStore: A store backed by a PostgreSQL database with pgvector.

Crawlers for discovering and converting source documents

crawl.CrawlScope: Declarative description of what a crawler should discover.
crawl.FetchedSource: A fetched source document and its metadata, prior to conversion.
crawl.BaseCrawler: Abstract base class for crawlers.
crawl.DirectoryCrawler: Crawl local files and optionally cache converted markdown.
crawl.WebCrawler: Crawl a website by fetching pages directly over HTTP.
crawl.CloudflareCrawler: Crawl a website using Cloudflare’s Browser Rendering API.

Embedding providers for generating vector representations

embedding.EmbeddingProvider: Interface for embedding function providers.
embedding.EmbedInputType: Specifies the type of input being embedded.
embedding.EmbeddingOpenAI: Creates an embedding function provider backed by OpenAI’s embedding models
embedding.EmbeddingCohere: Creates an embedding function provider backed by Cohere’s embedding models.
embedding.EmbeddingSentenceTransformers: Creates an embedding function provider backed by sentence-transformers models.

Text chunking utilities for splitting documents

chunker.BaseChunker: Abstract base class for chunkers.
chunker.MarkdownChunker: Chunk Markdown documents into overlapping segments at semantic boundaries.

Utility functions for reading and scraping content

read.read_as_markdown(): Read a document from a path or URL and convert it to Markdown.
scrape.find_links(): Discover links by crawling one or more starting pages.

Chunk data types

chunk.Chunk: A segment of text extracted from a document.
chunk.MarkdownChunk: A chunk extracted from a Markdown document.
chunk.RetrievedChunk: A chunk returned from a retrieval operation with associated metrics.
chunk.Metric: A named metric value associated with a retrieved chunk.

Document types for unchunked and chunked content

document.Document: A document containing text content to be chunked and indexed.
document.ChunkedDocument: A document with an attached sequence of chunks.
document.MarkdownDocument: A Markdown document with source tracking.
document.ChunkedMarkdownDocument: A Markdown document with an attached sequence of chunks.

Protocol types for type checking compatibility

types.ChunkLike: Structural type for any chunk raghilda can consume.
types.ChunkedDocumentLike: Structural type for a document that already carries its chunks.
types.DocumentLike: Structural type for any unchunked document raghilda can consume.
types.ChunkerLike: Structural type for any chunker raghilda can use.
types.IntoChunk: Structural type for objects that convert themselves into a Chunk.
types.IntoDocument: Structural type for objects that convert themselves into a Document.