---------------------------------------------------------------------- This is the API documentation for the raghilda library. ---------------------------------------------------------------------- ## Store Vector storage backends for storing and retrieving chunks BaseStore() Abstract base class for vector stores. A store is responsible for storing documents and their embeddings, and retrieving relevant chunks based on similarity search. Subclasses must implement all abstract methods to provide a concrete storage backend: - :py:class:`raghilda.store.DuckDBStore`: local storage with embedding and BM25 search. - :py:class:`raghilda.store.ChromaDBStore`: local storage using ChromaDB. - :py:class:`raghilda.store.OpenAIStore`: hosted storage using OpenAI's Vector Store API. DuckDBStore(con: _duckdb.DuckDBPyConnection, metadata: raghilda._store_metadata.EmbeddedAttributesStoreMetadata) A vector store backed by DuckDB. DuckDBStore provides local vector storage with support for both semantic search (using embeddings) and full-text search (using BM25). Data is persisted to a DuckDB database file. Examples -------- ```{python} #| eval: false from raghilda.store import DuckDBStore from raghilda.embedding import EmbeddingOpenAI # Create a new store with embeddings store = DuckDBStore.create( location="my_store.db", embed=EmbeddingOpenAI(), ) # Insert a chunked document from raghilda.document import MarkdownDocument from raghilda.chunker import MarkdownChunker doc = MarkdownDocument( origin="https://example.com/doc1.md", content="# Example This is a sample document.", ) store.upsert(MarkdownChunker().chunk(doc)) # Build indexes before retrieval store.build_index() # Retrieve similar chunks chunks = store.retrieve("How do I use this?", top_k=5) ``` ChromaDBStore(client: 'Any', collection: 'Any', metadata: 'AttributesStoreMetadata') A vector store backed by ChromaDB. ChromaDBStore provides local vector storage using Chroma's embedded client. Documents are chunked by raghilda and embeddings are generated by Chroma's embedding function (defaults to Chroma's built-in embedding). Examples -------- ```{python} #| eval: false from raghilda.store import ChromaDBStore store = ChromaDBStore.create(location="raghilda_chroma", name="docs") store.upsert(markdown_doc) chunks = store.retrieve("hello world", top_k=3) ``` OpenAIStore(client: Any, store_id: str, *, attributes_spec: collections.abc.Mapping[str, raghilda._attribute_schema.AttributeSpec] | None = None, attributes: collections.abc.Mapping[str, type[str] | type[int] | type[float] | type[bool] | raghilda._attribute_schema.AttributeFloatVectorType | raghilda._attribute_schema.AttributeStructType] | None = None) A vector store backed by OpenAI's Vector Store API. OpenAIStore uses OpenAI's hosted vector storage service for document storage and retrieval. Documents are uploaded as files and automatically chunked and embedded by OpenAI. Examples -------- ```{python} #| eval: false from raghilda.store import OpenAIStore # Create a new store store = OpenAIStore.create(name="my-store") # Or connect to an existing store store = OpenAIStore.connect(store_id="vs_abc123") # Insert documents from raghilda.document import MarkdownDocument doc = MarkdownDocument(content="# Hello\nWorld", origin="example.md") store.upsert(doc) # Retrieve similar chunks chunks = store.retrieve("greeting", top_k=5) ``` PostgreSQLStore(con: psycopg2.extensions.connection, metadata: dict, schema: str) A store backed by a PostgreSQL database with pgvector. Uses PostgreSQL for storage with two retrieval methods: - **Full-text search** via :meth:`retrieve_fts`: uses PostgreSQL's built-in ``tsvector``/``tsquery`` with ``ts_rank`` for ranking. A pre-computed ``tsvector`` column with a GIN index is created automatically. - **Vector similarity search** via :meth:`retrieve_vss`: uses pgvector for nearest-neighbor search over embeddings. An HNSW index for cosine distance is created automatically when an embedding provider is given. Use :meth:`build_index` to add indexes for other distance methods (L2, inner product). ## Crawl Crawlers for discovering and converting source documents CrawlScope(roots: 'RootsInput', include_patterns: 'PatternsInput' = None, exclude_patterns: 'PatternsInput' = None, depth: 'int | None' = None, limit: 'int | None' = None, include_types: 'Sequence[str] | None' = None, exclude_types: 'Sequence[str] | None' = None, include_external_links: 'bool' = False, include_subdomains: 'bool' = False) -> None Declarative description of what a crawler should discover. A `CrawlScope` is the traversal policy shared by every crawler. It names the starting points and the rules used to decide which origins are followed and yielded. The same scope can be reused across `DirectoryCrawler`, `WebCrawler`, and `CloudflareCrawler`, though a few fields are interpreted slightly differently per backend. Parameters ---------- roots Starting files, directories, or URLs. May be a single value or a sequence of values. include_patterns Patterns that an origin must match to be yielded. A `str` is treated as a glob: `*` matches any run of characters except `/`, and `**` matches across `/` (a trailing `/**` also matches the bare parent, so `/docs/**` matches `/docs` too). Pass a compiled `re.Pattern` to match by regular expression instead. Accepts a single pattern or a sequence; when `None`, all origins are allowed. exclude_patterns Patterns that drop an origin from the crawl, taking precedence over `include_patterns`. Uses the same glob-or-`re.Pattern` syntax. depth Number of link or directory levels to follow beyond the roots. `0` means only the roots themselves. When `None`, traversal is effectively unbounded. Must be non-negative. limit Maximum number of origins to yield. When `None`, no limit is applied. Must be non-negative. include_types Type labels to include, such as `"html"`, `"markdown"`, `"pdf"`, `"python"`, or `"text"`. When `None` or empty, all types are allowed. exclude_types Type labels to skip. Takes precedence over `include_types`. include_external_links Allow origins outside the root origin (a different scheme, host, or port). Defaults to `False`. include_subdomains Allow origins on subdomains of the root host. Defaults to `False`. FetchedSource(origin: 'str', body_path: 'Path', resolved_origin: 'str | None' = None, content_type: 'str | None' = None, status_code: 'int | None' = None, metadata: 'dict[str, Any] | None' = None, fetched_at: 'datetime | None' = None, revalidated_at: 'datetime | None' = None, markdown_path: 'Path | None' = None) -> None A fetched source document and its metadata, prior to conversion. A `FetchedSource` is the intermediate result returned by a crawler's `fetch_raw()`. It points at the raw body on disk and carries the metadata needed to convert it to a `MarkdownDocument`. Custom `convert` callables passed to `fetch_markdown()` or `markdown_documents()` receive an instance of this class. Parameters ---------- origin The canonical origin the source was requested from (a `file://` URI for local files, an `http(s)` URL for web sources). body_path Filesystem path to the raw fetched body. For local files this is the file itself; for web and Cloudflare sources it is a cached copy. resolved_origin The final origin after any redirects, when it differs from `origin`. content_type The reported MIME type, such as `"text/html"`, when available. status_code The HTTP status code for web sources, when available. metadata Backend-specific metadata, such as the detected `type_label`, validators (`etag`, `last_modified`), and source hashes. fetched_at When the source body was fetched, when known. revalidated_at When a cached body was last revalidated against the server, when known. markdown_path Filesystem path to already-converted Markdown, when the backend produced or cached it. `None` when conversion has not run. BaseCrawler() Abstract base class for crawlers. A crawler discovers source documents for a `CrawlScope` and converts them into `MarkdownDocument` objects ready for chunking and ingestion. All crawlers expose the same four public methods, so a scope and the surrounding workflow can be reused across backends. Subclasses provide a concrete discovery and fetching strategy: - `DirectoryCrawler`: walk local files and directories. - `WebCrawler`: fetch pages directly over HTTP and follow links. - `CloudflareCrawler`: delegate discovery and rendering to Cloudflare's Browser Rendering API. Attributes ---------- max_workers Number of worker threads used to fetch and convert sources concurrently in `markdown_documents()`. DirectoryCrawler(*, cache_dir: 'bool | str | Path | None' = None, max_workers: 'int' = 1) -> 'None' Crawl local files and optionally cache converted markdown. Use a `DirectoryCrawler` for local Markdown, notebooks, PDFs, text files, and other formats supported by `read_as_markdown()`. Directory traversal always reads the current filesystem state. The cache stores converted Markdown per file origin and is reused only when the current file hash and modification time still match the cached metadata. When the cache directory lives inside a crawled root, the crawler skips its own cache files. Parameters ---------- cache_dir Where to cache converted Markdown. `None` (default) disables caching. `True` uses `.raghilda/cache/directory` under the current working directory. A string or `Path` uses that location. max_workers Number of worker threads used to convert files concurrently in `markdown_documents()`. Must be at least 1. Default is 1. Examples -------- ```{python} #| eval: false from raghilda.crawl import CrawlScope, DirectoryCrawler crawler = DirectoryCrawler(cache_dir=True, max_workers=4) scope = CrawlScope( roots=["docs"], depth=3, include_patterns=[r".*\.(md|qmd|ipynb|pdf)$"], ) for document in crawler.markdown_documents(scope): print(document.origin) ``` WebCrawler(*, session: 'requests.Session | None' = None, cache_dir: 'bool | str | Path | None' = None, cache_stale_after: 'timedelta | None' = None, max_workers: 'int' = 1) -> 'None' Crawl a website by fetching pages directly over HTTP. A `WebCrawler` starts from one or more root URLs, fetches each page with `requests`, follows discovered links up to `scope.depth`, and yields matching pages as `MarkdownDocument` objects. Link following is constrained by the scope's patterns, types, and the `include_external_links` / `include_subdomains` flags. Fetched response bodies are cached on disk. When `cache_stale_after` is set, fresh cached responses are reused and stale ones are revalidated with `ETag` / `Last-Modified` headers when the server provides them. Pass `cache_force_refresh=True` to any method to bypass the cache for a run. Parameters ---------- session A `requests.Session` to use for requests. When omitted, a new session is created. A caller-supplied session also scopes the cache so entries are not shared across sessions. cache_dir Where to cache fetched bodies. `None` (default) uses a temporary directory. `True` uses `.raghilda/cache/web` under the current working directory. A string or `Path` uses that location. cache_stale_after How long a cached body stays fresh before it must be revalidated. When `None` (default), cached bodies are always considered fresh. max_workers Number of worker threads used to fetch pages concurrently. Must be at least 1. Default is 1. Examples -------- ```{python} #| eval: false from datetime import timedelta from raghilda.crawl import CrawlScope, WebCrawler crawler = WebCrawler(cache_dir=True, cache_stale_after=timedelta(days=1)) scope = CrawlScope( roots=["https://quarto.org/docs/guide/"], depth=2, include_patterns=[r"^https://quarto\.org/docs/guide/"], include_types=["html"], ) for document in crawler.markdown_documents(scope): print(document.origin) ``` CloudflareCrawler(*, account_id: 'str', api_token: 'str', cache_dir: 'bool | str | Path | None' = None, session: 'requests.Session | Any | None' = None, source: 'str' = 'all', render: 'bool' = True, cache_stale_after: 'timedelta | None' = None, modified_since: 'int | None' = None, poll_interval: 'float' = 5.0, max_poll_attempts: 'int' = 60, max_workers: 'int' = 1, base_url: 'str' = 'https://api.cloudflare.com/client/v4') -> 'None' Crawl a website using Cloudflare's Browser Rendering API. A `CloudflareCrawler` delegates page discovery, JavaScript rendering, and Markdown extraction to Cloudflare. It submits a crawl job, polls until the job completes, retrieves the rendered Markdown record for each discovered page, and yields them as `MarkdownDocument` objects. This is useful for single-page applications and other sites whose content only appears after client-side rendering. For Cloudflare crawls, `include_patterns` and `exclude_patterns` accept the same glob strings (such as `"https://example.com/docs/**"`) or compiled `re.Pattern` objects as the other crawlers. Glob strings are forwarded to Cloudflare's crawl request; regex patterns are enforced locally on the returned records. `include_external_links` / `include_subdomains` are passed through to the crawl request. Cached entries are invalidated automatically when `render`, `source`, or `modified_since` change between runs. Parameters ---------- account_id The Cloudflare account ID that owns the Browser Rendering subscription. api_token A Cloudflare API token with Browser Rendering permissions. Sent as a bearer token on each request. cache_dir Where to cache crawl results and page records. `None` (default) uses a temporary directory. `True` uses `.raghilda/cache/cloudflare` under the current working directory. A string or `Path` uses that location. session A `requests.Session` to use for API calls. When omitted, a new session is created. source How Cloudflare discovers pages: `"all"` (default) combines available methods, `"sitemap"` reads the site's sitemap, and `"crawl"` follows links from the rendered DOM. render Whether Cloudflare executes JavaScript before extracting content. Defaults to `True`. Set to `False` for server-rendered sites to reduce crawl time and API usage. cache_stale_after How long cached results stay fresh. When stale, the crawler requests updated content with a `maxAge` hint. When `None` (default), cached entries never expire. modified_since Restrict the crawl to pages modified after this Unix timestamp, for incremental refreshes. When `None` (default), all pages are eligible. poll_interval Seconds to wait between job-status polls. Default is `5.0`. max_poll_attempts Maximum number of status polls before a `TimeoutError` is raised. Default is `60` (a five-minute window at the default interval). max_workers Number of worker threads used to materialize page records concurrently. Must be at least 1. Default is 1. base_url Base URL of the Cloudflare API. Defaults to the public endpoint and is rarely overridden outside of testing. Examples -------- ```{python} #| eval: false import os from raghilda.crawl import CloudflareCrawler, CrawlScope crawler = CloudflareCrawler( account_id=os.environ["CLOUDFLARE_ACCOUNT_ID"], api_token=os.environ["CLOUDFLARE_API_TOKEN"], cache_dir=True, render=True, ) scope = CrawlScope( roots=["https://example.com/docs/"], depth=2, include_patterns=["https://example.com/docs/**"], ) for document in crawler.markdown_documents(scope): print(document.origin) ``` ## Embedding Embedding providers for generating vector representations EmbeddingProvider() Interface for embedding function providers. To create a custom embedding provider: 1. Subclass `EmbeddingProvider` and implement `embed()`, `get_config()`, and `from_config()` 2. Register it with `@register_embedding_provider("MyProvider")` Registered providers are automatically restored when connecting to a database that was created with that provider. Examples -------- ```{python} #| eval: false from raghilda.embedding import EmbeddingProvider, register_embedding_provider @register_embedding_provider("MyCustomEmbedding") class MyCustomEmbedding(EmbeddingProvider): def __init__(self, model: str = "default", api_key: str | None = None): self.model = model self.api_key = api_key # Initialize your embedding client here def embed(self, x, input_type=None): # Return list of embedding vectors ... def get_config(self): # Return config dict (exclude sensitive values like api_key) return {"type": "MyCustomEmbedding", "model": self.model} @classmethod def from_config(cls, config): return cls(model=config.get("model", "default")) ``` EmbedInputType(*values) Specifies the type of input being embedded. Some embedding models (e.g., Cohere) produce different embeddings for queries vs documents to optimize retrieval performance. EmbeddingOpenAI(model: str = 'text-embedding-3-small', base_url: str = 'https://api.openai.com/v1', api_key: str | None = None, batch_size: int = 20) -> None Creates an embedding function provider backed by OpenAI's embedding models Implements the [EmbeddingProvider](`raghilda.EmbeddingProvider`) interface. Parameters ---------- model The OpenAI embedding model to use. Default is "text-embedding-3-small" base_url The base URL for the OpenAI API. Default is "https://api.openai.com/v1". api_key The API key for authenticating with OpenAI. If None, it will use the OPENAI_API_KEY environment variable if set. batch_size The number of texts to process in each batch when calling the API. Examples -------- ```{python} #| eval: false from raghilda.embedding import EmbeddingOpenAI provider = EmbeddingOpenAI(model="text-embedding-3-small") embeddings = provider.embed(["hello world", "testing embeddings"]) print(len(embeddings)) print(len(embeddings[0])) # Dimension of the embedding print(embeddings[0][:10]) # The embedding vector ``` EmbeddingCohere(model: str = 'embed-english-v3.0', api_key: str | None = None, batch_size: int = 96) -> None Creates an embedding function provider backed by Cohere's embedding models. Implements the [EmbeddingProvider](`raghilda.EmbeddingProvider`) interface. Cohere's embedding models produce different embeddings for queries vs documents to optimize retrieval performance. Use `input_type=EmbedInputType.QUERY` when embedding search queries and `input_type=EmbedInputType.DOCUMENT` (default) when embedding documents for indexing. Parameters ---------- model The Cohere embedding model to use. Default is "embed-english-v3.0". api_key The API key for authenticating with Cohere. If None, it will use the CO_API_KEY environment variable if set. batch_size The number of texts to process in each batch when calling the API. Cohere supports up to 96 texts per request. Examples -------- ```{python} #| eval: false from raghilda.embedding import EmbeddingCohere, EmbedInputType provider = EmbeddingCohere(model="embed-english-v3.0") # Embed documents for indexing doc_embeddings = provider.embed( ["Hello world", "Testing embeddings"], input_type=EmbedInputType.DOCUMENT ) # Embed a query for search query_embedding = provider.embed( ["How do I test embeddings?"], input_type=EmbedInputType.QUERY ) ``` EmbeddingSentenceTransformers(model: str = 'all-MiniLM-L6-v2', device: str | None = None, batch_size: int = 64, prompts: dict[raghilda._embedding.EmbedInputType, str] | None = None) -> None Creates an embedding function provider backed by sentence-transformers models. Implements the [EmbeddingProvider](`raghilda.EmbeddingProvider`) interface. This provider runs models locally using the `sentence-transformers` library, enabling offline/private embedding without external API calls. Parameters ---------- model The sentence-transformers model to use. Default is "all-MiniLM-L6-v2". Any model from the Hugging Face Hub that is compatible with sentence-transformers can be used. device The device to run the model on (e.g., "cpu", "cuda", "mps"). If None, sentence-transformers will auto-detect the best available device. batch_size The number of texts to process in each batch. prompts Optional mapping from `EmbedInputType` to a prefix string to prepend to each text before encoding. This is useful for models that require task-specific prefixes (e.g., nomic-embed-text uses "search_query: " and "search_document: "). Examples -------- Install raghilda with sentence-transformers support: ```bash pip install raghilda[sentence-transformers] ``` ```{python} #| eval: false from raghilda.embedding import EmbeddingSentenceTransformers provider = EmbeddingSentenceTransformers(model="all-MiniLM-L6-v2") embeddings = provider.embed(["hello world", "testing embeddings"]) print(len(embeddings)) print(len(embeddings[0])) # Dimension of the embedding ``` For models that use task-specific prefixes: ```{python} #| eval: false from raghilda.embedding import EmbeddingSentenceTransformers, EmbedInputType provider = EmbeddingSentenceTransformers( model="nomic-ai/nomic-embed-text-v1.5", prompts={ EmbedInputType.QUERY: "search_query: ", EmbedInputType.DOCUMENT: "search_document: ", }, ) # Queries get "search_query: " prepended automatically query_emb = provider.embed(["Who is Laurens van Der Maaten?"], EmbedInputType.QUERY) # Documents get "search_document: " prepended automatically doc_emb = provider.embed(["TSNE is a dimensionality reduction algorithm"]) ``` ## Chunker Text chunking utilities for splitting documents BaseChunker() Abstract base class for chunkers. Chunking is the step that splits a document into smaller, retrievable passages. Good chunks are large enough to stand on their own but small enough that a search can return just the relevant part of a document rather than the whole thing. A chunker turns a `Document` into a `ChunkedDocument` whose `chunks` are ready to embed, index, and retrieve. `BaseChunker` only defines the interface; it is not used directly. A concrete chunker implements a strategy by overriding `chunk()` and `chunk_text()`. raghilda ships [`MarkdownChunker`](chunker.MarkdownChunker.qmd), which splits Markdown at semantic boundaries (headings, paragraphs, sentences). Any object implementing this interface (including third-party chunkers such as [chonkie](https://github.com/chonkie-inc/chonkie)'s) can be used wherever raghilda expects a chunker. MarkdownChunker(chunk_size: int = 1600, target_overlap: float = 0.5, *, max_snap_distance: int = 20, segment_by_heading_levels: list[int] | None = None) -> None Chunk Markdown documents into overlapping segments at semantic boundaries. This chunker divides Markdown text into smaller, overlapping chunks while intelligently positioning cut points at semantic boundaries like headings, paragraphs, sentences, and words. Rather than cutting rigidly at character counts, it nudges cut points to the nearest sensible boundary, producing more semantically coherent chunks suitable for RAG applications. Parameters ---------- chunk_size Target size for each chunk in characters. The chunker attempts to create chunks near this size, though actual sizes may vary based on semantic boundaries. Default is 1600 characters. target_overlap Fraction of overlap between successive chunks, from 0 to 1. Default is 0.5 (50% overlap). Even with 0, some overlap may occur because the last chunk is anchored to the document end. max_snap_distance Maximum distance (in characters) to move a cut point to reach a semantic boundary. If no boundary is found within this distance, the cut point stays at its original position. Default is 20. segment_by_heading_levels List of heading levels (1-6) that act as hard boundaries. When specified, no chunk will cross these headings, and segments between them are chunked independently. For example, `[1, 2]` ensures chunks never span across h1 or h2 headings. Examples -------- The example below chunks a short Markdown document with a deliberately small `chunk_size` so the splitting is easy to see. With `segment_by_heading_levels=[1, 2]`, no chunk crosses an `h1` or `h2`, so each section is chunked on its own: ```{python} from raghilda.chunker import MarkdownChunker chunker = MarkdownChunker( chunk_size=100, target_overlap=0.2, segment_by_heading_levels=[1, 2], ) text = '''# Introduction This is the introduction section with some content. ## Background Here is background information that provides context. ## Methods The methods section describes our approach. ''' chunks = chunker.chunk_text(text) for chunk in chunks: print(f"[{chunk.start_index}:{chunk.end_index}] {chunk.text[:40]}...") ``` Each printed line shows a chunk's character span (`start_index:end_index`) and the start of its text. To chunk a whole document instead of a bare string (preserving its `origin` and metadata), use `chunk()`, which returns a `ChunkedDocument`. Notes ----- The chunking algorithm works as follows: 1. Parse the Markdown to identify semantic boundaries (headings, paragraphs, sentences, lines, words) 2. If `segment_by_heading_levels` is set, split the document at those headings first 3. For each segment, calculate target chunk boundaries based on `chunk_size` and `target_overlap` 4. Snap each boundary to the nearest semantic boundary (preferring headings > paragraphs > sentences > lines > words) 5. Extract chunks with their positional information and heading context ## Utilities Utility functions for reading and scraping content read_as_markdown(uri: str, html_extract_selectors: list[str] | None = None, html_zap_selectors: list[str] | None = None, *args, **kwargs) -> raghilda.document.MarkdownDocument Read a document from a path or URL and convert it to Markdown. This is raghilda's entry point for turning a source into a `MarkdownDocument` ready for chunking. It reads the URI (a local file or an HTTP(S) URL) and converts it to Markdown using [MarkItDown](https://github.com/microsoft/markitdown), so it handles far more than Markdown files: HTML pages, PDFs, Word (`.docx`) documents, and the other formats MarkItDown supports are all converted to Markdown. For HTML, it keeps the `
` element and removes `