# Crawling and Ingestion

raghilda's core workflow is intentionally sequential: find a source, read it, chunk it, and upsert it. That is the recommended first path for building a store because every step is visible, easy to inspect, and easy to change.

As your source collection grows, store creation can become mostly waiting on network requests, file conversion, chunking, and writes. The crawling and ingestion API is the next step when you want that work to run concurrently. It can make store creation substantially faster while still letting you inspect the origins, fetched sources, converted Markdown documents, and final ingest summary.

The tradeoff is a few extra concepts. Use this API when the simple sequential workflow is too slow, or when you need a repeatable refresh job for a larger site, document collection, or codebase. The API has three parts:

- [CrawlScope](../reference/crawl.CrawlScope.md#raghilda.crawl.CrawlScope) describes what to crawl.
- A crawler discovers sources and returns [MarkdownDocument](../reference/document.MarkdownDocument.md#raghilda.document.MarkdownDocument) objects.
- `store.ingest()` prepares and upserts the stream.


# Crawl a website

Use [WebCrawler](../reference/crawl.WebCrawler.md#raghilda.crawl.WebCrawler) when you want raghilda to fetch pages directly with `requests`. The crawler starts from one or more roots, follows links up to `depth`, and yields matching pages as Markdown documents.


``` python
from datetime import timedelta

from raghilda.chunker import MarkdownChunker
from raghilda.crawl import CrawlScope, WebCrawler
from raghilda.embedding import EmbeddingOpenAI
from raghilda.store import DuckDBStore

store = DuckDBStore.create(
    location="docs.db",
    embed=EmbeddingOpenAI(),
    name="docs",
    overwrite=True,
)

crawler = WebCrawler(
    cache_dir=True,
    cache_stale_after=timedelta(days=1),
    max_workers=4,
)
scope = CrawlScope(
    roots=["https://quarto.org/docs/guide/"],
    depth=2,
    include_patterns=["https://quarto.org/docs/guide/**"],
    exclude_patterns=["**/reference/**"],
    include_types=["html"],
)

chunker = MarkdownChunker(chunk_size=1600, target_overlap=0.5)

summary = store.ingest(
    crawler.markdown_documents(scope),
    prepare=chunker.chunk,
    max_workers=4,
)
store.build_index()

print(summary)
```


[CrawlScope](../reference/crawl.CrawlScope.md#raghilda.crawl.CrawlScope) owns traversal policy:

| Field | Description |
|----|----|
| `roots` | Starting files, directories, or URLs. |
| `depth` | Number of link or directory levels to follow. `0` means only the roots. |
| `limit` | Maximum number of origins to yield. |
| `include_patterns` | Glob patterns (`*`, `**`) that origins must match. Pass a compiled `re.Pattern` for regex matching. |
| `exclude_patterns` | Glob patterns that remove origins from the crawl. Also accepts compiled `re.Pattern` objects. |
| `include_types` | Type labels to include, such as `html`, `markdown`, `pdf`, `python`, or `text`. |
| `exclude_types` | Type labels to skip. |
| `include_external_links` | Allow links outside the root origin. Defaults to `False`. |
| `include_subdomains` | Allow subdomains under the root host. Defaults to `False`. |

`include_patterns` and `exclude_patterns` use the same glob syntax across every crawler, so the same [CrawlScope](../reference/crawl.CrawlScope.md#raghilda.crawl.CrawlScope) behaves identically whether you run it through [WebCrawler](../reference/crawl.WebCrawler.md#raghilda.crawl.WebCrawler), [DirectoryCrawler](../reference/crawl.DirectoryCrawler.md#raghilda.crawl.DirectoryCrawler), or [CloudflareCrawler](../reference/crawl.CloudflareCrawler.md#raghilda.crawl.CloudflareCrawler). A single `*` matches any characters except `/` (one path segment), while `**` matches any characters including `/` (any number of segments). Patterns are matched against the full origin URL, and `exclude_patterns` always take priority over `include_patterns`. For matching that globs cannot express (alternation, anchors, character classes), pass a pre-compiled `re.Pattern` instead of a string; compiled regexes are matched with `re.search`. You can freely mix glob strings and compiled regexes in the same list.

`WebCrawler(cache_dir=True)` stores fetched response bodies under `.raghilda/cache/web`. With `cache_stale_after`, fresh cached responses are reused, and stale responses are revalidated with `ETag` or `Last-Modified` headers when the server provides them. Pass `cache_force_refresh=True` to [origins()](../reference/crawl.WebCrawler.md#raghilda.crawl.WebCrawler.origins), [fetch_raw()](../reference/crawl.WebCrawler.md#raghilda.crawl.WebCrawler.fetch_raw), [fetch_markdown()](../reference/crawl.BaseCrawler.md#raghilda.crawl.BaseCrawler.fetch_markdown), or [markdown_documents()](../reference/crawl.BaseCrawler.md#raghilda.crawl.BaseCrawler.markdown_documents) when a run must bypass the cache.


# Crawl local files

Use [DirectoryCrawler](../reference/crawl.DirectoryCrawler.md#raghilda.crawl.DirectoryCrawler) for local Markdown, notebooks, PDFs, text files, and other files supported by [read_as_markdown()](../reference/read.read_as_markdown.md#raghilda.read.read_as_markdown).


``` python
from raghilda.chunker import MarkdownChunker
from raghilda.crawl import CrawlScope, DirectoryCrawler
from raghilda.store import DuckDBStore

store = DuckDBStore.create(
    location="local-docs.db",
    embed=None,
    name="local_docs",
    overwrite=True,
)

crawler = DirectoryCrawler(cache_dir=True, max_workers=4)
scope = CrawlScope(
    roots=["docs"],
    depth=3,
    include_patterns=["**/*.md", "**/*.qmd", "**/*.ipynb", "**/*.pdf"],
    exclude_patterns=["**/_site/**", "**/.venv/**"],
)

chunker = MarkdownChunker()
summary = store.ingest(
    crawler.markdown_documents(scope),
    prepare=chunker.chunk,
    max_workers=4,
)

print(summary)
```


Directory crawling always reads the current filesystem tree. If you enable `cache_dir`, converted Markdown is reused only when the source file hash and modification time still match the cached metadata. The crawler also skips its own cache directory when the cache is inside a crawled root.


# Inspect before ingesting

The crawler interface is useful even when you are not ready to write to a store. Use [origins()](../reference/crawl.WebCrawler.md#raghilda.crawl.WebCrawler.origins) to inspect what the scope discovers, or use [fetch_markdown()](../reference/crawl.BaseCrawler.md#raghilda.crawl.BaseCrawler.fetch_markdown) to convert one source.


``` python
from raghilda.crawl import CrawlScope, WebCrawler

crawler = WebCrawler(cache_dir=True)
scope = CrawlScope(
    roots=["https://example.com/docs/"],
    depth=1,
    limit=10,
)

for origin in crawler.origins(scope):
    print(origin)

doc = crawler.fetch_markdown("https://example.com/docs/")
print(doc.origin)
print(doc.content[:500])
```


All crawler classes implement the same public methods:

| Method | Returns |
|----|----|
| `origins(scope)` | A lazy iterator of source origins. |
| `fetch_raw(origin)` | A [FetchedSource](../reference/crawl.FetchedSource.md#raghilda.crawl.FetchedSource) with the cached body path and metadata. |
| `fetch_markdown(origin)` | One [MarkdownDocument](../reference/document.MarkdownDocument.md#raghilda.document.MarkdownDocument). |
| `markdown_documents(scope)` | A lazy iterator of [MarkdownDocument](../reference/document.MarkdownDocument.md#raghilda.document.MarkdownDocument) objects. |


# Customize conversion

By default, crawlers convert fetched sources with raghilda's Markdown reader. Pass a `convert` function when a site or file collection needs custom cleanup. The function receives a [FetchedSource](../reference/crawl.FetchedSource.md#raghilda.crawl.FetchedSource) and returns a [MarkdownDocument](../reference/document.MarkdownDocument.md#raghilda.document.MarkdownDocument).


``` python
from raghilda.crawl import CrawlScope, FetchedSource, WebCrawler
from raghilda.document import MarkdownDocument
from raghilda.read import read_as_markdown


def convert_reference_page(source: FetchedSource) -> MarkdownDocument:
    doc = read_as_markdown(str(source.body_path))
    markdown = doc.content
    markdown = markdown.replace("Edit this page", "")
    return MarkdownDocument(origin=source.origin, content=markdown)


crawler = WebCrawler(cache_dir=True)
scope = CrawlScope(roots=["https://example.com/reference/"], depth=1)
documents = crawler.markdown_documents(scope, convert=convert_reference_page)
```


Keep chunking in `store.ingest(prepare=...)`, not in the converter. The converter should return one unchunked Markdown document per origin; `prepare` can then apply the same chunking policy to every document.


# Use Cloudflare crawling

Use [CloudflareCrawler](../reference/crawl.CloudflareCrawler.md#raghilda.crawl.CloudflareCrawler) when you want Cloudflare to perform the browser-rendered crawl and return Markdown records. This is useful for sites that need rendering or where you want Cloudflare's crawl service to manage discovery.


``` python
import os
from datetime import timedelta

from raghilda.chunker import MarkdownChunker
from raghilda.crawl import CloudflareCrawler, CrawlScope
from raghilda.store import DuckDBStore

store = DuckDBStore.create(
    location="rendered-docs.db",
    embed=None,
    name="rendered_docs",
    overwrite=True,
)

crawler = CloudflareCrawler(
    account_id=os.environ["CLOUDFLARE_ACCOUNT_ID"],
    api_token=os.environ["CLOUDFLARE_API_TOKEN"],
    cache_dir=True,
    cache_stale_after=timedelta(days=1),
    render=True,
)
scope = CrawlScope(
    roots=["https://example.com/docs/"],
    depth=2,
    include_patterns=["https://example.com/docs/**"],
    exclude_patterns=["https://example.com/docs/archive/**"],
    limit=250,
)

summary = store.ingest(
    crawler.markdown_documents(scope),
    prepare=MarkdownChunker().chunk,
    max_workers=4,
)

print(summary)
```


`include_patterns` and `exclude_patterns` use the same glob syntax here as for the other crawlers (for example, `https://example.com/docs/**`). Glob patterns are forwarded directly to Cloudflare's crawl API, which speaks the same `*`/`**` wildcard language. Compiled `re.Pattern` objects are also accepted: since the API cannot evaluate regexes, they are applied locally to the records Cloudflare returns. `include_external_links` and `include_subdomains` are passed through to the Cloudflare crawl request.


# Refresh a store

`store.ingest()` upserts each prepared document and returns an `IngestSummary` with counts for inserted, replaced, and skipped documents. The input stream is consumed lazily, and `prepare` runs in the worker pool.


``` python
summary = store.ingest(
    crawler.markdown_documents(scope, cache_force_refresh=True),
    prepare=chunker.chunk,
    max_workers=4,
)

print(f"Inserted: {summary.inserted}")
print(f"Replaced: {summary.replaced}")
print(f"Skipped: {summary.skipped}")
```


Use [upsert()](../reference/store.BaseStore.md#raghilda.store.BaseStore.upsert) directly when you need per-document `WriteResult` objects. Use [ingest()](../reference/store.BaseStore.md#raghilda.store.BaseStore.ingest) when you want one aggregate summary for a crawl or refresh job.