RAG

Retrieval-Augmented Generation (RAG) is a technique that can improve LLM output by grounding it with external, trusted content. RAG workflows can have varying degrees of sophistication, but at their core, they all share a retrieval step that fetches relevant information from a knowledge store. The retrieved information (along with the user query) is then provided as additional context to the LLM for response generation. In this article, you’ll learn how to do exactly this to improve the quality of responses in your chatlas applications.

Do you need RAG?

The term RAG gets thrown around a lot, and for good reason: it can be a very useful way to address the most serious limitations of LLMs. However, RAG is not always necessary, and sometimes something much simpler will do the trick, such as:

Adding trusted/missing content directly to the system prompt (or chat input), rather than retrieving a portion of it from a knowledge store.
Register a tool enabling the LLM to “ask” for trusted content as needed, possibly without needing to manage a separate knowledge store.

In theory, grounding the LLM’s response in trusted content helps to reduce hallucination. But in practice, RAG can be fickle. It’s hard to always retrieve the right information for every user query, and it’s not always predictable how the LLM will actually use the retrieved content. For this reason, it’s helpful for your RAG workflow to be transparent (so it’s easy to debug, understand, and modify) because some trial and error will be necessary. It can also be very helpful to combine RAG with additional techniques, such as:

Setting guidelines for how the LLM should use the retrieved content in the system prompt.
If needed, provide a tool to retrieve additional information from a knowledge store, effectively enabling the LLM to decide when it needs more information.

In any RAG workflow, you’ll always want to apply the 1st technique (i.e., set guidelines). However, before diving into the 2nd technique (i.e., dynamic retrieval), let’s first learn the basics.

Basic retrieval

Basic retrieval is the simplest form of RAG, where you retrieve (a fixed amount of) relevant content from a knowledge store based on the user’s query and provide it to the chat model. It looks something roughly like this:

from chatlas import ChatOpenAI

chat = ChatOpenAI(
    system_prompt="You are a helpful, but terse, assistant. "
    "If you can't answer the question based on the trusted content, say so.",
)

user_query = "Who created the unicorn programming language?"

# A placeholder for your retrieval logic
trusted_content = retrieve_trusted_content(user_query)

chat.chat(*trusted_content, user_query)

In the sections that follow, we’ll implement the retrieve_trusted_content() step of this workflow. And, as we’ll see, there are several moving parts to consider when implementing this step.

Obviously, in order to retrieve trusted content, we first need some content to retrieve. Typically content is retrieved from a knowledge store, which is essentially a database that stores documents in a way that allows for efficient retrieval based on semantic similarity. A knowledge store also often takes the form of a vector store or embedding index because of its efficiency in storing and retrieving content based on semantic similarity. This approach requires embedding the content into numerical vectors, which can be done using various machine learning models.

Create store

Python has a plethora of options for working with knowledge stores (llama-index, pinecone, raghilda, etc.). The examples in this article use raghilda, a Posit package made to work alongside chatlas. We chose it for its defaults, which handle a lot of what you’d otherwise set up by hand. It splits documents at headings and paragraphs, searches by both meaning and keywords, merges overlapping results, and saves the store to disk as you build it. Nothing in this article depends on raghilda, though. You can follow the same steps with llama-index or another library, and the tip at the end of Retrieve content shows what that looks like.

Prerequisites

Install raghilda alongside chatlas:

pip install raghilda chatlas

The examples here use OpenAI for both embeddings and chat, so you’ll also need an OpenAI API key set in the OPENAI_API_KEY environment variable (or see the “Free, local embeddings” tip below).

A raghilda store is a database that holds your document chunks along with their embeddings. The DuckDBStore backend keeps everything in a single local file, so there’s no server to run. For this example, let’s assume you have a directory (data) of Markdown (.md) files that you want to use as trusted content. This snippet reads each file, chunks it, embeds the chunks, and writes them to a store on disk:

from pathlib import Path

from raghilda.chunker import MarkdownChunker
from raghilda.embedding import EmbeddingOpenAI
from raghilda.read import read_as_markdown
from raghilda.store import DuckDBStore

store = DuckDBStore.create(
    location="knowledge.db",
    embed=EmbeddingOpenAI(),
    overwrite=True,
)

chunker = MarkdownChunker()
for path in Path("data").glob("*.md"):
    doc = read_as_markdown(str(path))
    store.upsert(chunker.chunk(doc))

store.build_index()

MarkdownChunker splits each document into overlapping chunks at natural boundaries (headings, paragraphs, and sentences), and each chunk remembers the headings it falls under. The final build_index() call builds two indexes: a vector index for semantic search and a BM25 index for keyword search. Since the store is written to knowledge.db as you go, there’s no separate step to save it. You build it once, and your app connects to it later.

Other inputs

read_as_markdown() accepts URLs as well as file paths, and it converts HTML, PDF, Word, and many other formats to Markdown along the way. To index an entire website, raghilda’s crawler can discover and fetch the pages for you.

Free, local embeddings

EmbeddingOpenAI is just one of raghilda’s embedding providers. If you’d rather not use an API key (or don’t want your documents to leave your machine), the SentenceTransformers provider runs open-source models locally. First install the extra:

pip install "raghilda[sentence-transformers]"

Then pass the provider to the store in place of EmbeddingOpenAI():

from raghilda.embedding import EmbeddingSentenceTransformers

store = DuckDBStore.create(
    location="knowledge.db",
    embed=EmbeddingSentenceTransformers(),
    overwrite=True,
)

raghilda also supports Cohere and NVIDIA NIM embeddings, and you can write your own provider for other services. See the raghilda reference for the full list.

Chunking defaults

Long documents (articles or books, say) need to be split into smaller chunks, because the part that answers a question is usually a small piece of the whole. MarkdownChunker aims for chunks of about 1,600 characters, with each one overlapping its neighbor by half. You can change both settings:

chunker = MarkdownChunker(chunk_size=800, target_overlap=0.3)

raghilda’s chunking article has advice on picking these values.

Other storage backends

DuckDBStore is a good default, but raghilda also provides stores backed by ChromaDB, OpenAI vector stores, and PostgreSQL. The retrieval code in the rest of this article works the same way with any of them.

Retrieve content

With the store on disk, we’re ready to implement retrieve_trusted_content(), the function that finds content in the knowledge store that’s relevant to the user’s query. In your app, connect to the store and search it:

from raghilda.store import DuckDBStore

store = DuckDBStore.connect("knowledge.db", read_only=True)

def retrieve_trusted_content(query: str, top_k: int = 5) -> list[str]:
    chunks = store.retrieve(query, top_k=top_k)
    return [f"<excerpt>{chunk.text}</excerpt>" for chunk in chunks]

The store remembers which embedding model it was created with, so connect() restores it for you and queries are embedded the same way as the documents were. Opening the store with read_only=True is a good habit in an app that only retrieves content.

Building and connecting in one session

DuckDB won’t open a file as read-only while the same Python session still has it open for writing. If you build the store and then call connect(read_only=True) in the same notebook or script, you’ll get a ConnectionException. Either build the store in a separate script (or restart your Python session before connecting), or keep using the store returned by create().

By default, retrieve() runs a hybrid search that combines semantic similarity with BM25 keyword matching. Semantic search finds passages that mean the same thing as the query even when they use different words, while keyword search catches exact terms (like function names or error messages) that embeddings can miss. The top_k limit applies to each kind of search separately before the results are combined, so you may get back more (or fewer) chunks than top_k. Overlapping chunks from the same document are also merged into one passage, so the model doesn’t see the same text twice. There’s no right value for top_k, so if the retrieved content seems too thin or off the mark, try raising it.

Once retrieve_trusted_content() is defined, the basic retrieval example at the top of this article runs as is. Each excerpt goes to .chat() as a separate input, with the question last, so the model gets the passages and the question together in one user turn.

Using llama-index instead

If you’d rather use llama-index, install it with pip install llama-index. Then build the index once and save it to disk:

from llama_index.core import SimpleDirectoryReader, VectorStoreIndex

docs = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(docs)
index.storage_context.persist(persist_dir="./storage")

In your app, load the saved index and use its retriever inside retrieve_trusted_content():

from llama_index.core import StorageContext, load_index_from_storage

storage_context = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(storage_context)

def retrieve_trusted_content(query: str, top_k: int = 5) -> list[str]:
    retriever = index.as_retriever(similarity_top_k=top_k)
    nodes = retriever.retrieve(query)
    return [f"<excerpt>{x.text}</excerpt>" for x in nodes]

This version only does semantic search. To change the embedding model or chunk size, use the Settings object in llama-index. The chat code stays the same no matter which library you use.

Dynamic retrieval

Dynamic retrieval is similar to basic retrieval, except that instead of the retrieval being a single fixed step before response generation, it is provided as a tool to the LLM. This results in a much more robust and flexible RAG workflow, as the LLM can decide if, when, and how much content to retrieve from the knowledge store before generating a response. It can also decide what to provide as input to the retrieval step(s), rather than just using the user query directly, which can be useful if the user query is ambiguous or incomplete.

from chatlas import ChatOpenAI
from raghilda.store import DuckDBStore

store = DuckDBStore.connect("knowledge.db", read_only=True)

def search_knowledge_store(query: str, top_k: int = 5) -> list[dict]:
    """
    Search the knowledge store for content relevant to the query.

    Parameters
    ----------
    query
        A description of the information to look for.
    top_k
        The number of passages to retrieve.
    """
    chunks = store.retrieve(query, top_k=top_k)
    return [
        {"text": chunk.text, "headings": chunk.context, "source": chunk.origin}
        for chunk in chunks
    ]

chat = ChatOpenAI(
    system_prompt="You are a helpful, but terse, assistant. "
    "Search the knowledge store before answering, and cite the source of any content you use. "
    "If you can't answer the question based on the trusted content, say so.",
)

chat.register_tool(search_knowledge_store)

chat.chat("Who created the unicorn programming language?")

As with any chatlas tool, the docstring and type hints tell the model what the tool does and how to call it, so it’s worth making them specific to your content. The top_k parameter lets the model ask for more results when it needs them. The tool returns a list of dictionaries, which chatlas converts to JSON before sending it to the model. Along with each passage’s text, the dictionaries include the headings it falls under (chunk.context, which is None when there aren’t any) and the file or URL it came from (chunk.origin). The headings help the model make sense of a passage taken out of context, and the source lets it cite where an answer came from.

Scope retrieval

A store that mixes different kinds of content (say, a user guide and an API reference) can return passages from the wrong part of it. raghilda lets you tag documents with attributes when you build the store and then filter on them at retrieval time. For example, suppose your data directory has two subfolders, guide and reference. When you build (or rebuild) the store, declare the attributes in create(), and then set them on each document before chunking it:

store = DuckDBStore.create(
    location="knowledge.db",
    embed=EmbeddingOpenAI(),
    attributes={"section": str},
    overwrite=True,
)

chunker = MarkdownChunker()
for section in ["guide", "reference"]:
    for path in Path("data", section).glob("*.md"):
        doc = read_as_markdown(str(path))
        doc.attributes = {"section": section}
        store.upsert(chunker.chunk(doc))

store.build_index()

Every chunk inherits its document’s attributes. You can then expose the filter as a tool parameter, so the model chooses which part of the store to search:

from typing import Literal, Optional

def search_knowledge_store(
    query: str,
    section: Optional[Literal["guide", "reference"]] = None,
) -> list[dict]:
    """
    Search the documentation for content relevant to the query.

    Parameters
    ----------
    query
        A description of the information to look for.
    section
        Limit the search to the user guide ("guide") or the API reference
        ("reference"). Leave unset to search everything.
    """
    attributes_filter = f"section = '{section}'" if section else None
    chunks = store.retrieve(query, top_k=5, attributes_filter=attributes_filter)
    return [{"text": chunk.text, "source": chunk.origin} for chunk in chunks]

Register this tool with chat.register_tool(), just as in the dynamic retrieval example. The Literal type hint tells the model which sections exist, and because your function builds the filter, the model only has to pick a section name rather than write filter syntax. raghilda’s attribute filters article covers more complex schemas and filter expressions.

Learn more

The raghilda documentation goes deeper into each part of the knowledge store:

Core Concepts covers building, indexing, and reconnecting to a store.
Chunking explains how to tune chunk size and overlap for better retrieval.
Using RAG with chatlas shows the same integration from raghilda’s side, including registering several search tools and choosing a model provider.