Skip to main content

Checking RAG Data

A retrieval-augmented generation (RAG) system answers a question by searching a corpus of text for the chunks most like it and giving them to a language model with the question. Its answers can only be as good as those chunks. A chunk with no text (from a PDF whose text couldn’t be extracted), the same paragraph in many chunks, a chunk without the source that the answer should cite, or a document that’s years out of date all make the answers worse, and none of them are errors that the system reports.

The evaluation set of a RAG system (questions with the documents that answer them) has problems of its own: a question asked twice counts twice in the scores, and a question whose document was removed from the corpus can never be answered. The pb.rag functions make validation steps for both, so they’re checked with the same reports, thresholds, and actions as any other table.

The Corpus

Let’s make a small corpus of chunks of help articles. Each chunk has an id, its text, the article it’s from, the groups of users that can see it (its access control list), and the date the article was last updated.

import datetime

import polars as pl

chunks = pl.DataFrame(
    {
        "chunk_id": ["acct-1", "acct-2", "bill-1", "bill-2", "bill-2", "ship-1", "hol-1"],
        "text": [
            "To reset your password, open Settings, choose Security, and select Reset password.",
            "Two-step verification can be turned on from the Security page of your Settings.",
            "Invoices can be downloaded as PDF files from the Billing page of your account.",
            "Invoices can be downloaded as PDF files from the Billing page of your account!",
            "Refunds are paid to the card used for the order within five working days.",
            " ",
            "Our offices are closed on public holidays.",
        ],
        "source": [
            "help/account",
            "help/account",
            "help/billing",
            "help/billing",
            "help/billing",
            "help/shipping",
            None,
        ],
        "acl": ["all", "all", "finance", "finance", "finance", "all", None],
        "updated_at": [datetime.date(2026, 9, 14)] * 6 + [datetime.date(2022, 12, 1)],
    }
)

pb.preview(chunks)
PolarsRows7Columns5
chunk_id
String
text
String
source
String
acl
String
updated_at
Date
1 acct-1 To reset your password, open Settings, choose Security, and select Reset password. help/account all 2026-09-14
2 acct-2 Two-step verification can be turned on from the Security page of your Settings. help/account all 2026-09-14
3 bill-1 Invoices can be downloaded as PDF files from the Billing page of your account. help/billing finance 2026-09-14
4 bill-2 Invoices can be downloaded as PDF files from the Billing page of your account! help/billing finance 2026-09-14
5 bill-2 Refunds are paid to the card used for the order within five working days. help/billing finance 2026-09-14
6 ship-1 help/shipping all 2026-09-14
7 hol-1 Our offices are closed on public holidays. None None 2022-12-01

Checking the Corpus

The pb.rag.corpus_checks() function makes a step for each check of the chunks. It returns a Steps collection, which is added to a validation with add_steps(). We’ll give it the column of the text and the other columns it needs for each check: the ids, the metadata that every chunk must have, and the dates of update (with how old a chunk can be). We’ll also ask for chunks of at least five words.

validation = (
    pb.Validate(data=chunks, tbl_name="chunks")
    .add_steps(
        pb.rag.corpus_checks(
            "text",
            chunk_id="chunk_id",
            metadata=["source", "acl"],
            min_tokens=5,
            updated_at="updated_at",
            max_age="104 weeks",
            reference_time=datetime.date(2026, 11, 1),
        )
    )
    .interrogate()
)

validation
STEP COLUMNS VALUES TBL EVAL UNITS PASS FAIL W E C EXT
#4CA64C66 1
col_vals_regex
col_vals_regex()

Every chunk has text in text.

 text [^ \t\r\n] ✓ 7 6
0.86		 1
0.14		 — — —
#4CA64C66 2
rows_distinct
rows_distinct()

The chunk ids in chunk_id are unique.

 chunk_id — ✓ 7 5
0.71		 2
0.29		 — — —
#4CA64C66 3
col_vals_not_null
col_vals_not_null()

Every chunk has a value of source.

 source — ✓ 7 6
0.86		 1
0.14		 — — —
#4CA64C66 4
col_vals_not_null
col_vals_not_null()

Every chunk has a value of acl.

 acl — ✓ 7 6
0.86		 1
0.14		 — — —
#4CA64C66 5
col_vals_scored
col_vals_scored()

Every chunk has at least 5 tokens.

 text ≥5
TOKENS		 ✓ 7 6
0.86		 1
0.14		 — — —
#4CA64C66 6
rows_near_distinct
rows_near_distinct()

No chunk is a near duplicate of another.

 text ≥0.85
MINHASH 5-GRAM		 ✓ 7 5
0.71		 2
0.29		 — — —
#4CA64C66 7
col_vals_ge
col_vals_ge()

Every chunk was updated in the 104 weeks before 2026-11-01.

 updated_at 2024-11-03 ✓ 7 6
0.86		 1
0.14		 — — —
Notes

Step 5 (scores) Scored by token_count(tokenizer='whitespace'): 7 distinct rows scored. Scores from 0 to 14 (median 13).

Step 6 (near_duplicates) 1 group of near duplicates, with 2 rows; MinHash of 5-word n-grams, with a Jaccard similarity of at least 0.85. The group of each row is in the pb_cluster_ column.

The steps check, in order (and the brief of each says what it checks):

  • Text. Every chunk has text: it isn’t missing, and it isn’t only whitespace. The chunk of the shipping article is a single space.
  • Unique ids. No two chunks have the same id. The id bill-2 is used for two different chunks, so a citation of bill-2 could mean either.
  • Metadata. Every chunk has a value of each column of metadata=, with a step for each. The chunk about holidays has no source to cite and no access control list. A chunk without an access control list is a problem of security as much as of quality: it may be shown to anyone, or to no one, depending on how the system treats a missing value.
  • Length. Every chunk has from min_tokens= to max_tokens= tokens. The chunk of the shipping article has none, so it fails this step too. The tokens are words by default; with tokenizer=, they’re the tokens of a model (as in pb.text.token_length()).
  • Near duplicates. No chunk is nearly the same as another, which would crowd the other chunks out of the results of a search. The two chunks about invoices differ only in their punctuation. This is a rows_near_distinct() step, and near_duplicates= sets its threshold of similarity (or turns it off).
  • Freshness. Every chunk was updated in the max_age= before the reference time. The article about holidays was last updated almost four years ago.

The checks of the ids, the length, and freshness are left out unless their columns or bounds are given. By default, the reference time is the time the steps are made, so that the cutoff of the check of freshness is fixed in the steps.

Checking the Evaluation Set

An evaluation set has questions, each with the id of the document that answers it (its gold document) and often the passage of that document with the answer. The pb.rag.eval_checks() function checks that every question has text, that no question is a near duplicate of another, that every question has a gold document, and that every gold document is in the corpus.

It can also check that each gold passage is relevant to its question, with a scorer of the two. A reranker, like pb.scorers.reranker(), is made for this. To keep this page self-contained, we’ll use a scorer that gives the share of the words of a passage that are in its question.

import re

questions = pl.DataFrame(
    {
        "question": [
            "How do I reset my password?",
            "Where can I download an invoice?",
            "how do I reset my password",
            "When are refunds paid?",
        ],
        "chunk_id": ["acct-1", "bill-1", "acct-1", "bill-9"],
        "passage": [
            "open Settings, choose Security, and select Reset password",
            "Invoices can be downloaded as PDF files",
            "open Settings, choose Security, and select Reset password",
            "Our offices are closed on public holidays.",
        ],
    }
)


def words(text):
    return re.findall(r"\w+", text.lower())


def word_overlap(rows):
    scores = []
    for row in rows:
        question, passage = set(words(row["question"])), words(row["passage"])
        scores.append(sum(word in question for word in passage) / len(passage))
    return scores


validation = (
    pb.Validate(data=questions, tbl_name="questions")
    .add_steps(
        pb.rag.eval_checks(
            "question",
            gold_doc_id="chunk_id",
            corpus=chunks,
            passage="passage",
            scorer=pb.scorers.from_function(word_overlap, name="word_overlap"),
            min_relevance=0.2,
        )
    )
    .interrogate()
)

validation
STEP COLUMNS VALUES TBL EVAL UNITS PASS FAIL W E C EXT
#4CA64C 1
col_vals_regex
col_vals_regex()

Every query has text in question.

 question [^ \t\r\n] ✓ 4 4
1.00		 0
0.00		 — — — —
#4CA64C66 2
rows_near_distinct
rows_near_distinct()

No query is a near duplicate of another.

 question ≥0.85
MINHASH 5-GRAM		 ✓ 4 2
0.50		 2
0.50		 — — —
#4CA64C 3
col_vals_not_null
col_vals_not_null()

Every query has a gold document in chunk_id.

 chunk_id — ✓ 4 4
1.00		 0
0.00		 — — — —
#4CA64C66 4
col_vals_in_table
col_vals_in_table()

Every gold document is in the corpus.

 chunk_id {'ref_table': shape: (7, 5) ┌──────────┬─────────────────────────────────┬───────────────┬─────────┬────────────┐ │ chunk_id ┆ text ┆ source ┆ acl ┆ updated_at │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str ┆ str ┆ date │ ╞══════════╪═════════════════════════════════╪═══════════════╪═════════╪════════════╡ │ acct-1 ┆ To reset your password, open S… ┆ help/account ┆ all ┆ 2026-09-14 │ │ acct-2 ┆ Two-step verification can be t… ┆ help/account ┆ all ┆ 2026-09-14 │ │ bill-1 ┆ Invoices can be downloaded as … ┆ help/billing ┆ finance ┆ 2026-09-14 │ │ bill-2 ┆ Invoices can be downloaded as … ┆ help/billing ┆ finance ┆ 2026-09-14 │ │ bill-2 ┆ Refunds are paid to the card u… ┆ help/billing ┆ finance ┆ 2026-09-14 │ │ ship-1 ┆ ┆ help/shipping ┆ all ┆ 2026-09-14 │ │ hol-1 ┆ Our offices are closed on publ… ┆ null ┆ null ┆ 2022-12-01 │ └──────────┴─────────────────────────────────┴───────────────┴─────────┴────────────┘, 'ref_column': 'chunk_id', 'columns': 'chunk_id'} ✓ 4 3
0.75		 1
0.25		 — — —
#4CA64C66 5
col_vals_scored
col_vals_scored()

Every passage in passage is relevant to its query.

 question, passage ≥0.2
word_overlap		 ✓ 4 2
0.50		 2
0.50		 — — —
Notes

Step 2 (near_duplicates) 1 group of near duplicates, with 2 rows; MinHash of 5-word n-grams, with a Jaccard similarity of at least 0.85. The group of each row is in the pb_cluster_ column.

Step 5 (scores) Scored by word_overlap: 4 distinct rows scored. Scores from 0.1429 to 0.25 (median 0.1964).

The first and third questions are the same once they’re in lower case and without punctuation, so both fail step 2. The last question’s gold document, bill-9, isn’t in the corpus (step 4), and its passage is about something else (step 5). The second passage fails step 5 as well, since it shares only one word with its question (invoices and downloaded aren’t the words invoice and download). That shows the limits of counting words: a reranker would see that a passage about downloading invoices answers it. The corpus can be any table that col_vals_in_table() takes, and corpus_id= names its column of ids if it’s different from the column of the evaluation set.

Which Checks Matter?

Like the checks of text data, these are ordinary validation steps. The rows that fail can be removed with get_sundered_data(), and an Ablation can measure how much each check matters to the quality of the answers, by evaluating the system with a corpus that has the failing chunks of each step removed. Every check runs on any backend, so a corpus in a database or a PySpark DataFrame can be checked where it is (the checks of length, near duplicates, and relevance collect the distinct values of their columns, score them in Python, and join the results back to the table).

Conclusion

A RAG system is only as good as its corpus and its evaluation set. The pb.rag.corpus_checks() function checks that the chunks have text, unique ids, the metadata the system needs, a useful length, no near duplicates, and recent updates; pb.rag.eval_checks() checks that the questions have text and aren’t repeated, that their gold documents are in the corpus, and that their gold passages are relevant to them.