rag.eval_checks()

Make validation steps that check the evaluation set of a RAG system.

Usage

Source

rag.eval_checks(
    query,
    gold_doc_id=None,
    corpus=None,
    corpus_id=None,
    passage=None,
    scorer=None,
    min_relevance=None,
    near_duplicates=True,
    pre=None,
    thresholds=None,
    actions=None,
    active=True,
)

A RAG system is evaluated with a set of queries, each with the document that has its answer (the gold document) and often the passage of it. The scores of the evaluation can only be trusted if the set itself is sound: every query has text, no query is asked twice (a near duplicate counts twice in the scores), every gold document is in the corpus (a query whose document was removed or renamed can never be answered), and each gold passage is relevant to its query. This function makes a validation step for each:

• every query has text (not missing, and not only whitespace): a col_vals_regex() step
• no query is a near duplicate of another, unless near_duplicates=False: a rows_near_distinct() step
• every query has a gold document, with gold_doc_id=: a col_vals_not_null() step
• every gold document is in the corpus, with corpus=: a col_vals_in_table() step
• every passage is relevant to its query, with passage=, scorer=, and min_relevance=: a col_vals_scored() step (with a reranker, like pb.scorers.reranker(), or another scorer of a query and a passage)

Parameters

query: str

The column with the queries.

gold_doc_id: str | None = None

The column with the id of the gold document of each query.

corpus: Any = None

The corpus (any table that col_vals_in_table() takes, such as a DataFrame, a database table, or the path of a file) to look up the gold documents in. It needs gold_doc_id=.

corpus_id: str | None = None

The column of the corpus with the document ids (by default, the same name as gold_doc_id=).

passage: str | None = None

The column with the gold passage of each query, for the check of relevance.

scorer: Callable[…, Any] | None = None

The scorer of the relevance of a passage to its query, given the values of query= and passage= (in that order).

min_relevance: float | None = None

The lowest score of relevance that a passage can have.

near_duplicates: bool | float = True

Whether to check for near-duplicate queries (True by default), or the threshold of the similarity of two queries for them to be near duplicates (the threshold= of rows_near_distinct(), 0.85 with True).

pre: Callable | None = None

A function that transforms the table before each step, as in the validation methods.

thresholds: int | float | bool | tuple | dict | Thresholds | None = None

The thresholds of the steps (by default, those of the validation).

actions: Actions | None = None

The actions of the steps.

active: bool | Callable = True

Whether the steps are active.

Returns

Steps

The steps, to be added to a validation with Validate.add_steps().

Examples

Let’s check an evaluation set of three queries against a corpus of three documents.

import pointblank as pb
import polars as pl

corpus = pl.DataFrame({"doc_id": ["account", "billing", "shipping"]})

queries = pl.DataFrame(
    {
        "query": [
            "How do I reset my password?",
            "Where can I download an invoice?",
            "how do I reset my password",
        ],
        "doc_id": ["account", "invoices", "account"],
    }
)

(
    pb.Validate(data=queries, tbl_name="queries")
    .add_steps(pb.rag.eval_checks("query", gold_doc_id="doc_id", corpus=corpus))
    .interrogate()
)

		STEP	COLUMNS	VALUES	TBL	EVAL	UNITS	PASS	FAIL	W	E	C	EXT
#4CA64C	1	col_vals_regex() Every query has text in query.	query	[^ \t\r\n]		✓	3	3 1.00	0 0.00	—	—	—	—
#4CA64C66	2	rows_near_distinct() No query is a near duplicate of another.	query	≥0.85 MINHASH 5-GRAM		✓	3	1 0.33	2 0.67	—	—	—
#4CA64C	3	col_vals_not_null() Every query has a gold document in doc_id.	doc_id	—		✓	3	3 1.00	0 0.00	—	—	—	—
#4CA64C66	4	col_vals_in_table() Every gold document is in the corpus.	doc_id	{'ref_table': shape: (3, 1) ┌──────────┐ │ doc_id │ │ --- │ │ str │ ╞══════════╡ │ account │ │ billing │ │ shipping │ └──────────┘, 'ref_column': 'doc_id', 'columns': 'doc_id'}		✓	3	2 0.67	1 0.33	—	—	—
Notes Step 2 (near_duplicates) 1 group of near duplicates, with 2 rows; MinHash of 5-word n-grams, with a Jaccard similarity of at least 0.85. The group of each row is in the pb_cluster_ column.

The first and third queries are the same once normalized, so both fail step 2, and the second query’s gold document (invoices) isn’t in the corpus. A check of relevance needs a scorer, such as scorer=pb.scorers.reranker(provider="cohere") with passage= and min_relevance=.