import pointblank as pb
import polars as pl
tbl = pl.DataFrame(
{
"product": ["kettle", "kettle", "toaster", "toaster", "lamp", "lamp"],
"review": [
"great kettle and fast",
"broke after a week",
"good toast every time",
"great value",
"too dim",
None,
],
}
)
POSITIVE = {"great", "good", "fast", "value", "every"}
def positivity(batch):
return [
sum(w in POSITIVE for w in row["review"].split()) / len(row["review"].split())
for row in batch
]
pb.preview(tbl)Validate.col_vals_scored()
Validate whether rows have a score from a model within bounds.
Usage
Validate.col_vals_scored(
columns,
scorer,
min_val=None,
max_val=None,
na_pass=False,
cache=True,
keep_scores=True,
pre=None,
segments=None,
thresholds=None,
actions=None,
brief=None,
active=True,
dimension=None,
)The col_vals_scored() validation method gives each row a score with a model (a scorer) and checks that the score is at least min_val=, at most max_val=, or both. It’s for the checks that are about meaning rather than form: whether a passage answers its query (a reranker), whether a document is of good quality or toxic (a classifier), whether text is in the expected language, or whether it’s fluent (perplexity). Each row is a test unit, and the scorer gets the values of columns= for the row.
The scorer can be any function of a batch of rows (see the Scorer protocol), or one of the adapters of pb.scorers, which use models from common packages: embedding_similarity(), reranker(), classifier(), langid(), and perplexity(). Pointblank never trains a model: the scorers use models that are given to them.
Parameters
columns: str | list[str]-
The column (or a list of columns) whose values the scorer gets for each row. A reranker, for example, needs two columns: the query and the document.
scorer: Callable[…, Any]-
The scorer: a function that takes a list of rows (dictionaries of the values of
columns=) and returns a score for each, or an adapter ofpb.scorers. A score is a number, orNonefor a row that can’t be scored. min_val: float | None = None-
The lowest score that passes (inclusive).
max_val: float | None = None-
The highest score that passes (inclusive). At least one of
min_val=andmax_val=is needed. na_pass: bool = False-
Should a row with a missing value in any of
columns=(which isn’t scored), or a row that the scorer gives no score, pass? The default isFalse, so such rows fail. cache: bool | str | Path = True-
Where to keep the scores, so that rows aren’t scored again:
True(the default) keeps them in memory for the session, a path is the directory of a cache on disk (an SQLite file,scores.sqlite) that’s kept between sessions, andFalsedoesn’t keep them. Scores are kept by the scorer’s name and version, and by the values of the row. A cache on disk needs a scorer with a stable name (see the Score Cache section). keep_scores: bool = True-
Should the results of the step keep the scores, in a column called
pb_score_? WithTrue(the default), the scores are in the extracts of failing rows, in the step report, and inget_row_flags()(aspb_score_<step>_), and they can be used byAblation.sweep_score(). pre: Callable | None = None-
An optional preprocessing function or lambda to apply to the data table during interrogation. This function should take a table as input and return a modified table. Have a look at the Preprocessing section for more information on how to use this argument.
segments: SegmentSpec | None = None-
An optional directive on segmentation, which serves to split a validation step into multiple (one step per segment). Can be a single column name, a tuple that specifies a column name and its corresponding values to segment on, or a combination of both (provided as a list).
thresholds: int | float | bool | tuple | dict | Thresholds | None = None-
Set threshold failure levels for reporting and reacting to exceedences of the levels. The thresholds are set at the step level and will override any global thresholds set in
Validate(thresholds=...). The default isNone, which means that no thresholds will be set locally and global thresholds (if any) will take effect. actions: Actions | None = None-
Optional actions to take when the validation step meets or exceeds any set threshold levels. If provided, the
Actionsclass should be used to define the actions. brief: str | bool | None = None-
An optional brief description of the validation step that will be displayed in the reporting table. You can use the templating elements like
"{step}"to insert the step number, or"{auto}"to include an automatically generated brief. IfTruethe entire brief will be automatically generated. IfNone(the default) then there won’t be a brief. active: bool | Callable = True-
A boolean value or callable that determines whether the validation step should be active. Using
Falsewill make the validation step inactive (still reporting its presence and keeping indexes for the steps unchanged). A callable can also be provided; it will receive the data table as its single argument and must return a boolean value. The callable is evaluated before anypre=processing. dimension: str | None = None-
An optional data quality dimension to categorize this validation step for health scoring. One of
"completeness","validity","uniqueness","consistency","timeliness", or"volume"(or any custom string). IfNone(the default), the dimension is"validity". This label appears in the validation report and feeds the overall and per-dimension health scores.
Returns
Validate- The Validate object with the added validation step.
Raises
TypeError-
If
columns=isn’t a column name or a list of them, ifscorer=isn’t callable, or ifmin_val=ormax_val=isn’t a number. ValueError-
If neither
min_val=normax_val=is given, ifmin_val=is greater thanmax_val=, or ifcache=is a directory and the scorer has no stable name.
Preprocessing
The pre= argument allows for a preprocessing function or lambda to be applied to the data table during interrogation. This function should take a table as input and return a modified table. This is useful for performing any necessary transformations or filtering on the data before the validation step is applied, such as taking a sample of the rows to keep the cost of scoring down.
The preprocessing function can be any callable that takes a table as input and returns a modified table. Regardless of which is used, the function should return a table with the columns of columns=.
How Rows Are Scored
The scorer gets the rows in batches (of the scorer’s batch_size, or 256 rows), as dictionaries of the values of columns=. Rows with the same values are scored once, and a row with a missing value in any of columns= isn’t scored at all (it passes or fails by na_pass=). A scorer that fails, or that returns something other than a number (or None) for each row, stops the step with an evaluation error and a note that says why, as does a scorer whose package isn’t installed (the note says how to install it).
Scoring runs in Python, on the rows’ values. A Polars or pandas table is scored as it is (a Polars LazyFrame is collected first). For a database table (through Ibis) or a PySpark DataFrame, the distinct rows of columns= are collected and scored, and the scores are joined back to the table on its own backend; other kinds of table give an evaluation error. The step’s note says which scorer was used, how many rows it scored, how many scores came from the cache, and the range of the scores.
Score Cache
Scores are kept in a cache by the scorer’s name and version (its name and version attributes) and by the values of the row, so that interrogating the same data again (as an Ablation does, or after a change to another step) doesn’t run the model again. The scores are written after each batch, so a run that’s interrupted picks up where it left off.
With cache=True, the scores are kept in memory until the end of the session (or until pb.scorers.clear_cache()). With a directory as cache=, they’re kept on disk and can be shared between sessions and projects; this needs a scorer with a stable name, which the adapters of pb.scorers have (unless they’re given a model object), as does a function defined at the top level of a module. A lambda can be given a name with pb.scorers.from_function(). Change the version of a scorer when its model changes, so that older scores aren’t used.
Examples
For the examples here, we’ll use a table of product reviews and a scorer that stands in for a model: it scores a review by the share of its words that are on a list of positive words. A real scorer would be a classifier, such as pb.scorers.classifier(hf_model=..., label="POSITIVE").
Let’s validate that each review scores at least 0.25.
validation = (
pb.Validate(data=tbl)
.col_vals_scored(columns="review", scorer=positivity, min_val=0.25)
.interrogate()
)
validationThe step has six test units, one for each row, and three fail: two reviews have no positive words, and the missing review isn’t scored (use na_pass=True to let such rows pass). The step’s note gives the range of the scores. The failing rows can be seen with their scores, in the pb_score_ column, with get_step_report().
validation.get_step_report(i=1)| Report for Validation Step 1 ✗ ASSERTION 3 / 6 TEST UNIT FAILURES EXTRACT OF ALL 3 ROWS (WITH TEST UNIT FAILURES IN RED): |
|||
The scores of a scored step can be used to see how the threshold affects a model trained on the data, by removing more or fewer rows; see Ablation.sweep_score().