Ablation.sweep_step()

Add a variant for each value of a step’s parameter, removing the rows that fail it.

Usage

Ablation.sweep_step(
    step,
    param,
    values,
)

A check’s bounds are often picked by hand: is min_val=0.5 the right cutoff, or would 0.3 keep more useful rows? This strategy measures it. For each value, the step is run again on the table with param= set to that value, and the variant removes the rows that fail it. The results trace how the model changes as the check gets stricter, and the report plots the curve, along with what removing as many random rows does.

Parameters

step: int: The number of the step to vary. It must be a row-level step that the ablation can use (active, and without a pre= function unless that function is marked with row_preserving()).
param: str: The name of the parameter of the step’s validation method to vary (e.g., "min_val" for col_vals_scored(), or "right" for col_vals_between()). It can’t be one of the parameters that don’t change which rows fail (thresholds=, actions=, brief=, active=, and dimension=), nor pre= or segments=.
values: list[float]: The values of param= to use, as numbers.

Returns

Ablation: The Ablation object, with the variants added.

Details

The step is run again as it was set up, with only param= changed, so a col_vals_scored() step takes its scores from the score cache instead of scoring the rows again. For a segmented step, only the step’s own segment is checked. A value that the validation method doesn’t accept (e.g., a min_val= above the step’s max_val=) gives an error. As with any other variant, a value whose step fails no rows is left out, and a value that removes the same rows as another version of the table isn’t evaluated again.

Examples

Here’s a table of temperature readings, some of them recorded in tenths of a degree, with a check that the readings are below an upper limit. The ‘model’ estimates the average temperature and is evaluated by its error against the true average of 20.

import pointblank as pb
import polars as pl

temps = [20 + (k * 37 % 11 - 5) * 0.6 for k in range(300)]
for k in range(0, 300, 25):
    temps[k] *= 10
readings = pl.DataFrame({"temp": temps})

def evaluate(train, *, seed):
    sample = train.sample(fraction=1.0, with_replacement=True, seed=seed)
    return abs(sample["temp"].mean() - 20)

validation = (
    pb.Validate(data=readings)
    .col_vals_lt(columns="temp", value=100)
    .interrogate()
)

ablation = (
    pb.Ablation(validation, evaluate=evaluate, metric="error", higher_is_better=False)
    .sweep_step(1, param="value", values=[21, 50, 190, 220])
    .run()
)

ablation

Variant	Removed	error	Δ vs random
Ablation report
error on the full table (300 rows): 7.392 ± 2.207 (lower is better). Each version was evaluated 5 times; Δ is the mean paired difference from removing the same number of random rows.
Sweep: Step 1, varying value
Step 1 with value=21 The rows that fail step 1 with `value=21`	117 39.0%	1.191 ± 0.065 Δ vs full −6.201	−5.803 [−8.436, −3.169] helps
Step 1 with value=50 The rows that fail step 1 with `value=50`	12 4.0%	0.109 ± 0.080 Δ vs full −7.283	−6.901 [−8.634, −5.168] helps
Step 1 with value=190 The rows that fail step 1 with `value=190`	7 2.3%	2.493 ± 1.666 Δ vs full −4.899	−4.978 [−7.564, −2.392] helps
Step 1 with value=220 The rows that fail step 1 with `value=220`	2 0.7%	5.764 ± 1.863 Δ vs full −1.628	−2.045 [−2.706, −1.385] helps
Intervals are 95% paired t-intervals, not adjusted for multiple comparisons; verdicts use Holm-adjusted p-values.

A limit of 50 removes every bad reading. A limit of 21 also removes many good readings (the warmer ones), which leaves the average too low, and the limits of 190 and 220 miss some of the bad readings.