Ablation.sweep_score()

Add a variant for each cutoff of a score, removing the rows on the wrong side of it.

Usage

Source

Ablation.sweep_score(
    score,
    quantiles=None,
    values=None,
    remove=None,
)

A score (e.g., from a quality classifier) doesn’t say where to draw the line between the rows to keep and those to remove. This strategy measures it: each variant removes more of the lowest-scoring rows than the last, so the results trace how the model changes as the cutoff gets stricter (a dose-response curve). The report plots the curve, along with what removing as many random rows does. The cutoffs are given as shares of the scored rows to remove (quantiles=) or as score values (values=).

Parameters

score: int | str

Where the scores come from: the number of a col_vals_scored() step that kept its scores (keep_scores=True, the default), or the name of a numeric column of the table.

quantiles: list[float] | None = None

The shares of the scored rows to remove, each greater than 0 and at most 1 (e.g., 0.1 removes the tenth of the rows with the lowest scores). This is the default, with the shares 0.05, 0.1, 0.2, 0.3, 0.4, and 0.5.

values: list[float] | None = None

Score cutoffs to use instead of quantiles=: each variant removes the rows that score below the cutoff (or above it, with remove="highest").

remove: str | None = None

Which rows to remove: "lowest" (the lowest-scoring rows) or "highest". By default, it’s "highest" for a step with only max_val= (e.g., a toxicity or perplexity score, where a high score is bad) and "lowest" otherwise.

Returns

Ablation

The Ablation object, with the variants added.

Details

The rows without a score (a missing value, a row the scorer gave no score, or a row outside a segmented step’s segment) are never removed, and a note says how many there are. With quantiles=, the rows are ranked by their score, and ties are broken by row order, so each share removes a set number of rows. Each point of the sweep is a variant like any other: it’s compared with removing the same number of random rows and gets a verdict. A point that removes no rows is left out, and a point that removes the same rows as another version of the table isn’t evaluated again.

Examples

Here’s a table of temperature readings, some of them recorded in tenths of a degree, and a scorer that rates how plausible each reading is (from 0 to 1). The ‘model’ estimates the average temperature and is evaluated by its error against the true average of 20.

import pointblank as pb
import polars as pl

temps = [20 + (k * 37 % 11 - 5) * 0.6 for k in range(300)]
for k in range(0, 300, 25):
    temps[k] *= 10
readings = pl.DataFrame({"temp": temps})

def plausibility(rows):
    return [1 / (1 + abs(row["temp"] - 20) / 5) for row in rows]

def evaluate(train, *, seed):
    sample = train.sample(fraction=1.0, with_replacement=True, seed=seed)
    return abs(sample["temp"].mean() - 20)

validation = (
    pb.Validate(data=readings)
    .col_vals_scored(columns="temp", scorer=plausibility, min_val=0.5)
    .interrogate()
)

ablation = (
    pb.Ablation(validation, evaluate=evaluate, metric="error", higher_is_better=False)
    .sweep_score(1, quantiles=[0.02, 0.04, 0.1, 0.2])
    .run()
)

ablation

Ablation report
error on the full table (300 rows): 7.392 ± 2.207 (lower is better). Each version was evaluated 5 times; Δ is the mean paired difference from removing the same number of random rows.
Variant	Removed	error	Δ vs random
Sweep: Step 1, removing the lowest-scoring rows
Step 1: lowest 2% of scores The 6 rows with the lowest scores (of 300 scored rows)	6 2.0%	3.449 ± 1.759 Δ vs full −3.943	−3.706 [−4.900, −2.512] helps
Step 1: lowest 4% of scores The 12 rows with the lowest scores (of 300 scored rows)	12 4.0%	0.109 ± 0.080 Δ vs full −7.283	−6.901 [−8.634, −5.168] helps
Step 1: lowest 10% of scores The 30 rows with the lowest scores (of 300 scored rows)	30 10.0%	0.113 ± 0.104 Δ vs full −7.279	−8.524 [−10.645, −6.403] helps
Step 1: lowest 20% of scores The 60 rows with the lowest scores (of 300 scored rows)	60 20.0%	0.111 ± 0.079 Δ vs full −7.281	−7.245 [−9.099, −5.392] helps
Intervals are 95% paired t-intervals, not adjusted for multiple comparisons; verdicts use Holm-adjusted p-values.

Removing the lowest-scoring 4% of the rows takes out every bad reading, and the curve below the report shows the error dropping to near zero there. Removing more rows than that makes no further difference.