Check the splits of a dataset for leaks, label coverage, and drift.
check_splits(
train=None,
valid=None,
test=None,
*,
splits=None,
columns=None,
label=None,
near=False,
threshold=0.8,
ngram=5,
num_perm=128,
drift_columns=None,
)
A model is trained on one split of a dataset and judged on another, so the judgment is only fair if the splits are independent and alike. This checks three ways that they can fail to be:
- Overlap. A row of a later split (e.g., of the test split) that’s also in an earlier one (e.g., in the training split) leaks: the model is judged on data it has seen, so it looks better than it is. Rows overlap if they have the same values of
columns=; with near=True, rows whose text is nearly the same (a Jaccard similarity of their word n-grams of at least threshold=, found with MinHash) overlap too, which catches copies that differ by a few words, by case, or by punctuation.
- Labels. With
label=, every label of the first split should be in the others (or the model is never judged on it), no split should have a label that isn’t in the first (the model never learns it), and the shares of the labels should be about the same.
- Drift. The distribution of each column of
drift_columns= in each split is compared with that in the first split, with the population stability index (PSI, from binned shares of the values) and, for a numeric column, the Kolmogorov-Smirnov (KS) statistic. A PSI of at least 0.1 is a moderate change, and one of at least 0.2 is a major change.
The splits can be tables of any kind that Pointblank can validate (or the paths of files). The columns that are checked are read into memory, so a split in a database is read in full.
Parameters
train: Any = None-
The splits, in that order (give at least two). Or use splits= instead.
valid: Any = None-
The splits, in that order (give at least two). Or use splits= instead.
test: Any = None-
The splits, in that order (give at least two). Or use splits= instead.
splits: Mapping[str, Any] | None = None-
A dictionary of the splits, by name, in order (the first one is the one that the others are compared with).
columns: str | list[str] | None = None-
The columns whose values identify a row, for the check of overlap: a column, or a list of them. By default, they’re all the columns that the splits share, except for label=.
label: str | None = None-
The column with the labels, if any. A numeric label with more than 50 distinct values (e.g., the target of a regression) is checked for drift only.
near: bool = False-
Whether to find rows whose text is nearly the same, as well as rows that are the same.
threshold: float = 0.8-
The smallest Jaccard similarity of two rows’ word n-grams for them to be nearly the same (with near=True).
ngram: int = 5-
The number of words of the n-grams (with near=True).
num_perm: int = 128-
The number of MinHash permutations (with near=True).
drift_columns: str | list[str] | bool | None = None-
The columns to check for drift. By default, they’re the numeric columns that the splits share and the others with few distinct values in the first split (at most 50, and at most half as many as its rows, which leaves out free text and identifiers), except for
label=. Use False to check none.
Examples
Here are the training and test splits of a dataset of support tickets, labelled with their topic. Two of the test tickets are copies of training tickets: one exactly, and one with different case and punctuation.
import pointblank as pb
import polars as pl
topics = ["billing", "login", "shipping", "billing", "returns"]
train = pl.DataFrame(
{
"text": [f"Ticket {k}: question number {k} about {topics[k % 5]}" for k in range(40)],
"topic": [topics[k % 5] for k in range(40)],
"length": [20 + k % 7 for k in range(40)],
}
)
test = pl.DataFrame(
{
"text": [
"Ticket 3: question number 3 about billing",
"ticket 7 -- question NUMBER 7 about shipping!",
]
+ [f"New ticket {k} about a lost parcel" for k in range(10)],
"topic": ["billing", "shipping"] + ["shipping"] * 10,
"length": [20, 22] + [40 + k for k in range(10)],
}
)
check = pb.check_splits(
train=train, test=test, columns="text", label="topic", near=True, ngram=3
)
check
| Split check |
train: 40 rows · test: 12 rows 4 issues found. |
| Check |
Result |
| Overlap |
| Rows of test also in train compared on text |
LEAK 2 of 12 rows (16.7%) 1 the same, 1 nearly the same |
| Labels |
| Labels of train in topic |
BASE billing 40% · login 20% · returns 20% · shipping 20% 4 labels |
| Labels of test in topic, compared with train |
ISSUE shipping 92% · billing 8% missing 'login', 'returns'; PSI 4.63 |
| Drift |
length in testcompared with train |
MAJOR PSI 5.6 numeric; KS 0.833 (p = 0) |
| Rows are nearly the same if the Jaccard similarity of their 3-word n-grams is at least 0.8 (found with MinHash, with 128 permutations). |
| PSI is the population stability index of the shares of the values (binned by the deciles of 'train', for a numeric column): a change is moderate from 0.1 and major from 0.2. KS is the Kolmogorov-Smirnov statistic. |
The report shows both copies, and that the test split has none of the tickets about logins or returns, so the model would never be judged on those. The tickets are also much longer in the test split. The overlapping rows can be had as a table:
check.get_overlap("train", "test")
shape: (2, 4)| train_row | test_row | overlap | text |
|---|
| i64 | i64 | str | str |
| 4 | 1 | "exact" | "Ticket 3: question number 3 ab… |
| 8 | 2 | "near" | "ticket 7 -- question NUMBER 7 … |
And check.issues lists the problems, so that a pipeline can stop when there are any.