DataScan.compare()

Compare this scan against a baseline and return the differences.

Usage

DataScan.compare(baseline)

Tables that are refreshed regularly (a monthly extract, a nightly load, a model’s input data) can change in ways that no single row reveals: a column is dropped, a type changes, or the values shift to a different range. Comparing a scan of the current table with a scan of an earlier version that’s known to be good finds those changes. The baseline is often a profile saved earlier with save_to_json() and brought back with load_from_json().

The scan this method is called on is the current one, and baseline is the one it’s compared against. The result is a DataScanDiff, which has the differences as attributes, as a dictionary through to_dict(), and as a report through get_tabular_report().

This compares profiles, not rows. To find out which rows of two tables differ, use compare() instead.

What Is Compared

Columns are matched by name. A column that’s only in the current scan is added, one that’s only in the baseline is removed, and one whose data type differs is type-changed. For every column in both scans, each statistic is compared for exact equality, so any difference at all (even in the last decimal place of a mean) is reported along with both values. When the type has changed, the statistics might no longer measure the same thing (a numeric column’s mean is of its values, but a string column’s mean is of its lengths).

Drift Scores

Statistics show that something changed but not how much the distribution of values moved. For that, drift scores are computed for columns whose type hasn’t changed:

numeric columns get the population stability index (psi), with the values divided into ten bins at the baseline’s deciles (at least ten values are needed on each side), and the Kolmogorov–Smirnov statistic (ks_statistic, the largest gap between the two cumulative distributions) with its p-value (ks_p_value). The p-value comes from SciPy if it’s installed and from an approximation otherwise
all other columns get a psi computed from how often each distinct value appears

A common rule of thumb reads a PSI below 0.1 as little or no shift, from 0.1 to 0.25 as a moderate shift, and above 0.25 as a large one. A score of zero is left out.

Drift scores need the values themselves, so they’re only computed when both scans were made from data. A scan restored with from_dict(), from_json(), or load_from_json() only has statistics, and then only Boolean columns (whose value counts are part of the profile) get a drift score.

Parameters

baseline: DataScan: The baseline DataScan to compare against (typically the older scan).

Returns

DataScanDiff: An object describing the differences between the two scans.

Examples

Let’s compare two monthly extracts of an orders table. Between January and February, the gift column was dropped, a coupon column was added, a new "app" channel appeared, and order amounts went up.

import pointblank as pb
import polars as pl

jan = pl.DataFrame(
    {
        "order_id": list(range(1, 21)),
        "amount": [20.0 + i for i in range(20)],
        "channel": ["web"] * 14 + ["store"] * 6,
        "gift": [i % 4 == 0 for i in range(20)],
    }
)

feb = pl.DataFrame(
    {
        "order_id": list(range(21, 45)),
        "amount": [35.0 + 1.5 * i for i in range(24)],
        "channel": ["web"] * 10 + ["store"] * 8 + ["app"] * 6,
        "coupon": ["SPRING"] * 5 + [None] * 19,
    }
)

scan_jan = pb.DataScan(jan, tbl_name="orders_jan")
scan_feb = pb.DataScan(feb, tbl_name="orders_feb")

diff = scan_feb.compare(scan_jan)

diff

DataScanDiff(added=1, removed=1, type_changed=0, stat_changed=3)

The printed summary counts one added column, one removed column, and three columns whose statistics changed. The report shows the details:

diff.get_tabular_report()

Column	Status	Type (Baseline)	Type (Current)	Changed Statistics	Drift Scores
Profile Comparison: orders_jan vs orders_feb
Row count: 20 (baseline) vs 24 (current)
order_id	Stats Changed	Int64	Int64	n_unique: 20 -> 24 mean: 10.5 -> 32.5 median: 10.5 -> 32.5 std: 5.916 -> 7.071 min: 1 -> 21 max: 20 -> 44 p05: 1.095 -> 21.11 q_1: 5.75 -> 26.75 q_3: 15.25 -> 38.25 p95: 19.05 -> 42.85 iqr: 9.5 -> 11.5	PSI: 8.2831 KS: 1.0000 (p=0.0000)
amount	Stats Changed	Float64	Float64	n_unique: 20 -> 24 mean: 29.5 -> 52.25 median: 29.5 -> 52.25 std: 5.916 -> 10.61 min: 20 -> 35 max: 39 -> 69.5 p05: 20.09 -> 35.17 q_1: 24.75 -> 43.62 q_3: 34.25 -> 60.88 p95: 38.05 -> 67.77 iqr: 9.5 -> 17.25	PSI: 6.7421 KS: 0.8750 (p=0.0000)
channel	Stats Changed	String	String	n_unique: 2 -> 3 mean: 3.6 -> 3.667 std: 0.9403 -> 0.9631	PSI: 2.1057
gift	Removed	Boolean
coupon	Added		String

The PSI of amount (about 6.7) and its KS statistic (0.875) are far beyond the rule of thumb, so the amounts really did move. The channel column has a PSI of about 2.1 because of the new "app" value. The shift in order_id is expected, since each month has its own IDs, which shows that drift scores still need to be read with the meaning of each column in mind.

What Is Compared

Drift Scores

Parameters

Returns

Examples

See Also