DataScanDiff

The result of comparing two DataScan profiles.

Usage

DataScanDiff(
    current,
    baseline,
)

A DataScanDiff is what DataScan.compare() returns, and it isn’t meant to be created directly. It records how a table’s profile changed between a baseline scan and a current one: which columns were added, removed, or changed type, which statistics changed and by how much, and how far each column’s distribution drifted (see DataScan.compare() for how the drift scores are computed).

The results can be used in three ways. The attributes and the has_changes property suit checks in code, such as stopping a pipeline when a column disappears. to_dict() gives every difference as plain data, for logging or alerting. And get_tabular_report() shows the differences as a table, for a person to review.

Parameters

current: DataScan: The newer DataScan (the one whose compare() method was called).
baseline: DataScan: The older DataScan that current is compared against.

Attributes

columns_added: list[str]: Column names present in the current scan but not the baseline, sorted alphabetically.
columns_removed: list[str]: Column names present in the baseline but not the current scan, sorted alphabetically.
columns_type_changed: list[str]: Column names whose data type changed between baseline and current.
column_diffs: list[_ColumnDiff]: Per-column diff details for all columns that appear in either scan, with the baseline’s columns first. Each entry has the attributes colname, coltype_baseline and coltype_current (None for a column missing from that scan), status (one of "added", "removed", "type_changed", "changed", or "unchanged"), stat_diffs (a dictionary mapping each changed statistic to a (baseline, current) tuple), and drift_scores (a dictionary of drift scores, which might be empty).

Examples

In this example, a table gains a row and a note column, and its score column changes from integers to strings.

import pointblank as pb
import polars as pl

old = pl.DataFrame({"id": [1, 2, 3], "score": [7, 9, 8]})

new = pl.DataFrame(
    {"id": [1, 2, 3, 4], "score": ["7", "9", "8", "6"], "note": ["a", None, "b", None]}
)

diff = pb.DataScan(new).compare(pb.DataScan(old))

diff.columns_added, diff.columns_removed, diff.columns_type_changed

(['note'], [], ['score'])

The schema changes are available directly as lists. For a look at each column, go through column_diffs:

for col in diff.column_diffs:
    print(col.colname, col.status, col.coltype_baseline, "->", col.coltype_current)

id changed Int64 -> Int64
score type_changed Int64 -> String
note added None -> String

The id column kept its type, but its statistics changed because of the new row. A pipeline could stop whenever columns_removed or columns_type_changed isn’t empty, since those changes tend to break code that reads the table.

Attributes

Name	Description
has_changes	Return `True` if any schema or statistical changes were detected.
row_count_diff	Return `(baseline_row_count, current_row_count)`.

has_changes

Return True if any schema or statistical changes were detected.

has_changes: bool

This is the quickest check of whether a table changed. It’s True if a column was added, removed, or changed type, or if any statistic or drift score differs between the two scans. A change in the row count alone doesn’t count, since tables often grow without changing in character. Check row_count_diff for that.

Examples

Here, a log of job runs grows from three rows to five, but every status is still "ok".

import pointblank as pb
import polars as pl

runs_before = pl.DataFrame({"status": ["ok"] * 3})
runs_after = pl.DataFrame({"status": ["ok"] * 5})

diff = pb.DataScan(runs_after).compare(pb.DataScan(runs_before))

diff.has_changes

False

The table has more rows, but nothing about its columns changed (they have the same values in the same proportions), so has_changes is False. A single "failed" status in the new rows would have changed the number of distinct values and made it True.

row_count_diff

Return (baseline_row_count, current_row_count).

row_count_diff: tuple[int, int]

The row counts are kept apart from the other differences, since a change in size is often expected (a table that collects events grows every day). Having both counts lets you decide what change is acceptable, such as a minimum growth rate or a maximum drop.

Examples

Let’s compare two versions of a log of job runs, where the newer one has two more rows.

import pointblank as pb
import polars as pl

runs_before = pl.DataFrame({"status": ["ok"] * 3})
runs_after = pl.DataFrame({"status": ["ok"] * 5})

diff = pb.DataScan(runs_after).compare(pb.DataScan(runs_before))

baseline_rows, current_rows = diff.row_count_diff

baseline_rows, current_rows

(3, 5)

The baseline has three rows and the current scan five. The has_changes property is False here, so code that should also react to a table shrinking (for instance, when a load was incomplete) needs to check these counts.

Methods

Name	Description
get_tabular_report()	Generate a GT table summarizing the differences between the two scans.
to_dict()	Export the comparison results as a dictionary.

get_tabular_report()

Generate a GT table summarizing the differences between the two scans.

Usage

Source

get_tabular_report()

The report is for a person reviewing the changes: it has one row per column of either scan, in the order of the baseline’s columns followed by the added ones. The title names the two tables (baseline first) and the subtitle gives both row counts. The Status column says what happened to each column:

Added or Removed: the column is in only one of the scans
Type Changed or Type + Stats Changed: the data type changed (and maybe the statistics too)
Stats Changed: the type is the same, but statistics or drift scores differ
OK: nothing changed

The types in the baseline and the current scan come next, then every changed statistic with its baseline and current values. If any column has drift scores, a final column shows them (PSI, and for numeric columns the KS statistic with its p-value).

This method needs Polars, whatever the type of the scanned tables.

Returns

GT: A styled Great Tables report showing schema and statistical drift.

Examples

Let’s compare a scan of the small_table dataset with a scan of an older version, where the f column only had "low" and "high" values and there was no e column.

import pointblank as pb
import polars as pl

small_table = pb.load_dataset(dataset="small_table", tbl_type="polars")

older = small_table.filter(pl.col("f") != "mid").drop("e")

diff = pb.DataScan(small_table, tbl_name="current").compare(
    pb.DataScan(older, tbl_name="older")
)

diff.get_tabular_report()

Column	Status	Type (Baseline)	Type (Current)	Changed Statistics	Drift Scores
Profile Comparison: older vs current
Row count: 11 (baseline) vs 13 (current)
date_time	Stats Changed	Datetime(time_unit='us', time_zone=None)	Datetime(time_unit='us', time_zone=None)	n_unique: 10 -> 12	PSI: 1.0467
date	Stats Changed	Date	Date	n_unique: 9 -> 11	PSI: 1.0467
a	Stats Changed	Int64	Int64	mean: 3.909 -> 3.769 std: 2.212 -> 2.088 p05: 1.05 -> 1.06 q_1: 2.5 -> 2 q_3: 5 -> 4 p95: 7.5 -> 7.4 iqr: 2.5 -> 2	PSI: 0.0397 KS: 0.0420 (p=1.0000)
b	Stats Changed	String	String	n_unique: 10 -> 12	PSI: 1.0467
c	Stats Changed	Int64	Int64	n_missing: 1 -> 2 n_unique: 6 -> 7 mean: 5.9 -> 5.727 std: 2.807 -> 2.724 p05: 2.045 -> 2.05	PSI: 0.0183 KS: 0.0545 (p=1.0000)
d	Stats Changed	Float64	Float64	n_unique: 10 -> 12 mean: 2071 -> 2305 median: 843.3 -> 1036 std: 2811 -> 2631 p05: 117.1 -> 118.9 q_1: 836 -> 837.9 q_3: 2287 -> 3291 p95: 6712 -> 6335 iqr: 1451 -> 2453	PSI: 0.0635 KS: 0.1259 (p=1.0000)
f	Stats Changed	String	String	n_unique: 2 -> 3 mean: 3.545 -> 3.462 median: 4 -> 3 std: 0.5222 -> 0.5189	PSI: 1.1540
e	Added		Boolean

The e column shows as added, and the other columns have changed statistics, since the older version had two fewer rows. The PSI of f (about 1.15) comes from the "mid" values that are only in the current version. The date_time, date, and b columns get a similar PSI, but for a different reason: nearly all of their values are distinct, so removing any rows changes their value frequencies a lot. The PSI of a non-numeric column means most for columns with a few distinct values, like f. Like other GT tables, the report can be saved as HTML with write_raw_html(), to share with others who want to review the changes.

to_dict()

Export the comparison results as a dictionary.

Usage

Source

to_dict()

The dictionary has every difference the comparison found, in plain Python data. That suits logging, alerting (by sending only the parts that matter), or storing the results next to the scans that produced them. Columns without changes are left out, so the dictionary only grows with the number of differences.

Returns

dict[str, Any]: A dictionary with these keys: "row_count" (a dictionary with the "baseline" and "current" row counts), "columns_added" and "columns_removed" (lists of column names), "columns_type_changed" (a list of dictionaries with "column", "baseline_type", and "current_type"), "stat_diffs" (a dictionary that maps each column with changed statistics to a dictionary that maps each statistic to its "baseline" and "current" values), and "drift_scores" (a dictionary mapping each column that has drift scores to those scores).

Examples

Let’s compare two versions of a table of orders. The newer version has one more order, a refund for -15.5.

import pointblank as pb
import polars as pl

orders = pl.DataFrame({"order_id": [1, 2, 3, 4], "amount": [25.0, 40.0, 15.5, 60.0]})

orders_now = pl.DataFrame(
    {"order_id": [1, 2, 3, 4, 5], "amount": [25.0, 40.0, 15.5, 60.0, -15.5]}
)

diff_dict = pb.DataScan(orders_now).compare(pb.DataScan(orders)).to_dict()

diff_dict["row_count"], diff_dict["columns_added"], diff_dict["columns_removed"]

({'baseline': 4, 'current': 5}, [], [])

There are no schema changes. The changed statistics of amount include its minimum:

diff_dict["stat_diffs"]["amount"]["min"]

{'baseline': 15.5, 'current': -15.5}

Picking out an entry like this one makes it easy to raise an alert on specific changes (here, a negative amount) and ignore others, such as a mean that moves a little as new orders come in.

Parameters

Attributes

Examples

See Also

Attributes

has_changes

Examples

row_count_diff

Examples

Methods

get_tabular_report()

Returns

Examples

to_dict()

Returns

Examples