Reference

Validate

When performing data validation, use the Validate class to get the process started. It takes the target table and options for metadata and failure thresholds (using the Thresholds class or shorthands). The Validate class has numerous methods for defining validation steps and for obtaining post-interrogation metrics and data.

Validate: Workflow for defining a set of validations on a table and interrogating for results.
Thresholds: Definition of threshold values.
Actions: Definition of action values.
FinalActions: Define actions to be taken after validation is complete.
Schema: Definition of a schema object.
DraftValidation: Draft a validation plan for a given table using an LLM.
EditValidation: Edit an existing validation plan with a plain-English instruction using an LLM.
MissingSpec: Specification for structured missing values in a column.

Contracts and Pipelines

Use Contract and Step to define declarative data quality contracts that specify what valid data looks like. Use Steps to build reusable step libraries with a fluent API and compose them into Validate plans via add_steps(). Use Pipeline to enforce contracts at both boundaries of a data transformation (source and target), producing a PipelineResult with full introspection into what passed and what failed.

Contract: A declarative boundary contract for pipeline data.
Step: A single validation step, stored as data.
Steps: A reusable collection of validation step definitions.
Validate.add_steps(): Add the steps from one or more Steps collections to the validation.
Pipeline: Enforce data contracts at both ends of a data transformation.
PipelineResult: The outcome of running a data pipeline.
ContractViolationError: The error raised when data fails a contract with on_violation="raise".

Contract Import/Export

Import external schema definitions (JSON Schema, Frictionless Table Schema, and more) into Pointblank validation workflows, or export Pointblank contracts to those formats. Use import_contract() as the entry point, export_contract() for the reverse, and register_adapter() to add support for custom formats.

import_contract(): Import a contract or schema written in another format.
export_contract(): Export a Pointblank validation or contract to another format.
list_adapters(): List the formats available for contract import and export.
register_adapter(): Register a contract adapter so that it can be used for import and export.
ContractImport: The result of importing a contract or schema from another format.
ContractAdapter: The base class for adapters that translate contracts to and from other formats.
adapters.MappedConstraint: A rule from an external format, translated into a Pointblank validation step.

Validation Steps

Validation steps are sequential validations on the target data. Call Validate’s validation methods to build up a validation plan: a collection of steps that provides good validation coverage.

Validate.col_vals_gt(): Are column data greater than a fixed value or data in another column?
Validate.col_vals_lt(): Are column data less than a fixed value or data in another column?
Validate.col_vals_ge(): Are column data greater than or equal to a fixed value or data in another column?
Validate.col_vals_le(): Are column data less than or equal to a fixed value or data in another column?
Validate.col_vals_eq(): Are column data equal to a fixed value or data in another column?
Validate.col_vals_ne(): Are column data not equal to a fixed value or data in another column?
Validate.col_vals_between(): Do column data lie between two specified values or data in other columns?
Validate.col_vals_outside(): Do column data lie outside of two specified values or data in other columns?
Validate.col_vals_in_set(): Validate whether column values are in a set of values.
Validate.col_vals_not_in_set(): Validate whether column values are not in a set of values.
Validate.col_vals_increasing(): Are column data increasing by row?
Validate.col_vals_decreasing(): Are column data decreasing by row?
Validate.col_vals_null(): Validate whether values in a column are Null.
Validate.col_vals_not_null(): Validate whether values in a column are not Null.
Validate.col_vals_regex(): Validate whether column values match a regular expression pattern.
Validate.col_vals_within_spec(): Validate whether column values fit within a specification.
Validate.col_vals_str_len(): Validate whether the length of string values falls within specified bounds.
Validate.col_vals_in_table(): Validate that column values exist in a reference table (referential integrity).
Validate.col_vals_expr(): Validate column values using a custom expression.
Validate.col_exists(): Validate whether one or more columns exist in the table.
Validate.col_pct_null(): Validate whether a column has a specific percentage of Null values.
Validate.col_pct_missing(): Validate that the percentage of structured missing values stays within a limit.
Validate.col_missing_coded(): Validate that all missing values in a column are coded (no uncoded nulls).
Validate.col_missing_only_coded(): Validate that a column contains only documented codes and legitimate values.
Validate.col_missing_consistent(): Validate that related columns share a consistent missingness pattern for a given reason.
Validate.rows_distinct(): Validate whether rows in the table are distinct.
Validate.rows_complete(): Validate whether row data are complete by having no missing values.
Validate.col_schema_match(): Do columns in the table (and their types) match a predefined schema?
Validate.row_count_match(): Validate whether the row count of the table matches a specified count.
Validate.col_count_match(): Validate whether the column count of the table matches a specified count.
Validate.data_freshness(): Validate that data in a datetime column is not older than a specified maximum age.
Validate.tbl_match(): Validate whether the target table matches a comparison table.
Validate.tbl_reconcile(): Validate that aggregates of the target table reconcile with a comparison table.
Validate.conjointly(): Perform multiple row-wise validations for joint validity.
Validate.specially(): Perform a specialized validation with customized logic.
Validate.prompt(): Validate rows using AI/LLM-powered analysis.

Aggregation Steps

These validation methods check aggregated column values (sums, averages, standard deviations) against fixed values or column references.

Validate.col_sum_gt(): Does the column sum satisfy a greater than comparison?
Validate.col_sum_lt(): Does the column sum satisfy a less than comparison?
Validate.col_sum_ge(): Does the column sum satisfy a greater than or equal to comparison?
Validate.col_sum_le(): Does the column sum satisfy a less than or equal to comparison?
Validate.col_sum_eq(): Does the column sum satisfy an equal to comparison?
Validate.col_avg_gt(): Does the column average satisfy a greater than comparison?
Validate.col_avg_lt(): Does the column average satisfy a less than comparison?
Validate.col_avg_ge(): Does the column average satisfy a greater than or equal to comparison?
Validate.col_avg_le(): Does the column average satisfy a less than or equal to comparison?
Validate.col_avg_eq(): Does the column average satisfy an equal to comparison?
Validate.col_sd_gt(): Does the column standard deviation satisfy a greater than comparison?
Validate.col_sd_lt(): Does the column standard deviation satisfy a less than comparison?
Validate.col_sd_ge(): Does the column standard deviation satisfy a greater than or equal to comparison?
Validate.col_sd_le(): Does the column standard deviation satisfy a less than or equal to comparison?
Validate.col_sd_eq(): Does the column standard deviation satisfy an equal to comparison?

Column Selection

Use the col() function along with column selection helpers to flexibly select columns for validation. Combine col() with starts_with(), matches(), etc. for selecting multiple target columns.

col(): Helper function for referencing a column in the input table.
starts_with(): Select columns that start with specified text.
ends_with(): Select columns that end with specified text.
contains(): Select columns that contain specified text.
matches(): Select columns that match a specified regular expression pattern.
everything(): Select all columns.
first_n(): Select the first n columns in the column list.
last_n(): Select the last n columns in the column list.
expr_col(): Create a column expression for use in conjointly() validation.

Segment Groups

Combine multiple values into a single segment using seg_*() helper functions.

seg_group(): Group together values for segmentation.

Interrogation and Reporting

The validation plan is executed when interrogate() is called. After interrogation, view validation reports, extract metrics, or split data based on results.

Validate.interrogate(): Execute each validation step against the table and store the results.
Validate.set_tbl(): Set or replace the table associated with the Validate object.
Validate.get_tabular_report(): Validation report as a GT table.
Validate.get_step_report(): Get a detailed report for a single validation step.
Validate.get_json_report(): Get a report of the validation results as a JSON-formatted string.
Validate.get_dataframe_report(): Get a report of the validation results as a DataFrame.
Validate.to_code(): Render this validation plan as canonical Pointblank Python code.
Validate.to_yaml(): Serialize this validation plan to a yaml_interrogate()-compatible YAML config.
Validate.to_json_schema(): Export this validation plan as a JSON Schema document.
Validate.suggest_improvements(): Propose AI-generated improvements to this validation plan.
Validate.from_prompt(): Build a validation plan for this table from a natural-language prompt.
Validate.get_sundered_data(): Get the data that passed or failed the validation steps.
Validate.get_data_extracts(): Get the rows that failed for each validation step.
Validate.all_passed(): Determine if every validation step passed perfectly, with no failing test units.
Validate.assert_passing(): Raise an AssertionError if all tests are not passing.
Validate.assert_below_threshold(): Raise an AssertionError if validation steps exceed a specified threshold level.
Validate.assert_dimension_scores(): Raise an AssertionError if any dimension’s health score falls below a minimum.
Validate.above_threshold(): Check if any validation steps exceed a specified threshold level.
Validate.get_dimension_scores(): Get per-dimension health scores from the validation results.
Validate.get_health_score(): Get the overall data quality health score from the validation results.
Validate.get_scorecard(): Get a data quality scorecard as a GT table.
Validate.n(): Provides a dictionary of the number of test units for each validation step.
Validate.n_passed(): Provides a dictionary of the number of test units that passed for each validation step.
Validate.n_failed(): Provides a dictionary of the number of test units that failed for each validation step.
Validate.f_passed(): Provides a dictionary of the fraction of test units that passed for each validation step.
Validate.f_failed(): Provides a dictionary of the fraction of test units that failed for each validation step.
Validate.warning(): Get the ‘warning’ level status for each validation step.
Validate.error(): Get the ‘error’ level status for each validation step.
Validate.critical(): Get the ‘critical’ level status for each validation step.

Inspection and Assistance

Functions for getting to grips with a new data table. Use DataScan for a quick overview, preview() for first/last rows, col_summary_tbl() for column summaries, and missing_vals_tbl() for missing value analysis.

DataScan: Get a summary of a dataset.
DataScanDiff: The result of comparing two DataScan profiles.
preview(): Display a table preview that shows some rows from the top, some from the bottom.
col_summary_tbl(): Generate a column-level summary table of a dataset.
missing_vals_tbl(): Display a table that shows the missing values in the input table.
load_dataset(): Load a dataset hosted in the library as specified table type.
get_data_path(): Get the file path to a dataset included with the Pointblank package.
connect_to_table(): Connect to a database table using a connection string.
print_database_tables(): List all tables in a database from a connection string.

DataScan Methods

Methods for the DataScan class

DataScan.get_tabular_report(): Get a tabular report of the scan.
DataScan.summary_data: The column statistics as a DataFrame.
DataScan.compare(): Compare this scan against a baseline and return the differences.
DataScan.save_to_json(): Save the profile to a JSON file.
DataScan.load_from_json(): Load a DataScan from a JSON file produced by save_to_json().
DataScan.to_json(): Export the profile as a JSON string.
DataScan.from_json(): Restore a DataScan from a JSON string.
DataScan.to_dict(): Export the profile as a structured dictionary.
DataScan.from_dict(): Restore a DataScan from a dictionary.

Table Comparison

Compare two tables row by row. compare() aligns the rows of a source and a target table (by key, by position, or as multisets) and reports missing, extra, and changed rows, plus schema differences. The result is a Comparison object with counts, row samples, and a tabular report. The same comparison is available as a validation step through Validate.tbl_match(), and Validate.tbl_reconcile() compares aggregates (e.g., totals per group) instead of rows.

compare(): Compare two tables row by row and report how they differ.
Comparison: The result of comparing a source table with a target table.
SchemaDiff: Differences between the schemas of the source and target tables.

Comparison Methods

Methods for the Comparison class

Comparison.identical: Whether the tables match under the comparison settings.
Comparison.schema_ok: Whether the schema check passed.
Comparison.status_counts: The number of test units with each row status.
Comparison.n: The number of test units (aligned rows).
Comparison.n_passed: The number of test units that passed.
Comparison.n_failed: The number of test units that failed.
Comparison.rows(): Get a sample of rows with a given status.
Comparison.column_summary(): Get a per-column summary of value mismatches.
Comparison.get_tabular_report(): Get a tabular report of the comparison.
Comparison.to_dict(): Get the comparison results as a JSON-serializable dict.
Comparison.to_json(): Get the comparison results as a JSON string.

Table Pre-checks

Helper functions for use with the active= parameter of validation methods. These inspect the target table before a step runs and conditionally skip the step when preconditions are not met.

has_columns(): Check whether one or more columns exist in a table.
has_rows(): Check whether a table has a certain number of rows.

YAML

Functions for using YAML to orchestrate validation workflows.

yaml_interrogate(): Execute a YAML-based validation workflow.
validate_yaml(): Validate YAML configuration against the expected structure.
yaml_to_python(): Convert YAML validation configuration to equivalent Python code.

Utility Functions

Functions for accessing metadata about the target data and managing configuration.

get_column_count(): Get the number of columns in a table.
get_row_count(): Get the number of rows in a table.
get_action_metadata(): Access step-level metadata when authoring custom actions.
get_validation_summary(): Access validation summary information when authoring final actions.
write_file(): Write a Validate object to disk as a serialized file.
read_file(): Read a Validate object from disk that was previously saved with write_file().
ref(): Reference a column from the reference data for aggregate comparisons.

Test Data Generation

Generate synthetic test data based on schema definitions. Use generate_dataset() to create data from a Schema object, or schema_from_tbl() to infer a generation-ready schema from an existing table (Polars, Pandas, or Ibis/DuckDB).

generate_dataset(): Generate synthetic test data from a schema.
schema_from_tbl(): Create a Schema from an existing table with inferred Field constraints.
int_field(): Create an integer column specification for use in a schema.
float_field(): Create a floating-point column specification for use in a schema.
string_field(): Create a string column specification for use in a schema.
bool_field(): Create a boolean column specification for use in a schema.
date_field(): Create a date column specification for use in a schema.
datetime_field(): Create a datetime column specification for use in a schema.
time_field(): Create a time column specification for use in a schema.
duration_field(): Create a duration column specification for use in a schema.
profile_fields(): Create a dict of string field specifications representing a person profile.

Prebuilt Actions

Prebuilt action functions for common notification patterns.

send_slack_notification(): Create an action that sends a Slack notification about a validation.
emit_otel(): Create a final action that sends a validation’s results to OpenTelemetry.

Metadata Import/Export

Import variable-level metadata from external data standards files (CDISC Define-XML, Controlled Terminology, SPSS .sav, SAS XPORT, Stata .dta, and more) and export metadata to various formats. Use import_metadata() as the entry point and export_metadata() for the reverse.

import_metadata(): Import metadata from a statistical-package file or a metadata standard.
export_metadata(): Export metadata as a Frictionless Table Schema.
load_metadata_example(): Get the path to a bundled metadata example file.
MetadataImport: The metadata for a single dataset.
MetadataPackage: The metadata for several datasets, read from one source.
VariableMetadata: The metadata for a single variable (column).
Codelist: A set of permitted values, each with a label.
CodelistEntry: A single permitted value in a codelist.
MissingValueCode: A declared missing-value code, as read from a metadata file.

SDTM Validation

Validate clinical datasets against CDISC SDTM domain templates. Use validate_sdtm() to generate a full Validate workflow, or validate_sdtm_structure() for a quick structural conformance check. Retrieve domain templates with get_sdtm_domain() and list_sdtm_domains().

validate_sdtm(): Build a validation plan for a dataset from an SDTM domain template.
validate_sdtm_structure(): Check a dataset’s variables against an SDTM domain template.
sdtm_to_metadata(): Convert an SDTM domain template to a MetadataImport object.
get_sdtm_domain(): Get the template for an SDTM domain.
list_sdtm_domains(): List the SDTM domains that have a template.
SDTMDomainTemplate: The structural template for an SDTM domain.
SDTMVariableSpec: The specification of one variable in an SDTM domain template.

ADaM Validation

Validate analysis datasets against CDISC ADaM templates. Use validate_adam() to generate a full Validate workflow, or validate_adam_structure() for a quick structural conformance check. Retrieve dataset templates with get_adam_dataset() and list_adam_datasets().

validate_adam(): Build a validation plan for a dataset from an ADaM dataset template.
validate_adam_structure(): Check a dataset’s variables against an ADaM dataset template.
adam_to_metadata(): Convert an ADaM dataset template to a MetadataImport object.
get_adam_dataset(): Get the template for a type of ADaM dataset.
list_adam_datasets(): List the ADaM dataset templates.
ADaMDatasetTemplate: The structural template for an ADaM dataset.
ADaMVariableSpec: The specification of one variable in an ADaM dataset template.

CDISC Submission Conformance

Validate SDTM datasets for CDISC conformance. validate_sdtmig() is the primary entry point: pass a dictionary of domain DataFrames and receive a ConformanceReport with 362 SDTMIG 3.4 rules evaluated in-process. To work with a whole submission (including cross-dataset referential integrity and SUPP– linkage, with validate_conformance(engine="validate")), use SubmissionPackage. For the authoritative CDISC-certified rule set, use validate_cdisc_submission() (requires the CORE CLI) or SubmissionPackage.validate_conformance(engine="core"). All paths return a ConformanceReport.

validate_sdtmig(): Check SDTM datasets with Pointblank’s bundled SDTMIG conformance rules.
validate_cdisc_submission(): Check a CDISC submission with the CDISC CORE engine.
SubmissionPackage: A study’s datasets, gathered for CDISC conformance validation.
ConformanceReport: The results of a CDISC conformance check.

Integrations

Classes for integrating Pointblank with external observability and monitoring systems. Use OTelExporter to export validation results as OpenTelemetry metrics, traces, and logs.

integrations.otel.OTelExporter: Export Pointblank validation results as OpenTelemetry signals.