MetadataImport

The metadata for a single dataset.

Usage

Source

MetadataImport(
    source_format,
    source_path=None,
    source_version=None,
    dataset_name=None,
    dataset_label=None,
    dataset_description=None,
    creation_date=None,
    study_id=None,
    domain=None,
    variables=list(),
    codelists=dict(),
    missing_value_codes=dict(),
)

A MetadataImport is what import_metadata() returns for a file that describes one dataset. It holds the dataset-level information (name, label, study and domain), a VariableMetadata for each variable, the codelists that the variables use, and any declared missing-value codes, all in the same form whatever the format of the file.

The methods turn the metadata into things Pointblank can use: to_validate() builds a validation plan from the declared types and constraints, to_schema() gives a Schema of the column types, and missing_specs() builds MissingSpec objects from the missing-value codes. Variables and codelists can be looked up with get_variable() and get_codelist(), and summary() describes the whole dataset in text.

A MetadataImport can also be created directly, which is a way to write down the rules for a dataset that doesn’t come with a metadata file and still use the methods above. Calling len() on it gives the number of variables.

Parameters

source_format: str

The format the metadata was read from, such as "frictionless", "cdisc_define" or "spss". Any string can be used for metadata created directly.

source_path: str | None = None

The path of the file that was read. The default is None.

source_version: str | None = None

The version of the format or standard, such as "Define-XML 2.0". The default is None.

dataset_name: str | None = None

The name of the dataset, such as "DM". The default is None.

dataset_label: str | None = None

A human-readable label for the dataset, such as "Demographics". The default is None.

dataset_description: str | None = None

A longer description of the dataset. The default is None.

creation_date: str | None = None

When the metadata was created, as given in the file. The default is None.

study_id: str | None = None

The study identifier, for clinical data. The default is None.

domain: str | None = None

The CDISC domain, such as "DM" or "AE". The default is None.

variables: list[VariableMetadata] = list()

The metadata of each variable, as a list of VariableMetadata objects. The default is an empty list.

codelists: dict[str, Codelist] = dict()

The codelists used by the variables, as a dictionary from codelist name to Codelist. The default is an empty dictionary.

missing_value_codes: dict[str, list[MissingValueCode]] = dict()

The declared missing-value codes, as a dictionary from column name to a list of MissingValueCode objects. The default is an empty dictionary.

Examples

Let’s import the bundled Frictionless Table Schema, which describes a table of sensor readings:

import pointblank as pb

meta = pb.import_metadata(pb.load_metadata_example("table_schema.json"))

print(len(meta))
print(meta.variable_names)
print(list(meta.codelists))

6
['sensor_id', 'reading_time', 'temperature', 'pressure_hpa', 'battery_pct', 'status']
['status_enum']

The schema describes six variables, and the allowed values of status were read into a codelist named "status_enum". Metadata can also be written down directly, here for a small survey table:

survey = pb.MetadataImport(
    source_format="custom",
    dataset_name="survey",
    dataset_label="Household survey",
    variables=[
        pb.VariableMetadata(name="respondent_id", dtype="String", required=True, unique=True),
        pb.VariableMetadata(name="age", dtype="Int64", required=True, min_val=18, max_val=99),
    ],
)

survey

MetadataImport(source_format='custom', variables=2, codelists=0)

Either way, the result can be turned into a validation plan with to_validate(), which has an example of checking a table against its metadata.

See Also

import_metadata()

Read a MetadataImport from a file.

MetadataPackage

The metadata for several datasets.

VariableMetadata

The metadata of one variable.

Attributes

Name	Description
variable_names	The names of all variables, in order.

variable_names

The names of all variables, in order.

variable_names: list[str]

Examples

import pointblank as pb

meta = pb.import_metadata(pb.load_metadata_example("weather_csvw.json"))

meta.variable_names

['station_id',
 'timestamp',
 'temperature_c',
 'humidity_pct',
 'wind_speed_kmh',
 'precipitation_mm',
 'condition']

These are the column names that a table described by this metadata should have. This is a property, so it’s used without parentheses.

Methods

Name	Description
to_validate()	Build a validation plan for a table from the metadata.
to_schema()	Get a Schema of the variables’ names and types.
get_variable()	Get the metadata of a variable by name.
get_codelist()	Get a codelist by name.
missing_specs()	Build a MissingSpec for each variable with missing values.
summary()	Get a text summary of the metadata.

to_validate()

Build a validation plan for a table from the metadata.

Usage

Source

to_validate(
    data,
    **kwargs,
)

This turns the declared types and constraints into validation steps, so that a table can be checked against its own documentation. The first step is a col_schema_match() with the schema from to_schema(). Then, for each variable in turn:

• col_vals_not_null() if it’s required
• rows_distinct() on the column if it’s unique
• col_vals_between() if it has both min_val and max_val, or col_vals_ge() or col_vals_le() if it has only one of them
• col_vals_in_set() if it has allowed_values
• col_vals_regex() if it has a pattern
• col_vals_not_in_set() with its missing_values, if it’s a string variable that declares some

The last step checks that the declared missing-value markers don’t appear in the data, as when a file was read without converting markers like "NA" to nulls. The min_length and max_length of a variable aren’t checked. None of the steps pass null values, so a null in an optional variable fails its range, set and pattern checks. To change any of the steps, get the plan as code with to_code() and edit it.

Parameters

data: Any

The table to validate.

**kwargs: Any

Arguments for Validate, such as tbl_name=, thresholds= or actions=. A label= replaces the default label, which is "Validation: <dataset name> (<format>)", or "Validation from <format> metadata" if the metadata has no dataset name.

Returns

Validate

A Validate object with the steps added. It hasn’t been interrogated yet.

Examples

The bundled Frictionless Data Package describes a table of sales transactions. Let’s check a few transactions against it:

import datetime
import polars as pl
import pointblank as pb

meta = pb.import_metadata(pb.load_metadata_example("datapackage.json"))

transactions = pl.DataFrame(
    {
        "transaction_id": ["TXN-00001", "TXN-00002", "TXN-00003", "TXN-00004"],
        "customer_id": ["CUST-101", "CUST-102", "CUST-101", "CUST-103"],
        "amount": [25.50, 1200.00, 0.00, 89.99],
        "quantity": [1, 3, 2, 1],
        "category": ["food", "electronics", "home", "toys"],
        "sale_date": [datetime.date(2024, 5, d) for d in (1, 1, 2, 3)],
        "discount_pct": [0.0, 10.0, 0.0, 5.0],
        "email": ["a@example.com", "b@example.com", "c@example.com", "not-an-email"],
    }
)

validation = meta.to_validate(data=transactions, tbl_name="transactions").interrogate()

validation

		STEP	COLUMNS	VALUES	TBL	EVAL	UNITS	PASS	FAIL	W	E	C	EXT
#4CA64C	1	col_schema_match()	—	SCHEMA		✓	1	1 1.00	0 0.00	—	—	—	—
#4CA64C	2	col_vals_not_null()	transaction_id	—		✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C	3	rows_distinct()	transaction_id	—		✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C	4	col_vals_not_in_set()	transaction_id	NA, N/A		✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C	5	col_vals_not_null()	customer_id	—		✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C	6	col_vals_not_in_set()	customer_id	NA, N/A		✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C	7	col_vals_not_null()	amount	—		✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C66	8	col_vals_between()	amount	[0.01, 99999.99]		✓	4	3 0.75	1 0.25	—	—	—
#4CA64C	9	col_vals_not_null()	quantity	—		✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C	10	col_vals_between()	quantity	[1.0, 1000.0]		✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C	11	col_vals_not_null()	category	—		✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C66	12	col_vals_in_set()	category	electronics, clothing, food, home, sports		✓	4	3 0.75	1 0.25	—	—	—
#4CA64C	13	col_vals_not_in_set()	category	NA, N/A		✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C	14	col_vals_not_null()	sale_date	—		✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C	15	col_vals_between()	discount_pct	[0.0, 50.0]		✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C66	16	col_vals_regex()	email	^[^@]+@[^@]+\.[^@]+$		✓	4	3 0.75	1 0.25	—	—	—
#4CA64C	17	col_vals_not_in_set()	email	NA, N/A		✓	4	4 1.00	0 0.00	—	—	—	—
Notes Step 1 (schema_check) ✓ Schema validation passed. Schema Comparison TABLE SCHEMA COLUMN DATA TYPE COLUMN DATA TYPE 1 transaction_id String 1 transaction_id ✓ String ✓ 2 customer_id String 2 customer_id ✓ String ✓ 3 amount Float64 3 amount ✓ Float64 ✓ 4 quantity Int64 4 quantity ✓ Int64 ✓ 5 category String 5 category ✓ String ✓ 6 sale_date Date 6 sale_date ✓ Date ✓ 7 discount_pct Float64 7 discount_pct ✓ Float64 ✓ 8 email String 8 email ✓ String ✓ Supplied Column Schema: [('transaction_id', 'String'), ('customer_id', 'String'), ('amount', 'Float64'), ('quantity', 'Int64'), ('category', 'String'), ('sale_date', 'Date'), ('discount_pct', 'Float64'), ('email', 'String')] Schema Match Settings COMPLETE IN ORDER COLUMN ≠ column DTYPE ≠ dtype float ≠ float64

The metadata gave 17 steps. The table matches the declared schema, and three of the constraint checks find a failing row: the amount of 0.00 is below the minimum of 0.01, "toys" isn’t an allowed category, and "not-an-email" doesn’t match the pattern for email. To see how the plan was built, or to start from it and change some steps, print it as code:

print(meta.to_validate(data=transactions).to_code())

import pointblank as pb

validation = (
    pb.Validate(
        data=your_data,  # Replace your_data with the actual data variable
        label="Validation: transactions (frictionless)",
    )
    .col_schema_match(schema=pb.Schema(columns=[("transaction_id", "String"), ("customer_id", "String"), ("amount", "Float64"), ("quantity", "Int64"), ("category", "String"), ("sale_date", "Date"), ("discount_pct", "Float64"), ("email", "String")]))
    .col_vals_not_null(columns="transaction_id")
    .rows_distinct(columns_subset="transaction_id")
    .col_vals_not_in_set(columns="transaction_id", set=["NA", "N/A"])
    .col_vals_not_null(columns="customer_id")
    .col_vals_not_in_set(columns="customer_id", set=["NA", "N/A"])
    .col_vals_not_null(columns="amount")
    .col_vals_between(columns="amount", left=0.01, right=99999.99)
    .col_vals_not_null(columns="quantity")
    .col_vals_between(columns="quantity", left=1.0, right=1000.0)
    .col_vals_not_null(columns="category")
    .col_vals_in_set(columns="category", set=["electronics", "clothing", "food", "home", "sports"])
    .col_vals_not_in_set(columns="category", set=["NA", "N/A"])
    .col_vals_not_null(columns="sale_date")
    .col_vals_between(columns="discount_pct", left=0.0, right=50.0)
    .col_vals_regex(columns="email", pattern="^[^@]+@[^@]+\\.[^@]+$")
    .col_vals_not_in_set(columns="email", set=["NA", "N/A"])
)

validation

Each step can be traced back to the metadata. For instance, the rows_distinct() step for transaction_id comes from its unique constraint, and the col_vals_not_in_set() steps check the "NA" and "N/A" markers declared for the string columns.

to_schema()

Get a Schema of the variables’ names and types.

Usage

Source

to_schema()

The schema has a column for each variable, in order, with the variable’s dtype (or "String" if the dtype isn’t known). It can be used with col_schema_match() to check that a table has the expected columns and types. Only the names and types are included: constraints such as ranges and allowed values aren’t part of a schema, and are checked by the steps that to_validate() adds after its schema check.

Returns

Schema

A Schema with one column per variable.

Examples

import pointblank as pb

meta = pb.import_metadata(pb.load_metadata_example("datapackage.json"))

meta.to_schema()

Schema(columns=[('transaction_id', 'String'), ('customer_id', 'String'), ('amount', 'Float64'), ('quantity', 'Int64'), ('category', 'String'), ('sale_date', 'Date'), ('discount_pct', 'Float64'), ('email', 'String')])

The schema lists the eight columns of the transactions table with their types. The dtype names are those of Polars, so a schema check against a table from another backend fails on the type names (pandas, for example, has object and int64 where Polars has String and Int64). Convert such a table to Polars before checking it.

get_variable()

Get the metadata of a variable by name.

Usage

Source

get_variable(name)

Parameters

name: str

The name of the variable. Names are matched exactly, including case.

Returns

VariableMetadata

The VariableMetadata of the variable.

Raises

KeyError

If there’s no variable with that name.

Examples

import pointblank as pb

meta = pb.import_metadata(pb.load_metadata_example("weather_csvw.json"))
humidity = meta.get_variable("humidity_pct")

print(humidity.label, humidity.dtype, humidity.min_val, humidity.max_val)

Relative Humidity (%) Float64 0.0 100.0

The CSVW file declares relative humidity as a number from 0 to 100. Use variable_names to see which names are available.

get_codelist()

Get a codelist by name.

Usage

Source

get_codelist(name)

The codelists are keyed by name in codelists, and a variable that uses one names it in its codelist_ref.

Parameters

name: str

The name of the codelist, such as "Sex".

Returns

Codelist

The Codelist with that name.

Raises

KeyError

If there’s no codelist with that name. The error message lists the available names.

Examples

import pointblank as pb

dm = pb.import_metadata(pb.load_metadata_example("define.xml"), dataset="DM")
race_ref = dm.get_variable("RACE").codelist_ref

dm.get_codelist(race_ref).to_set()

['WHITE',
 'BLACK OR AFRICAN AMERICAN',
 'ASIAN',
 'AMERICAN INDIAN OR ALASKA NATIVE',
 'NATIVE HAWAIIAN OR OTHER PACIFIC ISLANDER']

Starting from the variable’s codelist_ref finds the codelist that applies to RACE, and to_set() gives its permitted values.

missing_specs()

Build a MissingSpec for each variable with missing values.

Usage

Source

missing_specs()

Each variable that declares missing values (such as the user-defined missing values of an SPSS variable, or the missing-value markers of a Frictionless schema) gets a spec made by its to_missing_spec() method. The dictionary can be given to missing_vals_tbl() as its missing=, and a single spec from it can be given to the missing= parameter of a validation method such as col_vals_regex().

The specs keep the values as they were declared. Frictionless and CSVW markers are strings (like "NA") and are declared for every field, but in a table read with typed columns they can only appear in the string columns. Keep only the specs of those columns before using them, since a string marker can’t be compared with a numeric or date column.

Returns

dict[str, MissingSpec]

A dictionary from column name to MissingSpec. Variables that don’t declare missing values aren’t included.

Examples

The bundled Frictionless Data Package declares "NA" and "N/A" as missing-value markers, so every one of its variables gets a spec:

import pointblank as pb

meta = pb.import_metadata(pb.load_metadata_example("datapackage.json"))
specs = meta.missing_specs()

print(list(specs))
print(specs["customer_id"].reasons)

['transaction_id', 'customer_id', 'amount', 'quantity', 'category', 'sale_date', 'discount_pct', 'email']
{'NA': 'missing_na', 'N/A': 'missing_n_a'}

The reasons were made from the markers, as no labels were declared for them. Let’s keep the specs of the string columns and use one of them in a validation step:

import polars as pl

string_specs = {
    name: spec for name, spec in specs.items() if meta.get_variable(name).dtype == "String"
}

tbl = pl.DataFrame({"customer_id": ["CUST-101", "N/A", "CUST-103", "NA"]})

(
    pb.Validate(data=tbl)
    .col_vals_regex(columns="customer_id", pattern="^CUST-[0-9]+$")
    .col_vals_regex(
        columns="customer_id", pattern="^CUST-[0-9]+$", missing=string_specs["customer_id"]
    )
    .interrogate()
)

		STEP	COLUMNS	VALUES	TBL	EVAL	UNITS	PASS	FAIL	W	E	C	EXT
#4CA64C66	1	col_vals_regex()	customer_id	^CUST-[0-9]+$		✓	4	2 0.50	2 0.50	—	—	—
#4CA64C	2	col_vals_regex()	customer_id	^CUST-[0-9]+$ MISSING-AWARE		✓	4	4 1.00	0 0.00	—	—	—	—
Notes Step 2 (missing_spec) Missing codes: NA→missing_na, N/A→missing_n_a, null→unknown

Without the spec, the "N/A" and "NA" markers fail the pattern check in step 1. With it, step 2 treats them as missing data instead, so all four rows pass. The string_specs dictionary could be given to missing_vals_tbl() in the same way, to summarize the missing data in each string column of a table.

summary()

Get a text summary of the metadata.

Usage

Source

summary()

The summary gives the format, the source file, the dataset name, label and domain (when known), and the numbers of variables and codelists. It then lists each variable with its dtype, its label, and its constraints: required, unique, the range, the number of allowed values, and the codelist. Printing a MetadataImport with print() shows the same summary.

Returns

str

The summary, as a string with one line per item.

Examples

import pointblank as pb

survey = pb.MetadataImport(
    source_format="custom",
    dataset_name="survey",
    dataset_label="Household survey",
    variables=[
        pb.VariableMetadata(
            name="respondent_id",
            label="Respondent ID",
            dtype="String",
            required=True,
            unique=True,
        ),
        pb.VariableMetadata(
            name="age", label="Age in years", dtype="Int64", min_val=18, max_val=99
        ),
        pb.VariableMetadata(
            name="region", dtype="String", allowed_values=["north", "south", "east", "west"]
        ),
    ],
)

print(survey.summary())

Metadata Import (custom)
  Dataset: survey
  Label: Household survey
  Variables: 3
  Codelists: 0

  Variables:
    respondent_id (String) — Respondent ID [required, unique]
    age (Int64) — Age in years [range=[18, 99]]
    region (String) [4 allowed values]

The constraints in brackets are among those that to_validate() turns into validation steps, so the summary is a quick way to see what a table will be checked for (a pattern isn’t listed, though it’s checked too). For metadata read from a file, the summary also has a Source: line with the path of the file.