Skip to main content

import_metadata()

Import metadata from a statistical-package file or a metadata standard.

Usage

Source

import_metadata(
    source,
    format=None,
    **kwargs,
)

Many datasets come with a description of their variables: the variable labels, value labels and missing-value codes stored in an SPSS, Stata or SAS file, the field types and constraints in a Frictionless or CSVW schema, or the variable definitions and codelists in a CDISC Define-XML document. The import_metadata() function reads that description (not the data itself) into a MetadataImport object, with one VariableMetadata per variable and any codelists as Codelist objects.

The imported metadata is most useful as the starting point for validation. Its to_validate() method turns the declared types and constraints into a Validate plan, to_schema() gives a Schema of the column types, and missing_specs() turns declared missing-value codes into MissingSpec objects. Use export_metadata() to go the other way and write metadata out as a Frictionless Table Schema.

A source that describes several datasets (a Define-XML document for a whole study, a Data Package with several resources, or a Controlled Terminology file with several codelists) gives a MetadataPackage instead, which holds one MetadataImport per dataset.

Parameters

source: str | Path | Any

The path to the metadata file, as a string or a pathlib.Path. Other kinds of objects raise a TypeError.

format: str | None = None

The format of the file. If None (the default), the format is detected from the file: by extension for .sav, .zsav, .dta and .xpt files, and by content for .json files (Frictionless or CSVW) and .xml files (Define-XML or Controlled Terminology). Other extensions need an explicit format. See the Supported Formats section for the values.

**kwargs: Any
Options for the reader of the chosen format. The Define-XML reader takes dataset= (such as "DM") to return only that dataset as a MetadataImport. The Controlled Terminology reader takes codelist= to return only one codelist, and the Frictionless reader takes resource= (a name or an index) to pick one resource from a Data Package. The "cdisc_sdtm" and "cdisc_adam" formats need domain= and dataset=, respectively, and also take study_id=.

Returns

MetadataImport | MetadataPackage
A MetadataImport for a source that describes a single dataset, or a MetadataPackage for a source that describes several.

Raises

ValueError

If the format can’t be detected or isn’t supported, or if a required option such as domain= is missing.

FileNotFoundError

If the file doesn’t exist.

ImportError
If the format needs an optional package that isn’t installed.

Supported Formats

format= Reads Needs
"frictionless" (or "datapackage", "table_schema") Frictionless schema (.json)
"csvw" W3C CSV on the Web metadata (.json)
"cdisc_define" (or "define_xml") CDISC Define-XML 2.0 or 2.1 (.xml) lxml
"cdisc_ct" CDISC Controlled Terminology in ODM-XML (.xml) lxml
"spss" (or "sav") SPSS data file (.sav, .zsav) pyreadstat
"stata" (or "dta") Stata data file (.dta) pyreadstat
"xpt" (or "sas") SAS Transport file (.xpt) pyreadstat
"cdisc_sdtm" A built-in SDTM domain template
"cdisc_adam" A built-in ADaM dataset template

For SPSS, Stata and SAS files only the metadata is read, not the data. The last two formats don’t read a file at all: source= isn’t used (any string will do), and the template named by domain= or dataset= is returned, as with sdtm_to_metadata() and adam_to_metadata().

Examples

Pointblank bundles a few metadata files to experiment with, which can be found with load_metadata_example(). Let’s import a Frictionless Data Package that describes a table of sales transactions:

import pointblank as pb

meta = pb.import_metadata(pb.load_metadata_example("datapackage.json"))

meta
MetadataImport(source_format='frictionless', variables=8, codelists=1)

The format was detected from the file’s contents. The metadata describes eight variables and one codelist (the allowed values of category). Each variable’s metadata can be looked up by name:

amount = meta.get_variable("amount")

print(amount.dtype, amount.required, amount.min_val, amount.max_val)
Float64 True 0.01 99999.99

The amount variable must be a non-null float from 0.01 to 99999.99. Constraints like these can be checked against a table with to_validate(), which builds a validation plan from the metadata:

import datetime
import polars as pl

transactions = pl.DataFrame(
    {
        "transaction_id": ["TXN-00001", "TXN-00002", "TXN-00003", "TXN-00004"],
        "customer_id": ["CUST-101", "CUST-102", "CUST-101", "CUST-103"],
        "amount": [25.50, 1200.00, 0.00, 89.99],
        "quantity": [1, 3, 2, 1],
        "category": ["food", "electronics", "home", "toys"],
        "sale_date": [datetime.date(2024, 5, d) for d in (1, 1, 2, 3)],
        "discount_pct": [0.0, 10.0, 0.0, 5.0],
        "email": ["a@example.com", "b@example.com", "c@example.com", "not-an-email"],
    }
)

meta.to_validate(data=transactions).interrogate()
STEP COLUMNS VALUES TBL EVAL UNITS PASS FAIL W E C EXT
#4CA64C 1
col_schema_match
col_schema_match()
 — SCHEMA ✓ 1 1
1.00		 0
0.00		 — — — —
#4CA64C 2
col_vals_not_null
col_vals_not_null()
 transaction_id — ✓ 4 4
1.00		 0
0.00		 — — — —
#4CA64C 3
rows_distinct
rows_distinct()
 transaction_id — ✓ 4 4
1.00		 0
0.00		 — — — —
#4CA64C 4
col_vals_not_in_set
col_vals_not_in_set()
 transaction_id NA, N/A ✓ 4 4
1.00		 0
0.00		 — — — —
#4CA64C 5
col_vals_not_null
col_vals_not_null()
 customer_id — ✓ 4 4
1.00		 0
0.00		 — — — —
#4CA64C 6
col_vals_not_in_set
col_vals_not_in_set()
 customer_id NA, N/A ✓ 4 4
1.00		 0
0.00		 — — — —
#4CA64C 7
col_vals_not_null
col_vals_not_null()
 amount — ✓ 4 4
1.00		 0
0.00		 — — — —
#4CA64C66 8
col_vals_between
col_vals_between()
 amount [0.01, 99999.99] ✓ 4 3
0.75		 1
0.25		 — — —
#4CA64C 9
col_vals_not_null
col_vals_not_null()
 quantity — ✓ 4 4
1.00		 0
0.00		 — — — —
#4CA64C 10
col_vals_between
col_vals_between()
 quantity [1.0, 1000.0] ✓ 4 4
1.00		 0
0.00		 — — — —
#4CA64C 11
col_vals_not_null
col_vals_not_null()
 category — ✓ 4 4
1.00		 0
0.00		 — — — —
#4CA64C66 12
col_vals_in_set
col_vals_in_set()
 category electronics, clothing, food, home, sports ✓ 4 3
0.75		 1
0.25		 — — —
#4CA64C 13
col_vals_not_in_set
col_vals_not_in_set()
 category NA, N/A ✓ 4 4
1.00		 0
0.00		 — — — —
#4CA64C 14
col_vals_not_null
col_vals_not_null()
 sale_date — ✓ 4 4
1.00		 0
0.00		 — — — —
#4CA64C 15
col_vals_between
col_vals_between()
 discount_pct [0.0, 50.0] ✓ 4 4
1.00		 0
0.00		 — — — —
#4CA64C66 16
col_vals_regex
col_vals_regex()
 email ^[^@]+@[^@]+\.[^@]+$ ✓ 4 3
0.75		 1
0.25		 — — —
#4CA64C 17
col_vals_not_in_set
col_vals_not_in_set()
 email NA, N/A ✓ 4 4
1.00		 0
0.00		 — — — —
Notes

Step 1 (schema_check) ✓ Schema validation passed.

Schema Comparison
TABLE SCHEMA
COLUMN DATA TYPE COLUMN DATA TYPE
1 transaction_id String 1 transaction_id ✓ String ✓
2 customer_id String 2 customer_id ✓ String ✓
3 amount Float64 3 amount ✓ Float64 ✓
4 quantity Int64 4 quantity ✓ Int64 ✓
5 category String 5 category ✓ String ✓
6 sale_date Date 6 sale_date ✓ Date ✓
7 discount_pct Float64 7 discount_pct ✓ Float64 ✓
8 email String 8 email ✓ String ✓
Supplied Column Schema:
[('transaction_id', 'String'), ('customer_id', 'String'), ('amount', 'Float64'), ('quantity', 'Int64'), ('category', 'String'), ('sale_date', 'Date'), ('discount_pct', 'Float64'), ('email', 'String')]
Schema Match Settings
COMPLETE
IN ORDER
COLUMN ≠ column
DTYPE ≠ dtype
float ≠ float64

The table has the declared column types, so the schema check in step 1 passes. Three later steps find a failing row each: the amount of 0.00 is below the minimum, "toys" isn’t one of the allowed categories, and "not-an-email" doesn’t match the pattern given for email.

A Define-XML document usually describes all the datasets in a study, so importing one gives a MetadataPackage:

package = pb.import_metadata(pb.load_metadata_example("define.xml"))

package
MetadataPackage(name='STUDY.XYZ789', datasets=2)

The package holds the Demographics (DM) and Adverse Events (AE) datasets, and package["DM"] gives the MetadataImport for one of them. If only one dataset is needed, the dataset= option of the Define-XML reader returns it directly:

dm = pb.import_metadata(pb.load_metadata_example("define.xml"), dataset="DM")

print(dm.dataset_label)
print(dm.variable_names)
Demographics
['STUDYID', 'DOMAIN', 'USUBJID', 'SUBJID', 'RFSTDTC', 'RFENDTC', 'SITEID', 'AGE', 'AGEU', 'SEX', 'RACE', 'ARMCD', 'ARM']

Options like dataset= are passed through to the reader for the file’s format. The Supported Formats section lists the formats, and the **kwargs parameter describes the options that each reader accepts.

See Also

load_metadata_example()

Get the path to a bundled metadata file.

export_metadata()

Export metadata as a Frictionless Table Schema.

MetadataImport

The metadata for a single dataset.

MetadataPackage

The metadata for several datasets.