validate_sdtm()

Build a validation plan for a dataset from an SDTM domain template.

Usage

validate_sdtm(
    data,
    domain,
    study_id=None,
    check_dates=True,
    check_lengths=True,
    label=None,
    **kwargs,
)

The validate_sdtm() function reads the template for an SDTM domain (see get_sdtm_domain()) and turns it into a Validate plan for a dataset of that domain. The plan isn’t interrogated, so more steps can be added to it before calling interrogate().

The steps are generated in this order, for the template variables that are columns of the data:

col_vals_not_null() for each required variable (one with a core designation of "Req").
col_vals_in_set() to check that DOMAIN holds only the domain code.
col_vals_gt() to check that the sequence number (as in AESEQ) is greater than 0. DM has one record per subject and so has no sequence number.
If check_lengths=True, a col_vals_expr() step for each character variable, checking that its values are no longer than the template’s maximum length. Null values pass.
If check_dates=True, a col_vals_regex() step for each --DTC timing variable (such as RFSTDTC or AESTDTC), checking that its values are ISO 8601 dates or date-times. Null values pass.

Columns that aren’t in the template are left alone, and the data types of the columns aren’t checked. A template variable that isn’t a column of the data gets no steps, so a dataset that lacks a required variable isn’t reported for it. Use validate_sdtm_structure() to find missing variables, or add a col_exists() step to the plan (as in the second example below).

The date check accepts the right-truncated forms that SDTM uses for partial dates: a year (2024), a year and month (2024-03), a full date (2024-03-15), and a date with a time to the hour, minute or second (2024-03-15T10:30:00). Dates in other formats (such as 03/15/2024), fractional seconds, time zones and dates with missing middle components (such as 2024---15) fail. The check is on the format only, so an impossible date such as 2024-13-45 passes.

Parameters

data: Any: The dataset to validate, as a Polars or Pandas DataFrame.
domain: str: The SDTM domain code, such as "DM", "AE" or "LB". This is case-insensitive. See list_sdtm_domains() for the domains that have a template.
study_id: str | None = None: A study identifier to use in the default label, which becomes "SDTM {domain} — {study_id}". It is ignored if label= is given. The default is None.
check_dates: bool = True: Whether to add the ISO 8601 format checks for the --DTC timing variables. The default is True.
check_lengths: bool = True: Whether to add the maximum length checks for the character variables. The default is True.
label: str | None = None: A label for the validation. The default is "SDTM {domain} Validation".
**kwargs: Any: Other arguments for Validate, such as tbl_name= and thresholds=.

Returns

Validate: The validation plan, not yet interrogated.

Raises

KeyError: If there is no template for the domain.

Examples

Here is a small Demographics (DM) dataset with a few problems: the third row has the wrong DOMAIN and no USUBJID, the second row’s RFSTDTC isn’t an ISO 8601 date, and the fourth row’s SEX is longer than the two characters that SDTM allows:

import pointblank as pb
import polars as pl

dm = pl.DataFrame(
    {
        "STUDYID": ["STUDY01"] * 4,
        "DOMAIN": ["DM", "DM", "AE", "DM"],
        "USUBJID": ["STUDY01-001", "STUDY01-002", None, "STUDY01-004"],
        "SUBJID": ["001", "002", "003", "004"],
        "RFSTDTC": ["2024-01-15", "03/15/2024", "2024-02", "2024-02-10T09:30"],
        "SEX": ["M", "F", "F", "MALE"],
    }
)

validation = pb.validate_sdtm(dm, domain="DM").interrogate()

validation

		STEP	COLUMNS	VALUES	EVAL	UNITS	PASS	FAIL	W	E	C	EXT
#4CA64C	1	col_vals_not_null()	STUDYID	—	✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C	2	col_vals_not_null()	DOMAIN	—	✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C66	3	col_vals_not_null()	USUBJID	—	✓	4	3 0.75	1 0.25	—	—	—
#4CA64C	4	col_vals_not_null()	SUBJID	—	✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C	5	col_vals_not_null()	SEX	—	✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C66	6	col_vals_in_set()	DOMAIN	DM	✓	4	3 0.75	1 0.25	—	—	—
#4CA64C	7	col_vals_expr() STUDYID length <= 20	STUDYID	COLUMN EXPR	✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C	8	col_vals_expr() DOMAIN length <= 2	DOMAIN	COLUMN EXPR	✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C	9	col_vals_expr() USUBJID length <= 40	USUBJID	COLUMN EXPR	✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C	10	col_vals_expr() SUBJID length <= 20	SUBJID	COLUMN EXPR	✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C	11	col_vals_expr() RFSTDTC length <= 64	RFSTDTC	COLUMN EXPR	✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C66	12	col_vals_expr() SEX length <= 2	SEX	COLUMN EXPR	✓	4	3 0.75	1 0.25	—	—	—
#4CA64C66	13	col_vals_regex()	RFSTDTC	^(\d{4})(-\d{2}(-\d{2}(T\d{2}(:\d{2}(:\d{2})?)?)?)?)?$	✓	4	3 0.75	1 0.25	—	—	—

The plan starts with the null checks for the five required variables in the data, then checks DOMAIN, the lengths of all six columns, and the format of RFSTDTC. Four steps have a failing test unit, one for each problem. The partial date 2024-02 and the date-time 2024-02-10T09:30 both pass the date check, since SDTM allows dates to be truncated.

The plan doesn’t report the required DM variables that the dataset lacks, such as SITEID and COUNTRY. Since the plan isn’t interrogated yet, a col_exists() step for the template’s required variables can be added to it. This example uses an Adverse Events (AE) dataset and leaves out the length checks to keep the plan short:

ae = pl.DataFrame(
    {
        "STUDYID": ["STUDY01"] * 3,
        "DOMAIN": ["AE"] * 3,
        "USUBJID": ["STUDY01-001", "STUDY01-001", "STUDY01-002"],
        "AESEQ": [1, 2, 0],
        "AETERM": ["HEADACHE", "NAUSEA", "RASH"],
        "AESTDTC": ["2024-01-20", "2024-02-03", "2024-02"],
    }
)

required = pb.get_sdtm_domain("AE").required_variables

validation = (
    pb.validate_sdtm(ae, domain="AE", check_lengths=False, tbl_name="ae")
    .col_exists(columns=required)
    .interrogate()
)

validation

		STEP	COLUMNS	VALUES	EVAL	UNITS	PASS	FAIL	W	E	C	EXT
#4CA64C	1	col_vals_not_null()	STUDYID	—	✓	3	3 1.00	0 0.00	—	—	—	—
#4CA64C	2	col_vals_not_null()	DOMAIN	—	✓	3	3 1.00	0 0.00	—	—	—	—
#4CA64C	3	col_vals_not_null()	USUBJID	—	✓	3	3 1.00	0 0.00	—	—	—	—
#4CA64C	4	col_vals_not_null()	AESEQ	—	✓	3	3 1.00	0 0.00	—	—	—	—
#4CA64C	5	col_vals_not_null()	AETERM	—	✓	3	3 1.00	0 0.00	—	—	—	—
#4CA64C	6	col_vals_in_set()	DOMAIN	AE	✓	3	3 1.00	0 0.00	—	—	—	—
#4CA64C66	7	col_vals_gt()	AESEQ	0	✓	3	2 0.67	1 0.33	—	—	—
#4CA64C	8	col_vals_regex()	AESTDTC	^(\d{4})(-\d{2}(-\d{2}(T\d{2}(:\d{2}(:\d{2})?)?)?)?)?$	✓	3	3 1.00	0 0.00	—	—	—	—
#4CA64C	9	col_exists()	STUDYID	—	✓	1	1 1.00	0 0.00	—	—	—	—
#4CA64C	10	col_exists()	DOMAIN	—	✓	1	1 1.00	0 0.00	—	—	—	—
#4CA64C	11	col_exists()	USUBJID	—	✓	1	1 1.00	0 0.00	—	—	—	—
#4CA64C	12	col_exists()	AESEQ	—	✓	1	1 1.00	0 0.00	—	—	—	—
#4CA64C	13	col_exists()	AETERM	—	✓	1	1 1.00	0 0.00	—	—	—	—
#4CA64C66	14	col_exists()	AEDECOD	—	✓	1	0 0.00	1 1.00	—	—	—	—

The col_vals_gt() step on AESEQ fails for the third row, whose sequence number is 0. The col_exists() steps at the end show that AEDECOD, a required AE variable, is missing from the dataset. The tbl_name= argument was passed on to Validate.

Parameters

Returns

Raises

Examples

See Also