Skip to main content

from_folder()

Build a SubmissionPackage from a folder of dataset files.

Usage

Source

from_folder(
    path,
    define=None,
    standard="sdtmig",
    standard_version="3.4",
    ct_version=None,
    study_id=None,
)

Submission datasets are usually delivered as files, one per dataset, in a single folder. The from_folder() method reads the folder so the datasets don’t have to be loaded one by one. It reads these files (the folder’s subfolders are not searched):

  • SAS Transport files (.xpt), which need the pyreadstat package. The dataset name is the file name without its extension, in uppercase, so dm.xpt becomes DM.
  • CDISC Dataset-JSON files (.json), in the Dataset-JSON 1.1 layout (top-level columns and rows) or the older layout (clinicalData or referenceData). The dataset name is taken from the file’s name (or its item group OID), or else from the file name. .json files that aren’t Dataset-JSON are skipped.
  • A Define-XML file: an .xml file whose name starts with define. It is used as the package’s define unless define= is given.

The datasets are read into Pandas DataFrames. The package remembers the folder, so that validate_conformance(engine="core") can give the files straight to the CORE engine instead of writing the datasets out again.

Parameters

path: str | Path

The path to the folder that holds the dataset files.

define: str | Path | Any | None = None

The study’s Define-XML, as a path to a define.xml file or a MetadataPackage. If None (the default), a Define-XML file in the folder is used, if there is one.

standard: str = "sdtmig"

The data standard that the datasets follow, "sdtmig" or "adamig". The default is "sdtmig".

standard_version: str = "3.4"

The version of the standard. The default is "3.4".

ct_version: str | None = None

The version of CDISC Controlled Terminology that the study uses, which is shown by summary(). The default is None.

study_id: str | None = None
A study identifier. The default is None.

Returns

SubmissionPackage
A package with a dataset for each dataset file in the folder.

Raises

NotADirectoryError

If path isn’t a folder.

ImportError
If the folder has an .xpt file and pyreadstat isn’t installed.

Examples

Let’s make a folder with a DM dataset as an XPT file and a Define-XML file for the study. Both files come with Pointblank:

import pointblank as pb
import shutil
import tempfile
from pathlib import Path

folder = Path(tempfile.mkdtemp())

shutil.copy(pb.load_metadata_example("dm.xpt"), folder / "dm.xpt")
shutil.copy(pb.load_metadata_example("define.xml"), folder / "define.xml")

study = pb.SubmissionPackage.from_folder(folder, study_id="XYZ789")

print(study)
Submission Package
  Study: XYZ789
  Standard: sdtmig 3.4
  Define-XML: present
  Datasets (1): DM

The package has the DM dataset from dm.xpt, and it found the Define-XML file. The dataset is a Pandas DataFrame:

study.get_dataset("DM").head(3)
STUDYID DOMAIN USUBJID SUBJID RFSTDTC RFENDTC SITEID AGE SEX RACE ARMCD ARM
0 XYZ789 DM XYZ789-101 101 2024-01-15 2024-07-15 S01 45.0 M WHITE TRT Active 10mg
1 XYZ789 DM XYZ789-102 102 2024-01-20 2024-07-20 S01 62.0 F BLACK PBO Placebo
2 XYZ789 DM XYZ789-103 103 2024-02-01 2024-08-01 S02 38.0 M ASIAN TRT Active 10mg

Since a Define-XML file was found, the built-in rules engine also runs the rules that check the datasets against it. Here, one of the subjects has a RACE of BLACK, which isn’t a term in the study’s codelist for RACE:

report = study.validate_conformance()

[i["message"] for i in report.issues() if i["rule_type"] == "DEFINE_CODELIST_CHECK"]
['DM.RACE contains a value not in the Define-XML declared codelist.']

The other issues in the report come from rules that don’t need Define-XML, such as the rules for variables that DM should have.