get_sundered_data().
## Preprocessing
The `pre=` argument allows for a preprocessing function or lambda to be applied to the data table during interrogation. This function should take a table as input and return a modified table. This is useful for performing any necessary transformations or filtering on the data before the validation step is applied.
The preprocessing function can be any callable that takes a table as input and returns a modified table. For example, you could use a lambda function to filter the table based on certain criteria or to apply a transformation to the data. Note that the same preprocessing is **not** applied to the comparison table; only the target table is preprocessed. Regarding the lifetime of the transformed table, it only exists during the validation step and is not stored in the [Validate](Validate.md#pointblank.Validate) object or used in subsequent validation steps.
## Thresholds
The `thresholds=` parameter is used to set the failure-condition levels for the validation step. If they are set here at the step level, these thresholds will override any thresholds set at the global level in `Validate(thresholds=...)`.
There are three threshold levels: 'warning', 'error', and 'critical'. The threshold values can either be set as a proportion failing of all test units (a value between `0` to `1`), or, the absolute number of failing test units (as integer that's `1` or greater).
Thresholds can be defined using one of these input schemes:
1. use the Thresholds class (the most direct way to create thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and 'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units for the 'warning' level only
If the number of failing test units exceeds set thresholds, the validation step will be marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need to be set, you're free to set any combination of them.
Aside from reporting failure conditions, thresholds can be used to determine the actions to take for each level of failure (using the `actions=` parameter).
## Examples
Let's reconcile a small table of loaded orders against the source system's orders. The loaded table names the amount column differently, has a different number of rows (one order was split into two rows), and is missing the orders for one region.
``` python
import pointblank as pb
import polars as pl
source_orders = pl.DataFrame(
{
"region": ["north", "north", "south", "east", "west"],
"amount": [100.0, 250.0, 75.5, 30.0, 12.0],
}
)
loaded_orders = pl.DataFrame(
{
"region": ["north", "north", "north", "south", "east"],
"amt": [100.0, 200.0, 50.0, 75.5, 31.0],
}
)
validation = (
pb.Validate(data=loaded_orders)
.tbl_reconcile(
tbl_compare=source_orders,
keys="region",
measures="amount",
column_map={"amount": "amt"},
)
.interrogate()
)
validation
```
| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C66 | 1 | 
tbl_reconcile()
|
 |
✓ | 4 | 2 0.50 |
2 0.50 |
-- | -- | -- | CSV |
Report for Validation Step 1 ✗
ASSERTION tbl_reconcile(keys='region', measures='amount', agg='sum', column_map={'amount': 'amt'})
tbl_compare (4 groups) → target (3 groups), aligned by key regionDIFFERENT 2 of 4 groups fail (50.0%) match 2changed 1missing 1extra 0 sum by region
|
|
|---|---|
| Column mismatches (1 of 1 columns) | |
| amount_sum | |
| Changed groups (1 group) | |
| east | amount_sum: 30 → 31 (+1) |
| Missing from target (1 group) | |
| west | amount_sum: 12 |