import pointblank as pb
import polars as pl
def guess(text):
words = set(text.lower().split())
if words & {"the", "and", "is"}:
return {"en": 0.9, "fr": 0.1}
if words & {"le", "et", "est"}:
return {"en": 0.1, "fr": 0.9}
return {"en": 0.5, "fr": 0.5}
docs = pl.DataFrame(
{
"text": ["the soup is good", "le pain est bon", "the bread is fresh"],
"lang": ["en", "fr", "fr"],
}
)
(
pb.Validate(data=docs)
.col_vals_in_language(columns="text", expected="en", backend=guess)
.col_vals_in_language(columns="text", expected_column="lang", backend=guess)
.interrogate()
)Validate.col_vals_in_language()
Validate whether text is in the expected language.
Usage
Validate.col_vals_in_language(
columns,
expected=None,
expected_column=None,
min_confidence=0.5,
backend="langid",
model=None,
na_pass=False,
cache=True,
pre=None,
segments=None,
thresholds=None,
actions=None,
brief=None,
active=True,
dimension=None,
)The col_vals_in_language() validation method checks that the text of each row is in one of the expected languages. A language identification model gives the probability that the text is in each language, and a row fails if the probability of the expected ones is below min_confidence=. Each row is a test unit. The expected languages are the same for every row (expected=) or come from a column (expected_column=), for a corpus with documents in several languages.
The step is a col_vals_scored() step with a langid() scorer (it’s listed in the report as col_vals_scored(), with a brief that says what it checks), so the probabilities are in the step report and in the extracts of failing rows.
Parameters
columns: str | list[str]-
A single column or a list of columns to validate. If multiple columns are supplied, there will be a separate validation step generated for each column.
expected: str | list[str] | None = None-
The expected language (an ISO 639 code like
"en"), or a list of them. expected_column: str | None = None-
A column with the expected language of each row, instead of
expected=. min_confidence: float = 0.5-
The smallest probability that a text is in the expected languages for it to pass, from 0 to 1. The default is
0.5. backend: str | Callable[[str], Any] = "langid"-
The language identification model:
"langid"(the default, which needs the langid package),"fasttext"(with the path of the model asmodel=), or a function that gives the probabilities of a text by language (seelangid()). model: str | None = None-
The path of the fastText model, with
backend="fasttext". na_pass: bool = False-
Should a missing value pass? The default is
False. cache: bool | str | Path = True-
Where the probabilities are kept, so that a validation that’s run again doesn’t compute them again: in memory (
True, the default), in a directory, or not at all (False). pre: Callable | None = None-
An optional preprocessing function or lambda to apply to the data table during interrogation. This function should take a table as input and return a modified table.
segments: SegmentSpec | None = None-
An optional directive on segmentation, which serves to split a validation step into multiple (one step per segment). Can be a single column name, a tuple that specifies a column name and its corresponding values to segment on, or a combination of both (provided as a list).
thresholds: int | float | bool | tuple | dict | Thresholds | None = None-
Set threshold failure levels for reporting and reacting to exceedences of the levels. The thresholds are set at the step level and will override any global thresholds set in
Validate(thresholds=...). The default isNone, which means that no thresholds will be set locally and global thresholds (if any) will take effect. actions: Actions | None = None-
Optional actions to take when the validation step meets or exceeds any set threshold levels. If provided, the
Actionsclass should be used to define the actions. brief: str | bool | None = None-
The brief of the validation step, displayed in the reporting table. By default (
NoneorTrue), the brief says what the step checks (e.g., “The text oftextis inen(probability of at least 0.5).”). A string is used instead, with"{auto}"replaced by that description (and"{step}"by the step number). UseFalsefor no brief. active: bool | Callable = True-
A boolean value or callable that determines whether the validation step should be active. Using
Falsewill make the validation step inactive (still reporting its presence and keeping indexes for the steps unchanged). A callable can also be provided; it will receive the data table as its single argument and must return a boolean value. The callable is evaluated before anypre=processing. dimension: str | None = None-
An optional data quality dimension to categorize this validation step for health scoring. One of
"completeness","validity","uniqueness","consistency","timeliness", or"volume"(or any custom string). IfNone(the default), the dimension is"validity". This label appears in the validation report and feeds the overall and per-dimension health scores.
Returns
Validate- The Validate object with the added validation step.
Raises
TypeError-
If
min_confidence=isn’t a number. ValueError-
If
min_confidence=isn’t between 0 and 1, if neither (or both) ofexpected=andexpected_column=is given, or ifexpected_column=is one ofcolumns=.
Examples
A real check would use the langid package or a fastText model. To keep this example self-contained, the backend is a toy function that guesses the language from a few words.
The document in French fails step 1, which expects English for every row. In step 2, each row is checked against its own label, and the last document fails: it’s labelled as French, but it’s in English.