text.in_language()

Make a validation step that checks that text is in the expected language.

Usage

text.in_language(
    column,
    expected=None,
    expected_column=None,
    min_confidence=0.5,
    backend="langid",
    model=None,
    na_pass=False,
    cache=True,
    pre=None,
    segments=None,
    thresholds=None,
    actions=None,
    brief=None,
    active=True,
)

The step is a col_vals_scored() step with a langid() scorer: a language identification model gives the probability that the text of each row is in one of the expected languages, and a row fails if the probability is below min_confidence=. The expected languages are the same for every row (expected=) or come from a column (expected_column=), for a corpus with documents in several languages.

Parameters

column: str: The column with the text.
expected: str | list[str] | None = None: The expected language (an ISO 639 code like "en"), or a list of them.
expected_column: str | None = None: A column with the expected language of each row, instead of expected=.
min_confidence: float = 0.5: The smallest probability that a text is in the expected languages for it to pass.
backend: str | Callable[[str], Any] = "langid": The language identification model: "langid", "fasttext" (with the path of the model as model=), or a function that gives the probabilities of a text by language (see langid()).
model: str | None = None: The path of the fastText model, with backend="fasttext".
na_pass: bool = False: Whether a row with a missing value passes.
cache: bool | str = True: Where the scores are kept: in memory (True), in a directory, or not at all (False).
pre: Callable | None = None: A function that transforms the table before the step, as in the validation methods.
segments: SegmentSpec | None = None: The segments of the step, as in the validation methods.
thresholds: int | float | bool | tuple | dict | Thresholds | None = None: The thresholds of the step (by default, those of the validation).
actions: Actions | None = None: The actions of the step.
brief: str | bool | None = None: A brief for the step.
active: bool | Callable = True: Whether the step is active.

Returns

Steps: The step, to be added to a validation with Validate.add_steps().

Examples

A real check would use the langid package or a fastText model. To keep this example self-contained, the backend is a toy function that guesses the language from a few words.

import pointblank as pb
import polars as pl

def guess(text):
    words = set(text.lower().split())
    if words & {"the", "and", "is"}:
        return {"en": 0.9, "fr": 0.1}
    if words & {"le", "et", "est"}:
        return {"en": 0.1, "fr": 0.9}
    return {"en": 0.5, "fr": 0.5}

docs = pl.DataFrame(
    {
        "text": ["the soup is good", "le pain est bon", "the bread is fresh"],
        "lang": ["en", "fr", "fr"],
    }
)

(
    pb.Validate(data=docs)
    .add_steps(pb.text.in_language("text", expected="en", backend=guess))
    .add_steps(pb.text.in_language("text", expected_column="lang", backend=guess))
    .interrogate()
)

		STEP	COLUMNS	VALUES	EVAL	UNITS	PASS	FAIL	W	E	C
#4CA64C66	1	col_vals_scored()	text	≥0.5 LANGID	✓	3	2 0.67	1 0.33	—	—	—
#4CA64C66	2	col_vals_scored()	text, lang	≥0.5 LANGID	✓	3	2 0.67	1 0.33	—	—	—
Notes Step 1 (scores) Scored by `langid(expected=['en'], backend=__main__.guess)`: 3 distinct rows scored. Scores from 0.1 to 0.9 (median 0.9). Step 2 (scores) Scored by `langid(expected_column='lang', backend=__main__.guess, column='text')`: 3 distinct rows scored. Scores from 0.1 to 0.9 (median 0.9).

The document in French fails step 1, which expects English for every row. In step 2, each row is checked against its own label, and the last document fails: it’s labelled as French, but it’s in English.