scorers.langid()

Score text by how likely it is to be in the expected languages.

Usage

scorers.langid(
    expected=None,
    backend="langid",
    model=None,
    column=None,
    expected_column=None,
    name=None,
    version=None,
    batch_size=None,
)

The score of a row is the probability, according to a language identification model, that its text is in one of the expected languages. It’s a check that a corpus or a column of free text is in the languages it should be, with min_val= as the confidence needed (e.g., 0.5).

Parameters

expected: str | list[str] | None = None: The expected language (an ISO 639 code like "en"), or a list of them. A code matches the model’s languages with the same code or with a region or script after it ("en" matches "en_US"). Give this or expected_column=.
backend: str | Callable[[str], Any] = "langid": The language identification model: "langid" (the default, which needs the langid package), "fasttext" (a fastText model, which needs the fasttext package and the path of the model as model=), or a function that takes a text and returns its probabilities by language (a dictionary or a list of (language, probability) pairs).
model: str | None = None: The path of the fastText model (e.g., lid.176.bin), with backend="fasttext".
column: str | None = None: The column with the text to score. It can be left out if the step has one column (or two, with one of them as expected_column=).
expected_column: str | None = None: A column with the expected language of each row, instead of expected= (for a multilingual corpus, where each document is labelled with its language). The column must be one of the columns of the step.
name: str | None = None: A name for the scorer, for the score cache and the report. By default, it’s made from the expected languages (or expected_column=), the backend, and the model.
version: str | None = None: A version, which is part of the key of the score cache.
batch_size: int | None = None: The number of texts scored at a time (256 by default).

Returns

Scorer: A scorer for Validate.col_vals_scored().

Examples

A real check would use the langid package or a fastText model. To keep this example self-contained, the backend is a toy function that guesses the language from a few words.

import pointblank as pb
import polars as pl

def guess(text):
    words = set(text.lower().split())
    if words & {"the", "and", "is"}:
        return {"en": 0.9, "fr": 0.1}
    if words & {"le", "et", "est"}:
        return {"en": 0.1, "fr": 0.9}
    return {"en": 0.5, "fr": 0.5}

tbl = pl.DataFrame(
    {"review": ["the soup is good", "le pain est bon", "and the bread", "ok"]}
)

pb.Validate(data=tbl).col_vals_scored(
    columns="review", scorer=pb.scorers.langid(expected="en", backend=guess), min_val=0.6
).interrogate()

		STEP	COLUMNS	VALUES	TBL	EVAL	UNITS	PASS	FAIL	W	E	C	EXT
#4CA64C66	1	col_vals_scored()	review	≥0.6 LANGID		✓	4	2 0.50	2 0.50	—	—	—
Notes Step 1 (scores) Scored by `langid(expected=['en'], backend=__main__.guess)`: 4 distinct rows scored. Scores from 0.1 to 0.9 (median 0.7).

The review in French fails, and so does "ok", which can’t be told apart. In a corpus with documents in several languages, the expected language of each row can come from a column, with expected_column=:

tbl = pl.DataFrame(
    {
        "review": ["the soup is good", "le pain est bon", "le vin est bon"],
        "lang": ["en", "fr", "en"],
    }
)

pb.Validate(data=tbl).col_vals_scored(
    columns=["review", "lang"],
    scorer=pb.scorers.langid(expected_column="lang", backend=guess),
    min_val=0.6,
).interrogate()

		STEP	COLUMNS	VALUES	TBL	EVAL	UNITS	PASS	FAIL	W	E	C	EXT
#4CA64C66	1	col_vals_scored()	review, lang	≥0.6 LANGID		✓	3	2 0.67	1 0.33	—	—	—
Notes Step 1 (scores) Scored by `langid(expected_column='lang', backend=__main__.guess)`: 3 distinct rows scored. Scores from 0.1 to 0.9 (median 0.9).

The third review is labelled as English, but it’s in French, so it fails.