Validate.col_vals_text_quality()

Validate whether text passes the heuristic quality filters of web text.

Usage

Validate.col_vals_text_quality(
    columns,
    min_words=50,
    max_words=100000,
    min_mean_word_length=3,
    max_mean_word_length=10,
    max_symbol_ratio=0.1,
    max_bullet_lines=0.9,
    max_ellipsis_lines=0.3,
    min_alpha_words=0.8,
    min_stop_words=2,
    max_repeated_lines=0.3,
    stop_words=None,
    na_pass=True,
    pre=None,
    segments=None,
    thresholds=None,
    actions=None,
    brief=None,
    active=True,
    dimension=None,
)

Text collected from the web for training language models has a lot that isn’t prose: menus, lists of links, tables of numbers, placeholder text, and pages that repeat themselves. The quality filters of the Gopher models (Rae et al., 2021) catch much of it with a few simple statistics of each document, and the col_vals_text_quality() validation method adds a validation step for each of them, with the bounds used for Gopher as the defaults:

the number of words is between min_words= and max_words= (50 and 100,000)
the mean length of the words is between min_mean_word_length= and max_mean_word_length= (3 and 10 characters)
there are no more than max_symbol_ratio= hash symbols and ellipses for each word (0.1)
no more than max_bullet_lines= of the lines start with a bullet point (90%)
no more than max_ellipsis_lines= of the lines end with an ellipsis (30%)
at least min_alpha_words= of the words have a letter (80%)
the text has at least min_stop_words= distinct stop words (2 of the, be, to, of, and, that, have, and with)
no more than max_repeated_lines= of the lines repeat an earlier line (30%)

Each check is a col_vals_scored() step with a text_stat() scorer, so the report shows each one, with its own counts of the documents that fail, and a brief that says what it checks. Each row is a test unit. Set a bound to None to leave its check out (a check with two bounds is left out when both are None). Documents without the words or lines that a check needs pass it (the check of the number of words catches them).

The defaults are for English web text. Text in other languages needs other stop words (and perhaps other bounds), and short texts (like titles or chat messages) need a lower min_words=.

Parameters

columns: str | list[str]: A single column or a list of columns to validate. There will be a separate set of validation steps for each column.
min_words: int | None = 50: The fewest and the most words a document can have.
max_words: int | None = 50: The fewest and the most words a document can have.
min_mean_word_length: float | None = 3: The bounds of the mean length of the words of a document, in characters.
max_mean_word_length: float | None = 3: The bounds of the mean length of the words of a document, in characters.
max_symbol_ratio: float | None = 0.1: The most hash symbols and ellipses that a document can have for each word.
max_bullet_lines: float | None = 0.9: The largest share of the lines of a document that can start with a bullet point.
max_ellipsis_lines: float | None = 0.3: The largest share of the lines of a document that can end with an ellipsis.
min_alpha_words: float | None = 0.8: The smallest share of the words of a document that must have a letter.
min_stop_words: int | None = 2: The fewest distinct stop words that a document must have.
max_repeated_lines: float | None = 0.3: The largest share of the lines of a document that can be the same as an earlier line.
stop_words: Sequence[str] | None = None: The stop words, for min_stop_words=. By default, they’re the eight English words of the Gopher filters.
na_pass: bool = True: Should a missing value pass? The default is True, so that missing documents are left to a check like col_vals_not_null().
pre: Callable | None = None: An optional preprocessing function or lambda to apply to the data table during interrogation. This function should take a table as input and return a modified table.
segments: SegmentSpec | None = None: An optional directive on segmentation, which serves to split a validation step into multiple (one step per segment). Can be a single column name, a tuple that specifies a column name and its corresponding values to segment on, or a combination of both (provided as a list).
thresholds: int | float | bool | tuple | dict | Thresholds | None = None: Set threshold failure levels for reporting and reacting to exceedences of the levels. The thresholds are set at the step level and will override any global thresholds set in Validate(thresholds=...). The default is None, which means that no thresholds will be set locally and global thresholds (if any) will take effect.
actions: Actions | None = None: Optional actions to take when the validation step meets or exceeds any set threshold levels. If provided, the Actions class should be used to define the actions.
brief: str | bool | None = None: The brief of each validation step, displayed in the reporting table. By default (None or True), each step’s brief says what it checks (e.g., “Between 50 and 100,000 words in text.”). A string is used for every step, with "{auto}" replaced by that description (and "{step}" by the step number). Use False for no brief.
active: bool | Callable = True: A boolean value or callable that determines whether the validation steps should be active. Using False will make the validation steps inactive (still reporting their presence and keeping indexes for the steps unchanged). A callable can also be provided; it will receive the data table as its single argument and must return a boolean value. The callable is evaluated before any pre= processing.
dimension: str | None = None: An optional data quality dimension to categorize the validation steps for health scoring. One of "completeness", "validity", "uniqueness", "consistency", "timeliness", or "volume" (or any custom string). If None (the default), the dimension is "validity". This label appears in the validation report and feeds the overall and per-dimension health scores.

Returns

Validate: The Validate object with the added validation steps.

Raises

TypeError: If a bound isn’t a number or None.
ValueError: If min_words= is greater than max_words=, or min_mean_word_length= is greater than max_mean_word_length=.

Examples

Let’s check a small corpus of web documents. The documents are short, so we’ll use a lower min_words= than the default.

import pointblank as pb
import polars as pl

corpus = pl.DataFrame(
    {
        "text": [
            "The museum is open from nine to five, and the entry is free for children "
            "under the age of twelve, so it's a good place to spend the day with family.",
            "Home\nAbout\nHome\nAbout\nHome\nContact us",
            "#sale #deals #shoes #free #today ... click to see more of the best deals",
            "Our library lends books, films, and music to members of the town for up to "
            "three weeks, and the books can be returned to any of its branches.",
        ]
    }
)

(
    pb.Validate(data=corpus)
    .col_vals_text_quality(columns="text", min_words=10)
    .interrogate()
)

		STEP	COLUMNS	VALUES	EVAL	UNITS	PASS	FAIL	W	E	C	EXT
#4CA64C66	1	col_vals_scored() Between 10 and 100,000 words in `text`.	text	10–100000 WORD COUNT	✓	4	3 0.75	1 0.25	—	—	—
#4CA64C	2	col_vals_scored() A mean word length of between 3 and 10 characters in `text`.	text	3–10 MEAN WORD LENGTH	✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C66	3	col_vals_scored() At most 0.1 hash symbols and ellipses for each word in `text`.	text	≤0.1 SYMBOL RATIO	✓	4	3 0.75	1 0.25	—	—	—
#4CA64C	4	col_vals_scored() At most 90% of the lines start with a bullet point in `text`.	text	≤0.9 BULLET LINES	✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C	5	col_vals_scored() At most 30% of the lines end with an ellipsis in `text`.	text	≤0.3 ELLIPSIS LINES	✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C	6	col_vals_scored() At least 80% of the words have a letter in `text`.	text	≥0.8 ALPHA WORDS	✓	4	4 1.00	0 0.00	—	—	—	—
#4CA64C66	7	col_vals_scored() At least 2 distinct stop words in `text`.	text	≥2 STOP WORDS	✓	4	3 0.75	1 0.25	—	—	—
#4CA64C66	8	col_vals_scored() At most 30% of the lines repeat an earlier line in `text`.	text	≤0.3 REPEATED LINES	✓	4	3 0.75	1 0.25	—	—	—
Notes Step 1 (scores) Scored by `text_stat(stat='word_count')`: 4 distinct rows scored. Scores from 7 to 31 (median 21). Step 2 (scores) Scored by `text_stat(stat='mean_word_length')`: 4 distinct rows scored. Scores from 3.806 to 4.429 (median 4.143). Step 3 (scores) Scored by `text_stat(stat='symbol_ratio')`: 4 distinct rows scored. Scores from 0 to 0.4286 (median 0). Step 4 (scores) Scored by `text_stat(stat='bullet_lines')`: 4 distinct rows scored. Scores from 0 to 0 (median 0). Step 5 (scores) Scored by `text_stat(stat='ellipsis_lines')`: 4 distinct rows scored. Scores from 0 to 0 (median 0). Step 6 (scores) Scored by `text_stat(stat='alpha_words')`: 4 distinct rows scored. Scores from 0.9286 to 1 (median 1). Step 7 (scores) Scored by `text_stat(stat='stop_words')`: 4 distinct rows scored. Scores from 0 to 5 (median 4). Step 8 (scores) Scored by `text_stat(stat='repeated_lines')`: 4 distinct rows scored. Scores from 0 to 0.5 (median 0).

The menu in the second row has too few words and too many repeated lines, and few stop words. The third row, a list of hashtags, has too many symbols for its words. The two documents of prose pass every check.