import pointblank as pb
import polars as pl
tbl = pl.DataFrame(
{
"doc_id": [1, 2, 3, 4, 5, 6],
"text": [
"The library opens at nine in the morning and closes at six in the evening.",
"The trail to the summit is steep, so bring water and good walking shoes.",
"The library opens at nine in the morning and closes at six in the evening!",
"Tickets for the museum are free for children under the age of twelve.",
"the LIBRARY opens at nine in the morning, and closes at six in the evening",
"The trail to the summit is steep, so bring water and sturdy walking shoes.",
],
}
)
pb.preview(tbl)Validate.rows_near_distinct()
Validate whether the text of rows has no near duplicates.
Usage
Validate.rows_near_distinct(
columns,
method="minhash",
threshold=0.85,
ngram=5,
num_perm=128,
pre=None,
segments=None,
thresholds=None,
actions=None,
brief=None,
active=True,
dimension=None,
)The rows_near_distinct() validation method finds rows whose text is the same as, or nearly the same as, another row’s: the same document crawled twice, a template filled in with different names, or a passage with a changed footer. Such near duplicates make up a large share of many text corpora, and a model trained on them gives too much weight to the repeated text. Each row is a test unit, and every row in a group of near duplicates fails. Unlike rows_distinct(), which compares values exactly, the text is compared once normalized (in lower case, without punctuation or extra whitespace), and with method="minhash" (the default), texts that share most of their words in the same order are near duplicates too.
Parameters
columns: str | list[str]-
The column (or a list of columns) with the text of each row. The values of several columns are joined by newlines.
method: Literal["minhash", "exact"] = "minhash"-
How texts are compared:
"minhash"(the default) finds texts whose word n-grams are similar (see the How Texts Are Compared section) and"exact"finds texts that are the same once normalized. threshold: float = 0.85-
With
method="minhash", the lowest Jaccard similarity of the word n-grams of two texts at which they are near duplicates, between0and1. The default,0.85, finds texts that differ by a few words. ngram: int = 5-
With
method="minhash", the number of words in each n-gram (the default is5). A text shorter than this is one n-gram of all its words. num_perm: int = 128-
With
method="minhash", the number of hash functions of the MinHash signatures (the default is128). More find near duplicates more reliably, but take longer. pre: Callable | None = None-
An optional preprocessing function or lambda to apply to the data table during interrogation. This function should take a table as input and return a modified table.
segments: SegmentSpec | None = None-
An optional directive on segmentation, which serves to split a validation step into multiple (one step per segment). Can be a single column name, a tuple that specifies a column name and its corresponding values to segment on, or a combination of both (provided as a list).
thresholds: int | float | bool | tuple | dict | Thresholds | None = None-
Set threshold failure levels for reporting and reacting to exceedences of the levels. The thresholds are set at the step level and will override any global thresholds set in
Validate(thresholds=...). The default isNone, which means that no thresholds will be set locally and global thresholds (if any) will take effect. actions: Actions | None = None-
Optional actions to take when the validation step meets or exceeds any set threshold levels. If provided, the
Actionsclass should be used to define the actions. brief: str | bool | None = None-
An optional brief description of the validation step that will be displayed in the reporting table. You can use the templating elements like
"{step}"to insert the step number, or"{auto}"to include an automatically generated brief. IfTruethe entire brief will be automatically generated. IfNone(the default) then there won’t be a brief. active: bool | Callable = True-
A boolean value or callable that determines whether the validation step should be active. Using
Falsewill make the validation step inactive (still reporting its presence and keeping indexes for the steps unchanged). A callable can also be provided; it will receive the data table as its single argument and must return a boolean value. The callable is evaluated before anypre=processing. dimension: str | None = None-
An optional data quality dimension to categorize this validation step for health scoring. One of
"completeness","validity","uniqueness","consistency","timeliness", or"volume"(or any custom string). IfNone(the default), the dimension is"uniqueness". This label appears in the validation report and feeds the overall and per-dimension health scores.
Returns
Validate- The Validate object with the added validation step.
Raises
TypeError-
If
columns=isn’t a column name or a list of them, or ifthreshold=,ngram=, ornum_perm=isn’t a number. ValueError-
If
method=isn’t"minhash"or"exact", ifthreshold=isn’t between0and1, or ifngram=ornum_perm=is less than1.
How Texts Are Compared
Texts are first normalized: in lower case (with Unicode compatibility forms folded), with punctuation replaced by spaces, and with runs of whitespace made one space. Texts that are the same once normalized are always near duplicates. With method="minhash", each text is also split into its overlapping n-grams of ngram= words, and two texts whose sets of n-grams have a Jaccard similarity of at least threshold= (the share of their n-grams that they have in common) are near duplicates. Comparing every pair of texts would take too long for a large table, so candidate pairs are found with MinHash signatures and locality-sensitive hashing (the way that corpora for language models are deduplicated), and each candidate pair is confirmed with its exact Jaccard similarity. No pair below the threshold is ever grouped, and a pair just above it is missed only rarely.
Groups of near duplicates are connected: if text A is a near duplicate of text B, and B of C, then A, B, and C are one group, even if A and C are less alike. A row with a missing value in any of columns= passes.
The group of each failing row (numbered from 1, in the order of the first row of each group) is in a column of the results called pb_cluster_, which is in the extracts of failing rows and in the step report, where the rows of each group are shown together. The step’s note says how many groups were found and how many rows are in them.
Every row of a group fails, as with rows_distinct(). When the rows of a validation are used to clean a table (with get_row_flags(), or in an Ablation), duplicates= decides which rows of a group are kept: "keep_first" keeps the first row of each group, "keep_last" the last, and "flag_all" keeps none.
Comparing texts runs in Python. A Polars or pandas table is compared as it is (a Polars LazyFrame is collected first). For a database table (through Ibis) or a PySpark DataFrame, the distinct rows of columns= are collected and compared, and the groups are joined back to the table on its own backend.
Examples
For the examples here, we’ll use a small table of documents, some of which are near duplicates of others.
Let’s validate that the documents have no near duplicates.
validation = (
pb.Validate(data=tbl)
.rows_near_distinct(columns="text")
.interrogate()
)
validationThree of the six rows fail: documents 1, 3, and 5 differ only in their punctuation and letter case, so they’re one group. Documents 2 and 6 differ by a word, which leaves fewer than 85% of their 5-word n-grams in common, so they pass. A lower threshold= (or a smaller ngram=) finds them too:
validation = (
pb.Validate(data=tbl)
.rows_near_distinct(columns="text", threshold=0.5)
.interrogate()
)
validation.get_step_report(i=1)| Report for Validation Step 1 ✗ ASSERTION 5 / 6 TEST UNIT FAILURES EXTRACT OF ALL 5 ROWS (WITH TEST UNIT FAILURES IN RED): |
|||
The step report shows the rows of each group together, with the group in the pb_cluster_ column. Now five rows fail, in two groups: the library documents and the trail documents.