scorers.token_count()

Score text by its number of tokens.

Usage

Source

scorers.token_count(
    tokenizer,
    column=None,
    name=None,
    version=None,
    batch_size=None,
)

A language model reads text as tokens, and most have a limit on how many they can take, so a corpus for training (or a set of prompts) usually needs texts with neither too few tokens nor too many. The score of a row is the number of tokens of its text, with the tokenizer of a model, which makes min_val= and max_val= bounds on the length of each text in tokens.

Parameters

tokenizer: str | Any

The tokenizer. It can be:

• the name of a Hugging Face tokenizer (e.g., "gpt2" or "bert-base-uncased"), which needs the transformers package (special tokens aren’t counted)
• "tiktoken:" and the name of a tiktoken encoding or model (e.g., "tiktoken:cl100k_base" or "tiktoken:gpt-4o"), which needs the tiktoken package
• "whitespace", to count the words of the text, split on whitespace
• a tokenizer object with an encode() method (from tiktoken, tokenizers, or transformers)
• a function that takes a text and returns its tokens, or the number of them

column: str | None = None

The column with the text to score. It can be left out if the step has one column.

name: str | None = None

A name for the scorer, for the score cache and the report. A tokenizer given by name names the scorer; a tokenizer object or a function needs a name= for the scores to be kept on disk.

version: str | None = None

A version, which is part of the key of the score cache.

batch_size: int | None = None

The number of texts scored at a time (512 by default).

Returns

Scorer

A scorer for Validate.col_vals_scored().

Examples

A real check would use the tokenizer of a model, such as "tiktoken:cl100k_base". To keep this example self-contained, the tokenizer here splits text into words and punctuation.

import re

import pointblank as pb
import polars as pl

def tokenize(text):
    return re.findall(r"\w+|[^\w\s]", text)

tbl = pl.DataFrame(
    {
        "prompt": [
            "Summarize the article in two sentences.",
            "Hi",
            "Translate this paragraph into French, keeping the names as they are.",
        ]
    }
)

pb.Validate(data=tbl).col_vals_scored(
    columns="prompt",
    scorer=pb.scorers.token_count(tokenize),
    min_val=3,
    max_val=12,
).interrogate()

		STEP	COLUMNS	VALUES	TBL	EVAL	UNITS	PASS	FAIL	W	E	C	EXT
#4CA64C66	1	col_vals_scored()	prompt	3–12 TOKENS		✓	3	1 0.33	2 0.67	—	—	—
Notes Step 1 (scores) Scored by token_count(tokenizer=__main__.tokenize): 3 distinct rows scored. Scores from 1 to 13 (median 7).

The second prompt has one token and the third has thirteen, so both fail.