Validate.col_vals_chat_format()

Validate whether column values are well-formed chat conversations.

Usage

Validate.col_vals_chat_format(
    columns,
    roles=("system", "user", "assistant"),
    alternate=True,
    first_role=None,
    last_role=None,
    allow_empty=False,
    role_key="role",
    content_key="content",
    na_pass=False,
    pre=None,
    segments=None,
    thresholds=None,
    actions=None,
    brief=None,
    active=True,
    dimension=None,
)

The col_vals_chat_format() validation method checks that each value of a column is a well-formed conversation of the kind used to fine-tune a chat model: a list of messages, each with a role and some content. Each row is a test unit, and a row fails if its value isn’t a list of messages, has a message without a role or with a role that isn’t in roles=, has a system message anywhere but first, has turns that don’t alternate between roles, starts or ends with the wrong role, or has a message without content. Malformed conversations are common in chat data that’s been converted from other formats, and a training run can fail on them, or silently learn from them.

Parameters

columns: str | list[str] | Column | ColumnSelector | ColumnSelectorNarwhals: A single column or a list of columns to validate. Can also use col() with column selectors to specify one or more columns. If multiple columns are supplied or resolved, there will be a separate validation step generated for each column. The values are lists of messages (such as a Polars column of lists of structs, or a pandas column of lists of dictionaries) or the JSON text of them, either as a list or as an object with a "messages" key.
roles: Sequence[str] = ("system", "user", "assistant"): The roles that a message can have. The default is ("system", "user", "assistant"). If "system" is one of them, a system message can only be the first message.
alternate: bool = True: Should the turns (the messages after a first system message) alternate, so that no role follows itself? The default is True.
first_role: str | None = None: The role of the first turn (the first message after a system message), such as "user". The default, None, doesn’t check it.
last_role: str | None = None: The role of the last message, such as "assistant" (for a conversation that ends with the reply that a model should learn). The default, None, doesn’t check it.
allow_empty: bool = False: Can a message have empty content (missing, or only whitespace)? The default is False.
role_key: str = "role": The key of the role in each message. The default is "role".
content_key: str = "content": The key of the content in each message. The default is "content".
na_pass: bool = False: Should a missing value pass? The default is False.
pre: Callable | None = None: An optional preprocessing function or lambda to apply to the data table during interrogation. This function should take a table as input and return a modified table.
segments: SegmentSpec | None = None: An optional directive on segmentation, which serves to split a validation step into multiple (one step per segment). Can be a single column name, a tuple that specifies a column name and its corresponding values to segment on, or a combination of both (provided as a list).
thresholds: int | float | bool | tuple | dict | Thresholds | None = None: Set threshold failure levels for reporting and reacting to exceedences of the levels. The thresholds are set at the step level and will override any global thresholds set in Validate(thresholds=...). The default is None, which means that no thresholds will be set locally and global thresholds (if any) will take effect.
actions: Actions | None = None: Optional actions to take when the validation step meets or exceeds any set threshold levels. If provided, the Actions class should be used to define the actions.
brief: str | bool | None = None: An optional brief description of the validation step that will be displayed in the reporting table. You can use the templating elements like "{step}" to insert the step number, or "{auto}" to include an automatically generated brief. If True the entire brief will be automatically generated. If None (the default) then there won’t be a brief.
active: bool | Callable = True: A boolean value or callable that determines whether the validation step should be active. Using False will make the validation step inactive (still reporting its presence and keeping indexes for the steps unchanged). A callable can also be provided; it will receive the data table as its single argument and must return a boolean value. The callable is evaluated before any pre= processing.
dimension: str | None = None: An optional data quality dimension to categorize this validation step for health scoring. One of "completeness", "validity", "uniqueness", "consistency", "timeliness", or "volume" (or any custom string). If None (the default), the dimension is "validity". This label appears in the validation report and feeds the overall and per-dimension health scores.

Returns

Validate: The Validate object with the added validation step.

Raises

TypeError: If roles= isn’t a list of role names, or if role_key= or content_key= isn’t a string.
ValueError: If roles= is empty, or if first_role= or last_role= isn’t one of roles=.

The Problems Found

The first problem found with a conversation is in a column of the results called pb_reason_, which is in the extracts of failing rows and in the step report. The step’s note says how many rows have each problem. The problems are checked in this order:

not_messages: the value isn’t a list of messages (or the JSON text of one)
no_messages: the list of messages is empty
bad_message: a message isn’t a mapping with a role (under role_key=) that’s a string
unknown_role: a message has a role that isn’t in roles=
system_position: a system message isn’t the first message
first_role: the first turn doesn’t have the role of first_role=
not_alternating: a role follows itself (with alternate=True)
last_role: the last message doesn’t have the role of last_role=
empty_content: a message has no content, or content that’s only whitespace (unless allow_empty=True)

A Polars or pandas column can hold the messages as lists (of structs or dictionaries) or as JSON text. For a database table (through Ibis) or a PySpark DataFrame, the column must hold JSON text: its distinct values are collected and checked, and the problems are joined back to the table on its own backend.

Examples

For the examples here, we’ll use a small table of conversations, stored as JSON text.

import pointblank as pb
import polars as pl

tbl = pl.DataFrame(
    {
        "id": [1, 2, 3, 4],
        "messages": [
            '[{"role": "user", "content": "Hi!"}, '
            '{"role": "assistant", "content": "Hello! How can I help?"}]',
            '[{"role": "user", "content": "Hi!"}, {"role": "user", "content": "Hello?"}]',
            '[{"role": "system", "content": "Be brief."}, '
            '{"role": "user", "content": "Capital of France?"}, '
            '{"role": "assistant", "content": "Paris."}]',
            '[{"role": "user", "content": "What is 2 + 2?"}, '
            '{"role": "bot", "content": "4"}]',
        ],
    }
)

Let’s validate that each conversation is well formed and ends with a reply of the assistant.

validation = (
    pb.Validate(data=tbl)
    .col_vals_chat_format(columns="messages", last_role="assistant")
    .interrogate()
)

validation

		STEP	COLUMNS	VALUES	TBL	EVAL	UNITS	PASS	FAIL	W	E	C	EXT
#4CA64C66	1	col_vals_chat_format()	messages	CHAT ALTERNATING LAST: ASSISTANT		✓	4	2 0.50	2 0.50	—	—	—
Notes Step 1 (chat_format) 2 rows aren't well-formed conversations: 1 has roles that don't alternate (`not_alternating`) and 1 has a role that isn't allowed (`unknown_role`). The problem with each row is in the `pb_reason_` column.

Two of the four conversations fail: the second has two user messages in a row, and the fourth has a message with the role "bot". The step report shows the problem with each failing row in the pb_reason_ column.

validation.get_step_report(i=1)

	id Int64	messages String	pb_reason_ String
Report for Validation Step 1 ✗ ASSERTION `messages is a chat of {system, user, assistant}` 2 / 4 TEST UNIT FAILURES IN COLUMN 2 EXTRACT OF ALL 2 ROWS (WITH TEST UNIT FAILURES IN RED):
2	2	[{"role": "user", "content": "Hi!"}, {"role": "user", "content": "Hello?"}]	not_alternating
4	4	[{"role": "user", "content": "What is 2 + 2?"}, {"role": "bot", "content": "4"}]	unknown_role

The third conversation passes: its system message is first, and its turns alternate between the user and the assistant.