bluffbench2

Data science is a practice of noticing and caring about subtle data quality issues.

Source Logs

Among other things, data science is a practice of noticing and caring about subtle data quality issues. bluffbench2 is an LLM evaluation that measures how effectively AI agents raise data quality concerns when faced with minor artifacts in data visualizations.

The eval harness is a relatively generic coding agent harness with some prompting related to data analysis. The agent carries out a few “lull” turns, making a couple plots and tables unrelated to the eval. Then, at some point, it will be asked to produce a data visualization that produces a subtle visual artifact (that could feasibly result from a real data generating process):

In this example, the agent is then graded on whether it mentions the fact that there is a cluster of points that appear to follow the “fitted” line suspiciously tightly. Across samples, most artifacts get missed:

A bar plot showing scores for several frontier models. The two leaders, Gemini 3.5 Flash and Claude Fable 5, score in the mid-high teens. Models from OpenAI cluster at the bottom, never eclipsing 10%.

bluffbench2 is the successor to bluffbench.

Implemented in R with vitals.