Stories about PRISM
1 related stories
The Illusion of $\textit{What If}$: Evaluating the Breakdown of Counterfactual Reasoning in LLMs
AI InsightA new arXiv study introduces WhatIfBench, the first open-domain, long-horizon counterfactual causal reasoning benchmark with 220 questions across STEM, HSS, and Hybrid scenarios, plus PRISM which converts free-form explanations into semantic causal graphs for automatic evaluation. Compared with prior benchmarks that restrict variables and single gold answers, this shifts evaluation toward open-domain, open-form causal processes, filling a gap in counterfactual reasoning assessment.Key TakeawayCounterfactual evaluation shifts from constrained variables to open-domain long-horizon causal processes.Why It MattersCounterfactual reasoning affects model reliability and interpretability; this benchmark enables systematic measurement of causal reasoning weaknesses in LLMs.Who's Affected- AI ResearchersGain a new tool to measure open-domain counterfactual reasoning and diagnose causal capability gaps.
- Benchmark BuildersPRISM's causal-graph evaluation method can be reused for other free-form reasoning benchmarks.
- Model DevelopersShould watch WhatIfBench results and improve long-horizon causal reasoning.
What's NextWatch for PRISM's correlation with existing constrained benchmarks and whether WhatIfBench gets adopted by mainstream leaderboards.Importance 62/100