LEARNER · GLOBAL
What does an LLM preference test measure: the model or the test design?
arXiv meta-analysis: LLM preference measurements (what a model 'likes' inferred from prompts) can be 50% model behavior, 50% test instrument artifacts. Questionnaire design, phrasing, context order shape results as much as actual model values.
WHY IT MATTERS
When banks run LLM evaluations for fairness, bias, or alignment, results depend heavily on evaluation design. Two banks testing the same model on different benchmarks may reach opposite conclusions. Signals need for standardized evaluation protocols, not one-off testing.
Source: arXiv · 2026-08-27