Tag
该论文系统研究了用LLM作为新颖性评判者时的不稳定性，发现微小的提示设计变化就能将成对准确率改变50分以上甚至低于随机水平，并质疑了自动化创意系统报告的新颖性提升结果。
This paper introduces Fable, a controlled dataset of 190,911 English prompt variants derived from 174K real user prompts, and evaluates 34 open-weight LLMs to show that models mirror higher-level rhetorical and lexical qualities of non-native speakers' prompts, producing lower-quality and less-fluent responses for less fluent users.
AfriSyCo studies answer switching in AI models around African-language factual content, showing how assertive framing and verification prompts significantly affect accuracy across languages and model checkpoints.
This paper introduces SyPS, a framework to evaluate how prompt variations affect sycophantic behavior in large language models, using the Sycophancy Prompt Sensitivity Score (SPSS).
This paper investigates the sensitivity of LLM evaluation benchmarks to different harness configurations, finding that config-fragile items disproportionately affect performance gaps between models, and that the choice of harness can determine leaderboard rankings.
This paper introduces an Interaction-based Prompt Sensitivity (IPS) metric to evaluate and explain prompt sensitivity in large language models by analyzing interactions. It applies IPS to 50 open-source LLMs, identifying factors like fine-tuning and model scale that reduce sensitivity through low-order interactions.
This paper identifies evaluation inconsistencies in diffusion LLM decoding methods, showing that prompt template choice significantly impacts rankings, and proposes guidelines for reliable evaluation.
A user tests the Gemma-4-e4b local model with a trolley-problem-like moral dilemma and finds that the model's refusal to harm crew changes when the scenario is framed as hypothetical, raising questions about prompt sensitivity versus actual reasoning.
This paper investigates the run-to-run reliability of LLM-as-a-Judge evaluations, finding that pairwise preferences flip 13.6% of the time on average, with significant first-position bias in GPT-4o-mini, and recommends multi-trial aggregation and position randomization.
PromptAudit is a controlled evaluation framework that isolates the effects of prompt formulations on LLM-based vulnerability detection, finding that chain-of-thought prompting achieves the best overall performance while prompt sensitivity must be treated as a first-class system property.