Tag
PowerBench is a new open-source benchmark that evaluates how language models respond to power-shifting requests (self-empowerment, disempowerment, power grabbing) across 24 US and Chinese models, finding systematic refusal asymmetries tied to nationalities, agent users, and languages.
CuBEs 提出一种文化情境化的 LLM 行为评估框架，通过在测试场景中注入文化语境并在 12 种文化上构建人工标注数据集，评估了 13 个开源与闭源模型，发现文化盲的评估会遗漏非西方特有的偏见轴向（如宗教与殖民议题）。
This research paper examines how sociodemographic prompting affects LLMs' judgments in subjective tasks, finding that it often aligns models with majority groups while misrepresenting minority groups, with instruction-tuning identified as a potential cause.
This paper introduces ContextBias and ContextBench to evaluate bias persistence in text-to-image models, finding that bias increases in semantically unrelated contexts.
This pilot study uses Natural Language Autoencoders to probe whether Qwen2.5-7B internally represents Colombian identity, socioeconomic status, or stereotype-related information when processing Colombian-Spanish and English prompts, finding evidence of latent inferences before they are verbalized.
Introduces RPAM, a principled metric for evaluating associations in language models that demonstrates high predictive validity for downstream outputs, tested on Mistral-7B-Instruct, Mistral-7B, and GPT-2.
VIBE is a framework that evaluates generative bias in Large Audio-Language Models using open-ended tasks with human-recorded speech, revealing systematic biases triggered by gender and accent cues.
This paper presents the first bias evaluation of multimodal speech recognition models, finding significant accuracy differences across gender and ethnicity when pairing faces with audio, with implications for fairness in AI systems.