Tag
Dialpad researchers propose a sociolinguistic approach to authoring and measuring user persona fidelity in agentic system evaluation, showing that concrete stylistic-rate personas improve stylistic adherence over flat descriptive baselines across five customer-service agents.
This paper examines covert dialect bias in large language models by analyzing their internal probability distributions across four English dialects, finding that models associate more negative housing-related adjectives with African American Vernacular English and Nigerian Pidgin, reflecting inherited biases similar to human discrimination.
This study introduces the GAPA dataset to demonstrate that physical descriptions often carry gender associations, and LLMs show systematic misalignment with human ratings, challenging assumptions about gender-neutral communication.
This paper introduces a cross-evaluation framework for benchmarking LLMs on Arabic cultural and sociolinguistic knowledge, using human SME ground truth and automated judges. The authors contribute a dataset of prompt-rubric pairs for Egyptian and Iraqi Arabic, evaluating frontier LLMs and finding that cultural reasoning remains a primary failure mode for automated grading.
A thesis investigating geographic dialect alignment in place-based social media communities in New Zealand, examining how Reddit communities reflect patterns of language variation similar to geographic dialect communities through analysis of lexical, morphosyntactic, and semantic features.