Tag
Proposes Partially Adjudicated Design-Based Supervised Learning (PA-DSL), a method that corrects noisy human labels using a small set of adjudicated cases to debias automated classifiers, achieving nominal coverage and reducing RMSE by 10-17% in experiments.
This paper investigates how adding demographic attributes in prompts affects LLM-human agreement across tasks, finding that while a few high-signal attributes improve alignment, over-specification degrades it. The study uses five open-source LLMs and neuron probing to show that attribute signal quality and coherence matter more than quantity.