Which Answer Did the Seventeen-Year-Old Write? A Behavioral Case Study of Author-Age Misattribution, Miscalibrated Confidence, and Non-Corrective Self-Explanation in Six Frontier Language Models

arXiv cs.CL Papers
language-models bias calibration authorship-attribution confidence frontier-models behavioral-study

Summary

A behavioral case study across six frontier LLMs finds they confidently misattribute a 17-year-old's exam answer to an adult, with stated confidence uncorrelated or inversely correlated with correctness, and articulate retrospective bias recognition that fails to improve blind judgments.

arXiv:2610.31946v1 Announce Type: new Abstract: Three students answered the same exam prompt by hand: "What is happiness? Are you happy?" Their ages, recorded at collection, are 17, 28, and 45. The 17-year-old's answer is hedged and formally phrased. Asked to match each answer to an age and gender label, most frontier language models confidently assign it to an adult. This paper reports a structured case study of that failure across six models: Claude Fable 5, Gemini 3.1 Pro, GPT-5.6 Sol, Grok 4.5, GPT-4o (2024), and DeepSeek V4. We ran 1,081 API sessions and scored 3,225 responses across six experimental conditions that manipulate framing, disclosure format, and audience-awareness instruction. Three of six models perform at or near chance (1/3) on the matching task while reporting 73-88% mean confidence. The youngest writer's answer is misattributed to an adult in up to 90% of trials. Removing the multiple-choice scaffold collapses accuracy further: three models never place within three years of the true age once open-ended estimation is required. Institutional framing ("a community-college biology final") significantly degrades the best-performing model's accuracy; a neutral label does not. A single audience-awareness instruction significantly improves one model (+21 points) and moves another to ceiling, with no detectable effect on the three weakest models. Stated confidence is uncorrelated or inversely correlated with correctness for five of six models. After ground truth is revealed, one model names its own stereotyping in 68 of 114 explanations, yet shows the lowest accuracy and highest confidence of any model under blind presentation: articulate retrospective diagnosis that does not track the quality of its judgments. This is a three-item, single-topic, single-language case study meant to isolate a mechanism, not to estimate a population bias rate. Data, code, and raw model outputs are publicly released.
Original Article

Similar Articles

How “reverse game theory” could keep AI agents in check

MIT Technology Review

MIT Technology Review reports that alignment researchers are exploring "reverse game theory" (mechanism design) to control powerful AI agents by designing incentive systems—such as cross-model checking and competitive dynamics—that encourage rule-following regardless of a model's internal workings. The approach is supported by a recent preprint by economist Stephen Morris and colleagues at MIT, Yale, and Columbia targeting sandbagging behavior.

@FinanceYF5: Much of the “common knowledge” about LLMs may have a shelf life of just 15 months. Stanford researchers used 20 new models to retest 50 widely cited findings: 44% no longer held, 34% had significantly weakened, and only 16% still held true. Fabricated citations, hallucinations, and stereotypes are declining, but sycophancy toward users and…

X AI KOLs Following

Stanford researchers retested 50 widely cited findings about LLMs using 20 new models and found that 44% no longer held, 34% had significantly weakened, and only 16% still held, suggesting that LLM research findings may have a “shelf life” of only about 15 months.

Epistemically Isolated Artificial Intelligence (EIAI)

Reddit r/ArtificialInteligence

The author introduces the concept of Epistemically Isolated Artificial Intelligence (EIAI) — AI systems deliberately limited to a defined knowledge environment — highlighting "Talkie: an LM from 1930," a language model trained only on pre-1931 text, as a fascinating example of temporally bounded epistemic reasoning.

Anthropic warns that it’s very close to RSI capability, aka self-building AI

TechCrunch AI

In its revised S-1A filing, Anthropic warned it may imminently meet its Responsible Scaling Policy's definition of 'dramatically accelerated progress attributable to AI,' suggesting it is close to achieving recursive self-improvement (RSI), while pledging to keep humans in the frontier model development loop.