What would make an uploader-run refusal table independently reproducible?

Reddit r/LocalLLaMA News
ai-safety reproducibility model-cards refusal-tables machine-learning research-standards

Summary

The article discusses the requirements for making an uploader-run refusal table in AI models independently reproducible, highlighting the need for detailed replication packets including raw generations, decoding settings, per-prompt labels, and scoring code.

A refusal range in a model card is only reproducible if another person can reconstruct how every prompt reached its label. OrcaRouter's Qwen3.8-27B FP8 card reports 0–6% refusal for the abliterated checkpoint with thinking off, versus 63.6–99% for the base across eight listed sets. The card also says this is an uploader-run, opening-phrase classifier rather than an independent evaluation. The supplied material does not include the raw generations or an independent reproduction. That leaves several results hidden inside each aggregate: the decoding settings, classifier version, per-prompt labels, and how caveated compliance was handled. What is the smallest replication packet you would require before treating a refusal table as a reusable result: raw generations, exact decoding settings, per-prompt labels, executable scoring code, or all four?
Original Article

Similar Articles

@adithya_s_k: https://x.com/adithya_s_k/status/2067628584680710292

X AI KOLs Timeline

This article discusses how coding agents can cheat evaluations by copying known patches, and introduces Repo2RLEnv, a tool to create verifiable coding environments from real repositories to build robust benchmarks and training data for AI coding agents.

Same agent, same prompt, different runs. Which output do you ship?

Reddit r/AI_Agents

The author observes that running the same task with Claude Code across different sessions yields varying decision patterns, making it hard to choose outputs that are safe to ship, and highlights the lack of tooling for evaluating agent decision profiles.