Tag
The paper measures dynamic evaluation (taking gradient steps on text being read) across softmax-attention, hybrid linear-attention, and masked-diffusion language models, finding 5–20% next-token loss reductions but showing the adaptation captures style/vocabulary rather than factual retrieval, while low-rank distillation of a model's own conclusions reaches in-context performance without degrading GSM8K.
SWE-Interact has been updated to enable dynamic, user-driven evaluation sessions for software engineering tasks, with native support in Harbor for simulating multi-turn user-agent interactions.
This paper investigates contamination risks in dynamic evaluation for multimodal automated fact-checking, showing that even post-cutoff claims can be contaminated and that contamination significantly inflates performance.