Tag
Anthropic's Opus 5 beats GPT-5.6 on ARC-AGI-3, but Tibo claims GPT-5.6 Sol becomes SoTA with two setting changes involving multi-context reasoning and canonical compaction.
A claim that GPT-5.6 Sol achieves state-of-the-art on ARC-AGI-3 by enabling reasoning across multiple context windows using canonical compaction.
OpenAI reveals that enabling retained reasoning and context compaction tripled GPT-5.6 Sol's ARC-AGI-3 benchmark scores, highlighting how harness settings significantly impact measured model performance.
GPT-5.6 Sol, a model that solved open math problems, initially struggled with the ARC-AGI-3 benchmark due to a harness memory limitation. Enabling two API settings tripled scores with 6x fewer output tokens.
OpenAI discovered that enabling retained reasoning and compaction settings in the API harness tripled GPT-5.6 Sol's scores on the ARC-AGI-3 benchmark while cutting output tokens by 6x, revealing that benchmark performance is heavily influenced by harness design.
Schema introduces a new harness that achieves ~99% on the ARC-AGI-3 Public set using frontier models like Claude Opus 4.8 and Fable 5, by improving the process around models rather than modifying weights.