I asked Sol Max to compare the output of Claude Opus 5 High and GPT 5.6 Sol Max on a specific puzzle on ARC-AGI-3 where Opus 5 had a 98.81% score and GPT 5.6 Sol Max had a 21.42% score

Replays: https://arcprize.org/replay/ce05e7b4-5428-4142-87a9-e805102c0b45 - Opus 5 High https://arcprize.org/replay/8239e65e-dc8d-470d-ab49-6d37618f00ea - GPT 5.6 Sol Max We know that ARC-AGI-3 only passes forward the visible output and discards the thinking / reasoning text between each turn. --------------------------------------------------------------------------- Analysis from Sol Max For each turn: The harness sends the model the conversation history plus the latest game frame. It receives visible output, private reasoning/thinking, and token usage. It preserves the visible output and parsed game action. It discards private reasoning: Anthropic thinking blocks and, normally, OpenAI reasoning items. It sends the action to ARC, receives the updated frame/state, then adds that as the next user message. Comparison between the two replays Opus clearly outperformed Sol in these two runs: Opus reached WIN at 7/7 levels, while Sol ended NOT_FINISHED at 3/7. The decisive divergence happened on level 4. Metric Opus Sol Final result WIN — 7/7 Unfinished — 3/7 Click actions 406 344 In-level resets 2 8 Logged game overs 2 1 Elapsed wall time 7:19:16 8:50:52 Median visible response ~283 words ~24 words Correctly formatted/executed clicks 406/406 344/344 Progress by level Level Opus: clicks / cumulative time Sol: clicks / cumulative time 1 8 / 0:04:21 7 / 0:05:07 2 8 / 0:06:29 11 / 0:10:52 3 34 / 1:08:22 29 / 1:11:41 4 121 / 3:45:40 297 / unfinished 5 118 / 5:26:51 — 6 33 / 6:01:12 — 7 84 / 7:19:16 — The first three levels were remarkably close. Sol even used fewer clicks on levels 1 and 3. On level 4: Opus maintained detailed “Context Notes,” tracking coordinates, tank levels, hypotheses, and multi-action plans. It reached two game overs, incorporated what it learned, and solved the level on its third attempt. Sol’s visible reasoning was much more reactive and local. It restarted level 4 eight times and repeatedly changed its interpretation of the board. One particularly damaging Sol sequence interpreted the puzzle as a hazard/countdown, deliberately waited across several actions, and the final wait produced GAME_OVER. It later returned to the lock interpretation but never finished the level. Both models produced valid action syntax every time, so the difference was planning and state tracking—not an output parser problem. Opus was substantially more verbose: about 116,000 visible words versus 8,300 for Sol. Sol’s concise responses were not necessarily computationally cheaper, though: its log reports 2.28M output tokens, including 2.26M classified as reasoning, versus Opus’s 1.96M output tokens with zero separately reported reasoning tokens. That accounting is likely provider-specific, and all recorded cost fields are zero, so these files cannot support a reliable price comparison. Both runs used the same game ID, seven-level target, and identical initial 64×64 frame. However, this is still only one run per model, and the files do not identify exact model versions, settings, or why the Sol run stopped. The conclusion is therefore about these runs: Opus’s durable state representation overcame the long-horizon puzzle; Sol was more concise but became trapped in reset and interpretation loops. ------------------------------------------------------------------------ My understanding: Based on this, it feels that Opus is able to preserve more reasoning, hypothesis, state and planning between each turn by increasing the text content on the visible output, while 5.6 Sol is only saying brief state of what has happened / it is doing on the visible output, this means that if both models do not have access to their reasoning / thinking output because it is discarded, then Opus has way more information and context to perform better, because Sol is depending a lot more from the thinking / reasoning output than Opus. Opus 5 High GPT 5.6 Sol Max