Thinking Outside the Box: Can Language Models Rely on External Guidance Selectively?

Hugging Face Daily Papers Papers
language-models agents workflows reinforcement-learning fine-tuning benchmark multi-agent

Summary

The paper introduces Box² Bench to measure whether language models can selectively rely on external guidance, finding that capable models remain vulnerable to misleading workflows, and shows that counterfactual SFT plus RL trained only on bad workflows can teach models to resist unreliable guidance and better exploit helpful ones.

Agent harnesses often improve language models with human-designed workflows, but as models grow more capable, unreliable guidance can increasingly constrain their execution. We call the ability to benefit from useful guidance while overriding unreliable guidance thinking outside the box. We introduce Box^2-Bench, which holds the model and task fixed while varying workflow reliability to isolate how models regulate their reliance on guidance. On Box^2-Bench, frontier models often benefit from reliable guidance but remain vulnerable when it is misleading or becomes unreliable. To test whether this capability can be learned, we train two open-weight models using bad workflows, reserving good workflows for evaluation. We explore two complementary training strategies: counterfactual supervised fine-tuning improves robustness, while outcome-based reinforcement learning can shift the balance toward greater use of helpful workflows. We further find that this behavior extends beyond workflows to other forms of external information, improving peer correction and robustness to corrupted memory. Together, our results identify selective reliance on fallible external information as a dimension of agent reliability not captured by task performance alone.
Original Article
View Cached Full Text

Cached at: 10/01/26, 04:20 AM

Paper page - Thinking Outside the Box: Can Language Models Rely on External Guidance Selectively?

Source: https://huggingface.co/papers/2609.39578 thinkoutsidethebox

https://huggingface.co/papers/2609.39578#thinking-outside-the-box-can-language-models-rely-on-external-guidance-selectivelyThinking Outside the Box: Can Language Models Rely on External Guidance Selectively?

Language model agents increasingly rely on human designed workflows to solve complex tasks. Good workflows can substantially improve performance by providing useful intermediate steps and structure.

But what happens when the workflow is worse than the model itself?

As models become more capable, blindly following external guidance can become a limitation rather than an advantage. A strong agent should benefit from useful guidance while being able to override guidance when it becomes unreliable.

We call this abilitythinking outside the box.

https://huggingface.co/papers/2609.39578#measuring-selective-relianceMeasuring Selective Reliance

截屏2026-10-01 11.34.26

To study this behavior, we introduceBox² Bench.

The idea is simple: we keep themodel and task fixed, but vary the reliability of the workflow. We compare how the same model performs with no workflow, a helpful workflow, and a misleading workflow.

This lets us separate two abilities that are usually mixed together:

Can the model use good guidance? And can it resist bad guidance?

Across models and tasks, we find a consistent pattern. Reliable workflows often improve performance, but misleading workflows can still substantially hurt it. Even capable models remain sensitive to unreliable external guidance.

https://huggingface.co/papers/2609.39578#learning-to-think-outside-the-boxLearning to Think Outside the Box

截屏2026-10-01 11.34.54

We next ask whether selective reliance can be learned.

We train two open weight models using onlybad workflows, while reserving good workflows for evaluation.

Counterfactual supervised fine tuning makes models substantially more robust to misleading guidance. Outcome based reinforcement learning can further shift their behavior toward making greater use of helpful workflows.

Importantly, the models never see good workflows during training. Their improvement therefore suggests that training can shape a more general strategy for deciding how much to trust external guidance.

https://huggingface.co/papers/2609.39578#beyond-workflowsBeyond Workflows

This behavior is not limited to workflows.

Models trained in our setting also improve when interacting with other forms of fallible external information. We observe betterpeer correctionin multi agent reasoning and greater robustness tocorrupted memory.

These results point to a broader capability: agents need not only to use external information, but also to regulate their reliance on it.

https://huggingface.co/papers/2609.39578#toward-more-reliable-agentsToward More Reliable Agents

Most agent evaluations ask whether a task was solved.

Box² Bench asks an additional question:

Does the model know when to listen?

As agents increasingly operate with workflows, memories, tools, and other agents, selective reliance on fallible external information may become an important dimension of reliability that task performance alone does not capture.

Sometimes the best use of a box is knowing when to step outside it.

Similar Articles

Do Language Models Know Their Own Constraints?

arXiv cs.CL

The paper explores if language models can articulate constraints they've learned through fine-tuning. It discovers that behavioral compliance improves but explicit reporting diminishes.

Learning What to Learn: Stage-Specific Data Sets for SFT-then-RL in Small Language Model Reasoning

arXiv cs.CL

This paper proposes a difficulty-aware SFT-then-RL framework for training small language models (≤3B parameters) on reasoning tasks, arguing that data difficulty should be strategically aligned with the distinct roles of SFT (learning new skills) and RL (consolidating partial skills). The authors introduce a Bridge mechanism for hard SFT samples and Critique Fine-Tuning for RL failures, showing consistent improvements across five reasoning benchmarks.