TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation
Summary
TRACE-Bench proposes a compositional operator framework and benchmark to diagnose multi-reference image generation capabilities by decomposing tasks into atomic operations for detailed evaluation.
View Cached Full Text
Cached at: 08/18/26, 11:53 AM
Paper page - TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation
Source: https://huggingface.co/papers/2608.16765
Abstract
This work proposes a compositional operator framework and TRACE-Bench to diagnose multi-reference image generation capabilities across atomic operations.
Despite recent advances in unified multimodal models formulti-reference image generation, existing benchmarks remain organized around predefined task types (e.g., “subject composition”), which are ill-suited to this combinatorial setting and lead to fragmented coverage, uncontrolled complexity, and little diagnostic value. Recognizing that diverse multi-reference tasks share a common set of atomic operations, we adopt a capability-oriented perspective and formalize four operators:Anchor(f),Disentangle(g),Apply(oplus), andCompose(C). Any multi-reference prompt can then be represented as acompositional formulaover these operators, whose structural complexity is quantified by the number of operator slots. Building on this formulation, we constructTRACE-Bench, comprising approximately 1,600 evaluation cases across slot counts 1--8, built from 631 formula templates and around 4,000 reference images spanning diverse artistic styles and real-world subjects. The formula structure directly drives anoperator-aligned evaluationprotocol for per-capability scoring and adiagnostic tree analysisfor recursive failure localization. Evaluating 9 leading models reveals insights invisible to holistic scoring: the primary bottleneck lies indisentanglement(g) andattribute binding(oplus) rather than scene-level composition (C), with even the best model scoring only 0.74 on attribute fidelity. Project page: https://amuseum-whr.github.io/TraceBench
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2608\.16765
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.16765 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.16765 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.16765 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
MultiRef-Compass: Towards Comprehensive Evaluation of Multi-Reference-to-Audio-Video Generation
This paper introduces MultiRef-Compass, a comprehensive benchmark for multi-reference-to-audio-video generation, comprising 350 curated samples and an evaluation protocol with four dimensions including Basic Quality, Reference Consistency, Audio-Visual Consistency, and Instruction Following.
Trace: A Taxonomy-Guided Environment for Multidomain Visual Reasoning
TRACE is a taxonomy-guided environment with 1,000 visual reasoning tasks across 11 domains. Training Qwen2.5-VL-3B and Qwen2.5-VL-7B on 64,000 TRACE instances improves their macro-average performance across 24 external benchmarks by 3.51 and 4.06 percentage points respectively.
Is This Edit Correct? A Multi-Dimensional Benchmark for Reasoning-Aware Image Editing
This paper introduces RE-Edit, a benchmark for evaluating image editing systems across five reasoning dimensions (physical, environmental, cultural, causal, referential) to assess logical consistency beyond visual plausibility. The benchmark includes 1,000 samples and evaluates ten open-source and two commercial models, showing that even advanced systems struggle with implicit multi-dimensional reasoning.
CPI-Bench: A Comprehensive,Practical and Intelligent Benchmark for Real-World Image Editing
CPI-Bench is a comprehensive benchmark for real-world image editing that evaluates multi-image tasks, practical applications, and reasoning-based editing to better differentiate model performance.
MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
MemoBench is a diagnostic benchmark for evaluating video generation models' memory consistency in dynamically changing environments, where objects disappear and reappear in updated states. It includes 360 ground-truth clips and an evaluation suite combining automated metrics with VQA-based assessment, revealing insights into memory consistency challenges.