MPIE-Bench: Benchmarking Anatomically Plausible Multi-Person Interaction Editing

Hugging Face Daily Papers Papers
benchmark image-editing multi-person evaluation text-to-image

Summary

Introduces MPIE-Bench, a 2,500-sample benchmark for multi-person interaction image editing, along with MPIE-Eval, a mesh-based evaluation method that tracks human judgment more closely than VLM checklists across ten editors.

Text-to-image and personalized editing models now synthesize high-fidelity single-subject images with ease. Yet placing multiple named people into shared contact actions such as embrace, carry, or grapple still exposes major failures: fused limbs, invented extremities, and interpenetrating bodies. Existing evaluations largely overlook these anatomical and geometric issues, and VLM-as-a-judge checklists often saturate on Interaction while the errors remain obvious to humans. We introduce MPIE-Bench, a 2,500-sample benchmark of video-mined editing triplets spanning 405 scenes, 14 interaction categories, and four contact densities (C0-C3). We also propose MPIE-Eval, whose two new axes score contact-time geometry from a frozen public multi-person mesh reconstruction. Anatomy asks whether every human-like mass is explained by a complete set of reconstructed bodies, and Interaction asks whether the penetration and surface distance between those bodies match the contact the instruction asked for. Across ten editors, mesh Anatomy tops out at 0.65 and mesh Interaction at 0.72 on two different models, so no single editor is strong on both, while VLM checklists rate the same images above 0.95. A five-rater study confirms that both axes track human judgement more closely than a zero-shot VLM judge, and the rankings hold under ablation of every weight and threshold.
Original Article
View Cached Full Text

Cached at: 07/31/26, 05:52 AM

Paper page - MPIE-Bench: Benchmarking Anatomically Plausible Multi-Person Interaction Editing

Source: https://huggingface.co/papers/2607.27616

Abstract

Text-to-imageandpersonalizededitingmodelsnowsynthesizehigh-fidelitysingle-subjectimageswithease.Yetplacingmultiplenamedpeopleintosharedcontactactionssuchasembrace,carry,orgrapplestillexposesmajorfailures:fusedlimbs,inventedextremities,andinterpenetratingbodies.Existingevaluationslargelyoverlooktheseanatomicalandgeometricissues,andVLM-as-a-judgechecklistsoftensaturateonInteractionwhiletheerrorsremainobvioustohumans.WeintroduceMPIE-Bench,a2,500-samplebenchmarkofvideo-minededitingtripletsspanning405scenes,14interactioncategories,andfourcontactdensities(C0-C3).WealsoproposeMPIE-Eval,whosetwonewaxesscorecontact-timegeometryfromafrozenpublicmulti-personmeshreconstruction.Anatomyaskswhethereveryhuman-likemassisexplainedbyacompletesetofreconstructedbodies,andInteractionaskswhetherthepenetrationandsurfacedistancebetweenthosebodiesmatchthecontacttheinstructionaskedfor.Acrossteneditors,meshAnatomytopsoutat0.65andmeshInteractionat0.72ontwodifferentmodels,sonosingleeditorisstrongonboth,whileVLMchecklistsratethesameimagesabove0.95.Afive-raterstudyconfirmsthatbothaxestrackhumanjudgementmorecloselythanazero-shotVLMjudge,andtherankingsholdunderablationofeveryweightandthreshold.

View arXiv pageView PDFGitHub1Add to collection

Get this paper in your agent:

hf papers read 2607\.27616

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.27616 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.27616 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.27616 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

PaintBench: Deterministic Evaluation of Precise Visual Editing

Hugging Face Daily Papers

PaintBench is a new benchmark for evaluating precise visual editing in multimodal models, covering 20 operations across 4 categories with deterministic pixel-level evaluation. Testing 11 models reveals overall low performance, with the best model scoring only 17.1% mIoU.

MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

Hugging Face Daily Papers

MemoBench is a diagnostic benchmark for evaluating video generation models' memory consistency in dynamically changing environments, where objects disappear and reappear in updated states. It includes 360 ground-truth clips and an evaluation suite combining automated metrics with VQA-based assessment, revealing insights into memory consistency challenges.

Is This Edit Correct? A Multi-Dimensional Benchmark for Reasoning-Aware Image Editing

Hugging Face Daily Papers

This paper introduces RE-Edit, a benchmark for evaluating image editing systems across five reasoning dimensions (physical, environmental, cultural, causal, referential) to assess logical consistency beyond visual plausibility. The benchmark includes 1,000 samples and evaluates ten open-source and two commercial models, showing that even advanced systems struggle with implicit multi-dimensional reasoning.

SpeechEditBench: A Bilingual Multi-Attribute Benchmark for Instruction-Guided Speech Editing

Hugging Face Daily Papers

SpeechEditBench is a bilingual multi-attribute benchmark for evaluating instruction-guided speech editing across seven atomic tasks and compositional tasks, using an anchor-based evaluation protocol with three metrics. Evaluation of mainstream Speech LLMs reveals no single model excels across all dimensions, and compositional editing remains highly challenging.

VEFX-Bench: A Holistic Benchmark for Generic Video Editing and Visual Effects

Hugging Face Daily Papers

VEFX-Bench introduces a large-scale human-annotated video editing dataset (5,049 examples) with multi-dimensional quality labels and a specialized reward model for standardized evaluation of video editing systems. The paper addresses the lack of comprehensive benchmarks in AI-assisted video creation by providing VEFX-Dataset, VEFX-Reward, and a 300-video-prompt benchmark that reveals gaps in current editing models.