UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation

Hugging Face Daily Papers Papers
benchmark web-application-generation vision-language-models interaction-inference ui-generation visual-fidelity

Summary

UI2App presents a benchmark for evaluating vision-language models on generating executable web applications from UI screenshots, focusing on interaction inference beyond visual fidelity, and finds significant gaps in current models' ability to infer complete interaction behavior.

Large language models (LLMs) have demonstrated growing competence in web page generation. However, existing text-driven approaches rely on complex prompts that impose substantial demands on users and offer limited expressivity for page layout and cross-page visual coherence. Image-driven paradigms, which take UI screenshots as input, align more closely with real development workflows. However, current benchmarks focus primarily on visual fidelity and lack a systematic evaluation of the interaction capabilities in generated artifacts. To address this gap, we introduce UI2App, the first benchmark targeting interaction inference, the ability to recover application behavior from screenshots alone, without any textual or behavioral guidance. UI2App comprises 327 screenshots grouped into 45 state-coherent screenshot sets for runnable multi-route web applications. We design an end-to-end pipeline that evaluates each artifact along four dimensions: executability, navigation reachability, visual fidelity, and interaction inference. The interaction metric (IIS) assesses inferred interactions by functional correctness and state-management complexity, crediting any valid implementation rather than matching a single reference. Experiments on six frontier vision-language models reveal a marked capability mismatch between visual reconstruction and interaction realization: the visual-fidelity leader scores only 7.5 on IIS, ranking fourth and trailing the IIS leader by 5.2x. High-complexity interactions such as cross-page state remain a pervasive bottleneck, with half of the evaluated models scoring exactly zero on this dimension. Overall, the results indicate that inferring complete interaction behavior from static screenshots remains a key challenge for models.
Original Article
View Cached Full Text

Cached at: 07/21/26, 06:38 PM

Paper page - UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation

Source: https://huggingface.co/papers/2607.06306 Published on Jul 7

·

Submitted byhttps://huggingface.co/cestlavieChen

CHENon Jul 21

Abstract

Largelanguagemodels(LLMs)havedemonstratedgrowingcompetenceinwebpagegeneration.However,existingtext-drivenapproachesrelyoncomplexpromptsthatimposesubstantialdemandsonusersandofferlimitedexpressivityforpagelayoutandcross-pagevisualcoherence.Image-drivenparadigms,whichtakeUIscreenshotsasinput,alignmorecloselywithrealdevelopmentworkflows.However,currentbenchmarksfocusprimarilyonvisualfidelityandlackasystematicevaluationoftheinteractioncapabilitiesingeneratedartifacts.Toaddressthisgap,weintroduceUI2App,thefirstbenchmarktargetinginteractioninference,theabilitytorecoverapplicationbehaviorfromscreenshotsalone,withoutanytextualorbehavioralguidance.UI2Appcomprises327screenshotsgroupedinto45state-coherentscreenshotsetsforrunnablemulti-routewebapplications.Wedesignanend-to-endpipelinethatevaluateseachartifactalongfourdimensions:executability,navigationreachability,visualfidelity,andinteractioninference.Theinteractionmetric(IIS)assessesinferredinteractionsbyfunctionalcorrectnessandstate-managementcomplexity,creditinganyvalidimplementationratherthanmatchingasinglereference.Experimentsonsixfrontiervision-languagemodelsrevealamarkedcapabilitymismatchbetweenvisualreconstructionandinteractionrealization:thevisual-fidelityleaderscoresonly7.5onIIS,rankingfourthandtrailingtheIISleaderby5.2x.High-complexityinteractionssuchascross-pagestateremainapervasivebottleneck,withhalfoftheevaluatedmodelsscoringexactlyzeroonthisdimension.Overall,theresultsindicatethatinferringcompleteinteractionbehaviorfromstaticscreenshotsremainsakeychallengeformodels.

View arXiv pageView PDFProject pageGitHub1Add to collection

Get this paper in your agent:

hf papers read 2607\.06306

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.06306 in a model README.md to link it from this page.

Datasets citing this paper1

#### UI2App/UI2App Updated8 days ago • 492 • 1

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.06306 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Looks Right, Works Right: A Project-Level Benchmark for Multi-Screen Mobile App Generation

arXiv cs.AI

MobileForge is a benchmark for project-level multi-screen mobile app generation, evaluating multimodal LLMs on build success, cross-page navigation, visual fidelity, maintainability, and efficiency. Experiments on six frontier multimodal LLMs show current models can compile and reach pages but still struggle with interactive navigation and visual quality.