ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation

Hugging Face Daily Papers Papers
multimodal image-generation agentic tool-use reinforcement-learning

Summary

Introduces ToolArtist, a fully agentic image generation model built from a unified multimodal model, using SFT and reinforcement learning (RAD-GRPO) to dynamically orchestrate reasoning, tool use, and image generation.

Text-to-image (T2I) models can produce visually compelling images, yet they remain limited on open-world tasks that require complex semantic understanding, multi-step reasoning, and the integration of external world knowledge. Existing efforts introduce agent capabilities into image generation, but they either prescribe a fixed workflow or place only a subset of the open-world image generation process under agent control. Consequently, reasoning, tool invocation, and image generation are not coordinated by a single policy. We propose ToolArtist, a fully agentic image generation model obtained by post-training a Unified Multimodal Model (UMM). ToolArtist dynamically orchestrates reasoning, external tool use, and native image generation within one unified policy. During Supervised Fine-Tuning (SFT), we equip a teacher agent with search tools alongside an image-generation tool. We then convert the collected trajectories into a UMM compatible format, where the image-generation tool is concealed while the resulting generated images are retained. During Reinforcement Learning (RL), we develop an agentic RL infrastructure for UMMs and introduce Reason-Act-Draw GRPO (RAD-GRPO), which uses complementary intent and quality rewards to jointly optimize the model. Experiments show that placing the entire open-world image-generation process under an agent policy consistently outperforms approaches with fixed pipelines or only partially agent-controlled components. We release the training data and the complete post-training infrastructure.
Original Article
View Cached Full Text

Cached at: 08/06/26, 05:50 AM

Paper page - ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation

Source: https://huggingface.co/papers/2608.04436

Abstract

Text-to-image(T2I)modelscanproducevisuallycompellingimages,yettheyremainlimitedonopen-worldtasksthatrequirecomplexsemanticunderstanding,multi-stepreasoning,andtheintegrationofexternalworldknowledge.Existingeffortsintroduceagentcapabilitiesintoimagegeneration,buttheyeitherprescribeafixedworkfloworplaceonlyasubsetoftheopen-worldimagegenerationprocessunderagentcontrol.Consequently,reasoning,toolinvocation,andimagegenerationarenotcoordinatedbyasinglepolicy.WeproposeToolArtist,afullyagenticimagegenerationmodelobtainedbypost-trainingaUnifiedMultimodalModel(UMM).ToolArtistdynamicallyorchestratesreasoning,externaltooluse,andnativeimagegenerationwithinoneunifiedpolicy.DuringSupervisedFine-Tuning(SFT),weequipateacheragentwithsearchtoolsalongsideanimage-generationtool.WethenconvertthecollectedtrajectoriesintoaUMMcompatibleformat,wheretheimage-generationtoolisconcealedwhiletheresultinggeneratedimagesareretained.DuringReinforcementLearning(RL),wedevelopanagenticRLinfrastructureforUMMsandintroduceReason-Act-DrawGRPO(RAD-GRPO),whichusescomplementaryintentandqualityrewardstojointlyoptimizethemodel.Experimentsshowthatplacingtheentireopen-worldimage-generationprocessunderanagentpolicyconsistentlyoutperformsapproacheswithfixedpipelinesoronlypartiallyagent-controlledcomponents.Wereleasethetrainingdataandthecompletepost-traininginfrastructure.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2608\.04436

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.04436 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.04436 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.04436 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

GenClaw: Code-Driven Agentic Image Generation

Hugging Face Daily Papers

GenClaw introduces a code-driven agentic image generation framework that breaks the black-box paradigm by mimicking the human creative process: conceptualizing, sketching with code (SVG/HTML/Three.js), and then using generative models for texture and photorealism.

IA para imágenes artísticas

Reddit r/AI_Agents

An introduction to AI tools for generating and editing artistic images, likely covering popular models or platforms.