Reward-Driven LLM Agent Workflows: Synthesizing POMDP Routing and Self-Correction for Autonomous Decision-Making

# Reward-Driven LLM Agent Workflows: Synthesizing POMDP Routing and Self-Correction for Autonomous Decision-Making
Source: [https://arxiv.org/html/2607.17038](https://arxiv.org/html/2607.17038)
Amez Amanj AliSchool of Computer Science and Technology Harbin Institute of Technology, Shenzhen 25sf51117@stu\.hit\.edu\.cnKuo\-Kun TsengSchool of Computer Science and Technology Harbin Institute of Technology, Shenzhen kktseng@hit\.edu\.cn

###### Abstract

This paper addresses key technical challenges in current large language model \(LLM\) applications, including long\-horizon planning, sparse reward attribution, and dynamic environmental interaction, by designing and optimizing an intelligent agent workflow\. The proposed architecture is fundamentally based on the synthesis of core AI paradigms, including Visual, Language, Generative, Reinforcement, and Agent Intelligence\. Unlike conventional baseline models that rely heavily on static zero\-shot or few\-shot prompting techniques and inherently lack robust perception\-action loops, our approach introduces a sophisticated Partially Observable Markov Decision Process \(POMDP\) routing mechanism\. This mechanism is augmented with an internal, self\-correcting reward model that actively evaluates decision trajectories before execution\. By integrating multimodal inputs and advanced reinforcement learning principles—such as proximal policy optimization and value function approximation—the agent effectively maintains long\-term structural memory and dynamically adapts its reasoning pathways to mitigate error accumulation\. Extensive empirical experiments conducted on complex public datasets, including the ALFWorld embodied simulation environment and the WebShop online navigation benchmark, as well as a custom multimodal simulation platform, demonstrate a remarkable 24\.5% absolute improvement in task success rate and trajectory efficiency over mainstream baselines such as the standard ReAct framework\. Comprehensive ablation studies further confirm the independent, highly significant contribution of the reward\-driven critique module in suppressing hallucination rates\. This research successfully bridges abstract theoretical foundations of reinforcement learning and GNN\-based memory with state\-of\-the\-art autonomous agent workflows\. Ultimately, the resulting architecture offers a practical, scalable reference framework for developing artificial intelligence technologies in complex, multi\-step autonomous systems requiring high reliability and self\-reflection\. The complete code implementation and framework are publicly available at[https://github\.com/01Amez/RLAW\_Implementation](https://github.com/01Amez/RLAW_Implementation)\.

Keywords:Large Language Model, Agent Intelligence, Reinforcement Learning, Multimodal Fusion, Autonomous Workflow, Graph Neural Networks, Partially Observable Markov Decision Process\.

## 1Introduction

### 1\.1Research Background & Problem Statement

Artificial intelligence research has recently undergone a monumental paradigm shift, transitioning from the development of isolated, static pattern recognition models toward the deployment of interactive, fully autonomous systems capable of executing complex workflows in dynamic environments\. Building upon the taxonomy of structural and cognitive modules in artificial intelligence, we observe that AI has evolved from rudimentary perception to highly interconnected, cognitive decision\-making systems\. Historically, Structural Intelligence established the mathematical bedrock of artificial neural networks, enabling non\-linear feature extraction through backpropagation and gradient descent\. Following this, Visual and Language Intelligence revolutionized the processing of highly dimensional sensory and textual data via convolutional architectures and self\-attention mechanisms, respectively\. However, despite the undeniable success of these specialized modules in isolated benchmarks, the isolated application of these models is now increasingly insufficient for solving real\-world, open\-ended tasks that require sequential reasoning, environmental interaction, and multi\-modal alignment\.

The recent emergence and rapid scaling of Large Language Models \(LLMs\) have catalyzed the development of a new frontier: Agent Intelligence\. In this paradigm, agents are designed to autonomously perceive their environment, formulate strategic plans, invoke external tools, and execute actions to achieve high\-level goals specified by natural language instructions\. By synthesizing Generative Intelligence with logical deduction, these agents emulate human\-like cognitive processes\. Nevertheless, the transition from theory to practical application remains fraught with critical challenges\. Chief among these is the problem of compounding reasoning degradation over long time horizons\. Traditional LLM agents frequently suffer from ”cascading hallucination errors,” where a single illogical assumption early in a reasoning trace completely derails the entire execution pathway\. Furthermore, these agents suffer from limited long\-term memory capacity; they struggle to maintain an accurate structural representation of the world state, often forgetting key environmental constraints over extended interactions\.

Perhaps the most significant theoretical gap in current agent architectures is the absence of formal mathematical objectives that govern the decision\-making process\. Without an explicit, reward\-driven critique mechanism, LLMs generate actions based purely on maximum likelihood estimation over their pre\-training distribution, rather than selecting actions that explicitly maximize goal completion probabilities\. This fundamental disconnect often leads to infinite execution loops, redundant tool invocations, or invalid actions within partially observable environments where state transitions are not fully deterministic\. To resolve these systemic vulnerabilities, this paper directly addresses these issues by formulating the agentic planning problem through the mathematical lens of Reinforcement Intelligence\. We propose a mathematically grounded, self\-correcting LLM agent workflow that enforces a strict ”propose\-critique\-execute” loop, effectively reducing reasoning degradation over long time horizons and aligning generative capabilities with formal value maximization principles\.

### 1\.2Research Objectives & Core Contributions

In order to bridge the gap between static language generation and dynamic, goal\-oriented autonomy, this research integrates multi\-module theoretical knowledge to engineer a significantly improved, unified agent workflow\. The overarching objective is to demonstrate how these foundational principles of reinforcement learning, language generation, and graph belief propagation can be synthesized to solve state\-of\-the\-art problems in Agent Intelligence\. Specifically, we aim to augment the cognitive engine of an LLM with rigorous evaluation metrics derived from Reinforcement Learning\. The main objectives and core contributions of this study are detailed as follows:

- •Theoretical Framing and Architectural Optimization:Leveraging the deep theoretical interplay between Language Intelligence and Agent Intelligence, we formally optimize the standard agent architecture by introducing a Partially Observable Markov Decision Process \(POMDP\) routing mechanism\. Rather than treating the LLM merely as an autoregressive text generator, this approach elevates the LLM to a formal policy function within a strict reinforcement learning framework\. By explicitly defining the state space, action space, and transition dynamics, we impose mathematical rigor on the agent’s generative outputs\.
- •Advanced Multi\-Module Knowledge Integration:We deeply integrate multi\-module knowledge by specifically synthesizing Reinforcement Intelligence concepts \(such as value functions, advantage estimation, and reward modeling\) with Generative Intelligence capabilities\. We propose a novel, self\-correcting critique module acting as an internal ”Critic\.” This module actively evaluates candidate reasoning paths proposed by the main LLM ”Actor” and decisively eliminates suboptimal or hallucinated trajectories before they are executed in the external environment\. This drastically reduces critical hallucination rates and prevents error accumulation\.
- •Comprehensive Empirical Validation:We provide extensive empirical evidence validating the proposed architecture across multiple challenging environments\. Through rigorous quantitative analysis on both physical simulation tasks \(ALFWorld\) and web\-based navigation tasks \(WebShop\), we demonstrate that our architecture not only achieves higher absolute success rates but also significantly improves parameter efficiency—allowing smaller, localized models to outperform massive, unconstrained baselines\. These results offer highly actionable insights for the scalable, industrial deployment of intelligent agents in computationally constrained scenarios\.

### 1\.3Paper Organization

The remainder of this academic report is systematically structured as follows: Section 2 provides a comprehensive review of the foundational modules of intelligence and surveys the state\-of\-the\-art research progress in the domain of autonomous language agents\. Section 3 formalizes the target problem and details the theoretical foundations and core algorithm design of our proposed Reward\-Driven LLM Agent Workflow \(RLAW\)\. Section 4 presents the rigorous experimental setup, including dataset descriptions and baseline comparisons, followed by a detailed analysis of the quantitative results, ablation studies, and architectural visualizations\. Section 5 engages in an in\-depth discussion interpreting the findings, acknowledging current research limitations, and offering a detailed reflection on design insights and future directions\. Finally, Section 6 succinctly concludes the paper by summarizing our primary achievements and outlining prospective directions for future multi\-module integration research\.

## 2Related Work

### 2\.1Foundational Knowledge Review

The intellectual foundation of this research is deeply rooted in the systematic technical evolution of cognitive computing architectures\[[5](https://arxiv.org/html/2607.17038#bib.bib5),[1](https://arxiv.org/html/2607.17038#bib.bib1)\]\. To provide necessary context and establish the theoretical prerequisites for our proposed architecture, we first conduct a comprehensive review of the eight core intelligent modules that collectively form the modern AI landscape, summarized in Table[1](https://arxiv.org/html/2607.17038#S2.T1)\.

- •Structural Intelligence:Serving as the absolute foundational bedrock of modern deep learning, Structural Intelligence encompasses the architecture and optimization of basic neural paradigms, including Multi\-Layer Perceptrons \(MLPs\), traditional Convolutional Neural Networks \(CNNs\), and Recurrent Neural Networks \(RNNs\)\. These architectures establish the fundamental principles of gradient descent, backpropagation, and non\-linear feature extraction\. Without a rigorous understanding of how loss landscapes are navigated mathematically, higher\-level cognitive structures cannot be optimized effectively\.
- •Visual Intelligence:Expanding directly upon the spatial invariance properties of CNNs, Visual Intelligence tackles complex downstream tasks such as real\-time object detection, instance segmentation, and semantic scene parsing\. By learning hierarchical spatial representations, this module allows artificial systems to parse and understand complex visual environments\. In the context of embodied AI, visual intelligence is a highly critical component, as it provides the raw sensory inputs necessary for an agent to navigate physical or simulated 3D environments without relying solely on abstracted text descriptions\.
- •Language Intelligence:Unquestionably the catalyst for the current AI renaissance, this module is powered by the revolutionary Transformer architecture\[[1](https://arxiv.org/html/2607.17038#bib.bib1)\]\. Language Intelligence fundamentally focuses on sequence modeling, semantic understanding, and probabilistic text generation\. The core mechanism of scaled dot\-product self\-attention allows the network to process massive context windows by dynamically weighing the relevance of all past tokens\. This capability has enabled Large Language Models \(LLMs\) such as GPT\-3 and LLaMA\[[8](https://arxiv.org/html/2607.17038#bib.bib8)\]to serve as the highly capable cognitive and reasoning engines driving modern autonomous agents\.
- •Generative Intelligence:Moving beyond discriminative classification, Generative Intelligence encompasses models explicitly designed to map and sample from complex, high\-dimensional data distributions\. Prominent architectures include Variational Autoencoders \(VAEs\), Generative Adversarial Networks \(GANs\)\[[7](https://arxiv.org/html/2607.17038#bib.bib7)\], and the recently dominant Diffusion Models\. In advanced agent workflows, generative capabilities are heavily utilized not only for creative synthesis but also for ”world modeling”—allowing an agent to internally simulate the potential outcomes of its actions before executing them, effectively synthesizing hypothetical memory structures\.
- •Graph Intelligence:Traditional neural networks often struggle with non\-Euclidean data structures\. Graph Intelligence, spearheaded by Graph Convolutional Networks \(GCNs\)\[[9](https://arxiv.org/html/2607.17038#bib.bib9)\]and Graph Attention Networks \(GATs\), provides the mathematical framework for processing relational data defined by nodes and edges\. Within complex agent workflows, Graph Intelligence is highly applicable for structuring semantic memory banks, maintaining knowledge graphs of environmental constraints, and dynamically mapping the interconnected relationships between various entities encountered during task execution\.
- •Multimodal Intelligence:True general intelligence requires the synthesis of disparate sensory inputs\. Multimodal Intelligence focuses on the deep fusion of different data modalities—primarily aligning continuous visual embeddings with discrete language tokens, as demonstrated in architectures like Vision Transformers \(ViT\)\[[10](https://arxiv.org/html/2607.17038#bib.bib10)\]and CLIP\. For an autonomous agent to operate successfully in the real world, it is absolutely essential that it perceives a unified, coherent state space derived simultaneously from both linguistic instructions and dynamic visual frames\.
- •Reinforcement Intelligence:While supervised learning relies on static datasets, Reinforcement Intelligence formalizes the dynamic process of learning through trial, error, and delayed environmental feedback\. Core algorithms such as Deep Q\-Networks \(DQN\) and Proximal Policy Optimization \(PPO\)\[[5](https://arxiv.org/html/2607.17038#bib.bib5)\]provide the rigorous mathematical framework necessary for an agent to maximize long\-term cumulative rewards\. Reinforcement learning shifts the optimization objective from simply predicting the next word accurately to actually completing a complex, multi\-step goal\.
- •Agent Intelligence:Representing the integration layer of cognitive AI, this module synthesizes all previously discussed forms of intelligence\. Agent Intelligence embeds the generative capabilities of LLMs within a continuous perception\-action\-reward loop\. By equipping the model with access to external software tools, specialized APIs, read\-write memory banks, and iterative planning algorithms, the agent transitions from a passive answering machine into an active, autonomous problem solver capable of long\-horizon goal resolution\.

Table 1:Taxonomy and architectural synthesis of the eight core intelligent course modules, outlining their primary architectures, core functional objectives, mathematical optimization foundations, and explicit integration role in the RLAW framework\.
### 2\.2State\-of\-the\-Art Research Progress

The broader landscape of artificial intelligence research has recently pivoted sharply, focusing entirely on transitioning from static, prompt\-response models to interactive, goal\-driven agents\. This paradigm shift was initially sparked by the introduction of Reinforcement Learning from Human Feedback \(RLHF\)\[[16](https://arxiv.org/html/2607.17038#bib.bib16)\]and Chain\-of\-Thought \(CoT\) prompting\[[6](https://arxiv.org/html/2607.17038#bib.bib6)\], which significantly improved the logical deduction capabilities of LLMs by forcing them to articulate intermediate reasoning steps before arriving at a final conclusion\. Building upon this, the seminal ReAct framework\[[2](https://arxiv.org/html/2607.17038#bib.bib2)\]further revolutionized the field by directly combining CoT reasoning traces with executable environmental actions\. However, while ReAct demonstrated impressive zero\-shot capabilities, empirical studies reveal that it remains highly prone to error accumulation; a single incorrect assumption during the reasoning phase can cascade, causing the agent to fall into repetitive action loops or fail the task entirely\.

To expand the functional boundary of what agents can achieve, subsequent frameworks such as Toolformer\[[11](https://arxiv.org/html/2607.17038#bib.bib11)\], HuggingGPT\[[12](https://arxiv.org/html/2607.17038#bib.bib12)\], and fully autonomous agent loops like AutoGPT\[[17](https://arxiv.org/html/2607.17038#bib.bib17)\]successfully demonstrated the capability of LLMs to autonomously select, format, and invoke external APIs\. This allowed language models to bypass their inherent computational limitations by offloading math to calculators, factual queries to search engines, and complex data processing to specialized sub\-models\. While this vastly expanded the scope of Agent Intelligence, the problem of reliability and self\-correction remained largely unsolved\.

In response, recent cutting\-edge studies have begun exploring advanced tree\-search decoding and self\-reflection methodologies\. Notably, the Reflexion framework\[[13](https://arxiv.org/html/2607.17038#bib.bib13)\]introduced the concept of ”verbal reinforcement learning,” wherein an agent utilizes a separate critique prompt to analyze its past failures and explicitly write a natural language summary of what went wrong, which is then appended to its memory for future trials\. However, despite the success of these reflective methods, they are often heuristically driven and critically lack formal mathematical reward mechanisms\. They rely heavily on the LLM’s inherent ability to recognize its own mistakes, which is notoriously unreliable in smaller, quantized models\.

Furthermore, recent advances in prompting structures have migrated from linear reasoning paths to tree\-based and graph\-based searching paradigms\. Models such as Tree of Thoughts \(ToT\)\[[18](https://arxiv.org/html/2607.17038#bib.bib18)\]and Graph of Thoughts \(GoT\)\[[19](https://arxiv.org/html/2607.17038#bib.bib19)\]leverage systematic search algorithms \(e\.g\., Breadth\-First Search, Depth\-First Search, orA∗A^\{\*\}\) over LLM\-generated thoughts\. This allows the system to backtrack, explore multiple alternative reasoning paths, and evaluate intermediate steps\. However, these search\-based methods typically rely on heuristic evaluations \(e\.g\., prompting the LLM itself to score thoughts from 1 to 5\), which are prone to self\-delusion and command significant computational overhead\. RLAW, on the other hand, introduces a formal reinforcement learning\-based value function approximation as the critique model\. This ensures a more grounded, mathematically sound evaluation process that can run efficiently on smaller models\. Our research directly addresses this critical gap in the literature\. By formally integrating a mathematically structured, reward\-driven critique model into the reasoning loop, we leverage the theoretical foundations of PPO and actor\-critic methods to provide a robust, reliable, and mathematically sound approach to agentic self\-correction\. A detailed structural and optimization comparison of these state\-of\-the\-art architectures is presented in Table[2](https://arxiv.org/html/2607.17038#S2.T2)\.

Table 2:Comparative analysis of state\-of\-the\-art autonomous LLM agent frameworks across core structural dimensions, including search topology, validation protocols, computational complexity, and mathematical optimization targets\.
### 2\.3Synergy of Reinforcement Learning and Language Models

This research project serves as a direct synthesis and practical culmination of language modeling and reinforcement learning\. In general language modeling tasks, standard autoregressive models exhibit significant semantic drift during open\-ended generation\. Concurrently, traditional reinforcement learning agents suffer from sample inefficiency in high\-dimensional state spaces\. Fusing these paradigms, we utilize high\-capacity language models as the initialization policy space, while reinforcement learning provides the formal optimization target\. This synthesis directly resolves the limitations observed when applying either model in isolation\.

## 3Methodology / Proposed Method

### 3\.1Problem Definition & Symbol Explanation

To systematically analyze and optimize the behavior of an intelligent agent within a complex environment, it is imperative to establish a rigorous mathematical framework\. We define the target artificial intelligence task as a sequential decision\-making process wherein an autonomous agent interacts with a highly dynamic, partially observable environment over discrete time steps to achieve a specific natural language goal\. Because the agent rarely has access to the complete underlying state of the environment \(for instance, the contents of a closed drawer in a simulated room are unknown until opened\), we standardly model this interactive process using the formal structure of a Partially Observable Markov Decision Process \(POMDP\)\[[14](https://arxiv.org/html/2607.17038#bib.bib14)\]\.

The POMDP is mathematically defined by the 7\-tuple⟨S,A,T,R,Ω,O,γ⟩\\langle S,A,T,R,\\Omega,O,\\gamma\\rangle, where each component governs a specific aspect of the interaction loop\. The mathematical definition, space, physical interpretation, and concrete manifestation of each parameter within our proposed RLAW framework are systematically structured and detailed in Table[3](https://arxiv.org/html/2607.17038#S3.T3)\.

Table 3:Mathematical formalization of the 7\-tuple POMDP decision\-making framework utilized in the proposed RLAW agent architecture\.Given this formal POMDP structure, the primary objective of our research is to learn an optimal policyπ∗\\pi^\{\*\}\. Because the environment is partially observable, the policy cannot simply map the current observation to an action; instead, it must map the entire history of past observations and actionsHt=\(o1,a1,o2,a2,…,ot\)H\_\{t\}=\(o\_\{1\},a\_\{1\},o\_\{2\},a\_\{2\},\\dots,o\_\{t\}\)to a probability distribution over the action spaceAA\. The optimal policy must strictly maximize the expected cumulative discounted reward:

π∗=arg⁡maxπ⁡𝔼π​\[∑t=0∞γt​rt\+1\]\\pi^\{\*\}=\\arg\\max\_\{\\pi\}\\mathbb\{E\}\_\{\\pi\}\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\_\{t\+1\}\\right\]\(1\)
Since the underlying statests\_\{t\}is hidden, the agent must rely on its belief statebtb\_\{t\}, which is a probability distribution over all possible statess∈Ss\\in S\. The belief update equation is given by:

bt\+1​\(s′\)=η​O​\(ot\+1∣s′\)​∑s∈ST​\(s′∣s,at\)​bt​\(s\)b\_\{t\+1\}\(s^\{\\prime\}\)=\\eta O\(o\_\{t\+1\}\\mid s^\{\\prime\}\)\\sum\_\{s\\in S\}T\(s^\{\\prime\}\\mid s,a\_\{t\}\)b\_\{t\}\(s\)\(2\)whereη=1/P​\(ot\+1∣bt,at\)\\eta=1/P\(o\_\{t\+1\}\\mid b\_\{t\},a\_\{t\}\)is a normalization constant\. In our language\-agent workflow, tracking the exact continuous belief statebtb\_\{t\}analytically is mathematically intractable due to the environment’s infinite dimensionality\. Therefore, we approximate the belief state representation by combining the raw observation historyo≤to\_\{\\leq t\}with a structured Graph MemoryMM, which serves as a compressed, non\-Euclidean representation of the agent’s current belief about the world state\. The Graph Memory is formally updated at each step via a memory transition function:

Mt=fu​p​d​a​t​e​\(Mt−1,ot,at−1\)M\_\{t\}=f\_\{update\}\(M\_\{t\-1\},o\_\{t\},a\_\{t\-1\}\)\(3\)wherefu​p​d​a​t​ef\_\{update\}updates the node attributes and edge weights based on the new textual and visual observations\.

By explicitly defining the problem within this theoretical POMDP framework, we can bridge the stochastic text\-generation properties of Large Language Models and the rigorous optimization objectives of Reinforcement Learning\.

### 3\.2Course Theoretical Preliminaries

Before detailing the specifics of our proposed architectural solution, we must revisit and establish the fundamental mathematical foundations of our models\. Our solution is fundamentally a synthesis of two major modules: Language Intelligence and Reinforcement Intelligence\. In the domain of Language Intelligence, our agent’s cognitive engine relies entirely on the Transformer architecture\[[1](https://arxiv.org/html/2607.17038#bib.bib1)\]\. Unlike recurrent networks that process data sequentially, the Transformer relies on a highly parallelizable mechanism known as scaled dot\-product attention\. Given a sequence of input tokens, the model projects them into Query \(QQ\), Key \(KK\), and Value \(VV\) matrices\. The attention scores are calculated as follows:

Attention​\(Q,K,V\)=softmax​\(Q​KTdk\)​V\\text\{Attention\}\(Q,K,V\)=\\text\{softmax\}\\left\(\\frac\{QK^\{T\}\}\{\\sqrt\{d\_\{k\}\}\}\\right\)V\(4\)wheredkd\_\{k\}is the dimensionality of the key vectors, used as a scaling factor to prevent the softmax function from entering regions with vanishing gradients\. This attention mechanism allows the LLM agent to dynamically ”attend” to the most relevant pieces of information across its entire historical context windowHtH\_\{t\}\. When generating a reasoning trace, the agent uses this equation to weigh all past observations, internal thoughts, and previous tool failures contextually, effectively simulating a working memory system\.

To process long\-horizon histories, the Transformer projects the query, key, and value vectors intohhdistinct representation subspaces:

MultiHead​\(Q,K,V\)=Concat​\(head1,head2,…,headh\)​WO\\text\{MultiHead\}\(Q,K,V\)=\\text\{Concat\}\(\\text\{head\}\_\{1\},\\text\{head\}\_\{2\},\\dots,\\text\{head\}\_\{h\}\)W^\{O\}\(5\)where each head is calculated as:

headi=Attention​\(Q​WiQ,K​WiK,V​WiV\)\\text\{head\}\_\{i\}=\\text\{Attention\}\(QW\_\{i\}^\{Q\},KW\_\{i\}^\{K\},VW\_\{i\}^\{V\}\)\(6\)andWiQ∈ℝdm​o​d​e​l×dkW\_\{i\}^\{Q\}\\in\\mathbb\{R\}^\{d\_\{model\}\\times d\_\{k\}\},WiK∈ℝdm​o​d​e​l×dkW\_\{i\}^\{K\}\\in\\mathbb\{R\}^\{d\_\{model\}\\times d\_\{k\}\},WiV∈ℝdm​o​d​e​l×dvW\_\{i\}^\{V\}\\in\\mathbb\{R\}^\{d\_\{model\}\\times d\_\{v\}\}, andWO∈ℝh​dv×dm​o​d​e​lW^\{O\}\\in\\mathbb\{R\}^\{hd\_\{v\}\\times d\_\{model\}\}are learnable parameter projection matrices\.

Concurrently, in the domain of Reinforcement Intelligence, the fundamental objective is to evaluate the quality of a given policyπ\\pi\. This evaluation is governed by the Bellman equation\[[5](https://arxiv.org/html/2607.17038#bib.bib5)\], which recursively defines the value functionVπ​\(s\)V^\{\\pi\}\(s\)\. The value function represents the expected return starting from statessand following policyπ\\pithereafter:

Vπ​\(s\)=𝔼π​\[∑k=0∞γk​rt\+k\+1∣st=s\]V^\{\\pi\}\(s\)=\\mathbb\{E\}\_\{\\pi\}\\left\[\\sum\_\{k=0\}^\{\\infty\}\\gamma^\{k\}r\_\{t\+k\+1\}\\mid s\_\{t\}=s\\right\]\(7\)Similarly, the state\-action value function, or Q\-function,Qπ​\(s,a\)Q^\{\\pi\}\(s,a\), defines the expected return after taking a specific actionaain statessand then followingπ\\pi\. In traditional reinforcement learning, these value functions are estimated using massive lookup tables or simple deep neural networks \(DQNs\)\. However, in the high\-dimensional, open\-ended semantic space of language tasks, standard DQNs fail entirely\.

For policy optimization under reinforcement learning, Proximal Policy Optimization \(PPO\)\[[5](https://arxiv.org/html/2607.17038#bib.bib5)\]utilizes a clipped surrogate objective function to prevent destabilizingly large policy updates:

LC​L​I​P​\(θ\)=𝔼^t​\[min⁡\(rt​\(θ\)​A^t,clip​\(rt​\(θ\),1−ϵ,1\+ϵ\)​A^t\)\]L^\{CLIP\}\(\\theta\)=\\hat\{\\mathbb\{E\}\}\_\{t\}\\left\[\\min\\left\(r\_\{t\}\(\\theta\)\\hat\{A\}\_\{t\},\\text\{clip\}\(r\_\{t\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\)\\hat\{A\}\_\{t\}\\right\)\\right\]\(8\)where the probability ratiort​\(θ\)r\_\{t\}\(\\theta\)is defined as:

rt​\(θ\)=πθ​\(at∣st\)πθo​l​d​\(at∣st\)r\_\{t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\}\{\\pi\_\{\\theta\_\{old\}\}\(a\_\{t\}\\mid s\_\{t\}\)\}\(9\)andA^t\\hat\{A\}\_\{t\}is the estimated advantage function at time steptt\. By utilizing this objective, we can align the LLM policyπθ\\pi\_\{\\theta\}to maximize expected rewards while maintaining training stability\. The advantage functionA^t\\hat\{A\}\_\{t\}measures whether a proposed actionata\_\{t\}is better or worse than the average action expected in statests\_\{t\}, which is mathematically computed using the value function baselineVϕ​\(st\)V^\{\\phi\}\(s\_\{t\}\)as:

A^t=Rt\+γ​Vϕ​\(st\+1\)−Vϕ​\(st\)\\hat\{A\}\_\{t\}=R\_\{t\}\+\\gamma V^\{\\phi\}\(s\_\{t\+1\}\)\-V^\{\\phi\}\(s\_\{t\}\)\(10\)whereRtR\_\{t\}is the discounted cumulative reward\.

Our proposed methodology essentially fuses these two profound paradigms\. We utilize the Transformer’s scaled dot\-product attention \(Equation[4](https://arxiv.org/html/2607.17038#S3.E4)\) to handle the complex state representation and natural language generation, thereby granting the agent immense cognitive processing power\. Simultaneously, we explicitly train an auxiliary component to approximate the value function defined by the Bellman equation \(Equation[7](https://arxiv.org/html/2607.17038#S3.E7)\), granting the agent a directive, goal\-oriented focus\. This fusion transforms the LLM from a passive text predictor into a rigorous, value\-maximizing actor within a dynamic environment\.

### 3\.3Core Model & Algorithm Design

We propose the Reward\-Driven LLM Agent Workflow \(RLAW\), illustrated in Figure[1](https://arxiv.org/html/2607.17038#S3.F1)\. RLAW consists of three phases: Generation, Critique, and Execution\.

EnvironmentStatests\_\{t\}, Rewardrtr\_\{t\}MultimodalPerceptionObsoto\_\{t\}LLM Policyπθ\\pi\_\{\\theta\}Reasoningztz\_\{t\}Actiona^t\\hat\{a\}\_\{t\}Critique ModelInternal RewardRc​r​i​t​i​q​u​eR\_\{critique\}Action Executionata\_\{t\}Proposea^t\\hat\{a\}\_\{t\}Pass \(R≥τR\\geq\\tau\)Fail \(R<τR<\\tau\)RegenerateInteractFigure 1:System architecture of the Reward\-Driven LLM Agent Workflow \(RLAW\)\. The diagram illustrates the closed\-loop Perception\-Action cycle enhanced by the internal critique model, representing a robust POMDP process\.1\. Generation Phase \(Actor\):At any given discrete time steptt, the agent receives a highly complex, multimodal observationoto\_\{t\}from the environment\. Rather than processing this observation in isolation, the agent accesses its internal structural memory bankMM\. This memory bank is not a simple text log; it is actively processed using Graph Intelligence techniques \(specifically, Graph Attention Networks\) to model non\-Euclidean relationships among entities, such as spatial proximity \(e\.g\., ”key is inside drawer”\) or semantic hierarchy\. Fusing the current observationoto\_\{t\}with the updated memory graphMM, the core LLM acts as the stochastic ”Actor\.” It autoregressively generates a candidate internal reasoning traceztz\_\{t\}\(the ”thought”\), followed immediately by a candidate executable actionata\_\{t\}\. The behavioral policy is deeply parameterized by the frozen, pre\-trained LLM weightsθ\\theta:

PromptActor=Concat​\(Goal,Ht,Serialize​\(Mt\)\)\\text\{Prompt\}\_\{\\text\{Actor\}\}=\\text\{Concat\}\(\\text\{Goal\},H\_\{t\},\\text\{Serialize\}\(M\_\{t\}\)\)\(11\)\(zt,at\)∼πθ​\(z,a∣PromptActor\)\(z\_\{t\},a\_\{t\}\)\\sim\\pi\_\{\\theta\}\(z,a\\mid\\text\{Prompt\}\_\{\\text\{Actor\}\}\)\(12\)This formulation \(Equation[12](https://arxiv.org/html/2607.17038#S3.E12)\) ensures that the proposed action is heavily conditioned not only on the immediate visual and textual stimuli but also on the historical trajectory of past actionsa<ta\_\{<t\}and the structurally preserved world\-stateMM\. The graph representation is converted to a serialized, textual context description\. The GAT network acts as an auxiliary module that can be used for downstream value classification or state estimation, while the LLM consumes the structured text mapping to maintain standard sequence\-to\-sequence interfaces\.

To maintain structural relationships within the memoryMM, we construct a semantic scene graph in which nodesvi∈Vv\_\{i\}\\in Vrepresent discovered entities or rooms, and edgesei,j∈Ee\_\{i,j\}\\in Erepresent spatial or functional connections\. The node featureshi\(l\)h\_\{i\}^\{\(l\)\}at thellth layer of our Graph Attention Network \(GAT\) memory processor are dynamically updated using a self\-attention mechanism:

hi\(l\+1\)=σ​\(∑j∈𝒩​\(i\)αi,j​W\(l\)​hj\(l\)\)h\_\{i\}^\{\(l\+1\)\}=\\sigma\\left\(\\sum\_\{j\\in\\mathcal\{N\}\(i\)\}\\alpha\_\{i,j\}W^\{\(l\)\}h\_\{j\}^\{\(l\)\}\\right\)\(13\)where𝒩​\(i\)\\mathcal\{N\}\(i\)denotes the set of immediate spatial neighbors of nodeii,W\(l\)W^\{\(l\)\}is a shared linear projection weight matrix, andαi,j\\alpha\_\{i,j\}are the normalized attention coefficients computed as:

αi,j=exp⁡\(LeakyReLU​\(𝐚T​\[W​hi∥W​hj\]\)\)∑k∈𝒩​\(i\)exp⁡\(LeakyReLU​\(𝐚T​\[W​hi∥W​hk\]\)\)\\alpha\_\{i,j\}=\\frac\{\\exp\\left\(\\text\{LeakyReLU\}\\left\(\\mathbf\{a\}^\{T\}\[Wh\_\{i\}\\parallel Wh\_\{j\}\]\\right\)\\right\)\}\{\\sum\_\{k\\in\\mathcal\{N\}\(i\)\}\\exp\\left\(\\text\{LeakyReLU\}\\left\(\\mathbf\{a\}^\{T\}\[Wh\_\{i\}\\parallel Wh\_\{k\}\]\\right\)\\right\)\}\(14\)where𝐚\\mathbf\{a\}is a learnable attention parameter vector, and∥\\paralleldenotes the concatenation operator\. This graph\-attention formulation allows the memory module to selectively integrate information from distant spatial locations and objects based on the current goal statement, thereby maintaining a sharp, highly compressed world representation\.

2\. Critique Phase \(Internal Reward Mechanism\):The most critical innovation of our proposed RLAW architecture is the introduction of a strict, self\-reflecting evaluation loop\. Unlike the standard ReAct framework, which immediately executesata\_\{t\}in the external environment, our workflow intercepts the candidate action\. A secondary, highly specialized lightweight critique model \(the ”Critic”\) evaluates the proposed action against the overarching semantic goal\. We formally define an internal critique reward function,Rc​r​i​t​i​q​u​e​\(ot,zt,at\)R\_\{critique\}\(o\_\{t\},z\_\{t\},a\_\{t\}\), that outputs a continuous scalar representing the action’s logical consistency, safety, and probability of advancing toward the goal state\.

The Critique Model’s reward evaluation is parameterized by a lightweight neural modelfϕf\_\{\\phi\}, which is fine\-tuned to classify whether a given transition is logical\. The Critique model is optimized using causal cross\-entropy loss to output evaluation decisions conforming to a strict JSON schema:

ℒ​\(ϕ\)=−∑i=1Llog⁡Pϕ​\(yi∣y<i,𝐱\)\\mathcal\{L\}\(\\phi\)=\-\\sum\_\{i=1\}^\{L\}\\log P\_\{\\phi\}\(y\_\{i\}\\mid y\_\{<i\},\\mathbf\{x\}\)\(15\)where𝐱\\mathbf\{x\}contains the goal, history, and proposed action, andyyis the serialized target JSON containing the rating metrics\. The critique reward is then set asRc​r​i​t​i​q​u​e=σ​\(fϕ​\(ot,zt,at\)\)R\_\{critique\}=\\sigma\(f\_\{\\phi\}\(o\_\{t\},z\_\{t\},a\_\{t\}\)\)\.

This critique phase emulates ”System 2” slow thinking in cognitive psychology\. If the predicted internal rewardRc​r​i​t​i​q​u​eR\_\{critique\}falls below a strictly defined hyperparameter thresholdτ\\tau, the candidate actionata\_\{t\}is outright rejected\. The Critic then generates a targeted natural language diagnostic explaining why the action was suboptimal \(e\.g\., ”The drawer is locked; attempting to open it without the key will fail\.”\)\. This feedback is appended to the Actor’s context window, forcing a regeneration of\(zt,at\)\(z\_\{t\},a\_\{t\}\)\. This internal loop iterates until a mathematically viable action is proposed \(up to a strict maximum of 3 regenerations per step, as enforced in the execution engine\), fundamentally suppressing hallucination rates before they can affect the external environment\.

3\. Execution Phase \(Environmental Transition\):Once an action successfully passes the Critique threshold \(Rc​r​i​t​i​q​u​e≥τR\_\{critique\}\\geq\\tau\), it is physically executed in the environment\. The environment then transitions according toTT, and returns a sparse objective rewardRe​n​vR\_\{env\}alongside a new observationot\+1o\_\{t\+1\}\. The overall expected return to be mathematically maximized by our policy combines the sparse, delayed environment reward with the dense, immediate internal critique reward:

J​\(πθ\)=𝔼τ∼πθ​\[∑t=0Hγt​\(Re​n​v​\(st,at\)\+λ​Rc​r​i​t​i​q​u​e​\(ot,at\)\)\]J\(\\pi\_\{\\theta\}\)=\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\}\}\\left\[\\sum\_\{t=0\}^\{H\}\\gamma^\{t\}\\left\(R\_\{env\}\(s\_\{t\},a\_\{t\}\)\+\\lambda R\_\{critique\}\(o\_\{t\},a\_\{t\}\)\\right\)\\right\]\(16\)whereτ\\taurepresents the entire sampled trajectory, and the hyperparameterλ∈\[0,1\]\\lambda\\in\[0,1\]carefully balances the influence of the internal critique mechanism against the environment’s absolute ground truth\. This architectural optimization \(Equation[16](https://arxiv.org/html/2607.17038#S3.E16)\) drastically reduces the occurrence of catastrophic actions and minimizes the sample complexity required to solve long\-horizon tasks\.

### 3\.4Experimental Implementation Details

To ensure strict reproducibility and to adhere to realistic local hardware constraints, our entire experimental framework is implemented in Python 3\.10 using the PyTorch 2\.0 deep learning library\.

The core cognitive engine \(the Actor\) utilizes the cutting\-edge, open\-source LLaMA\-3\-8B\-Instruct model\[[15](https://arxiv.org/html/2607.17038#bib.bib15)\]\. To facilitate efficient inference on a single consumer\-grade NVIDIA RTX 4090 GPU \(24GB VRAM\), the model weights are aggressively quantized to 4\-bit precision using the NF4 format provided by the BitsAndBytes library\. This quantization ensures that the massively parameterized LLM can fit entirely within GPU memory without incurring catastrophic swap overhead, thereby maintaining viable token\-generation throughput\.

For training the Critique Model, we leveraged Low\-Rank Adaptation \(LoRA\)\[[20](https://arxiv.org/html/2607.17038#bib.bib20)\]via the Hugging Face PEFT and TRL \(‘SFTTrainer‘\) libraries to fine\-tune themeta\-llama/Meta\-Llama\-3\-8B\-Instructbase model on a specialized dataset containing 50,000 reasoning and critique traces\. The LoRA parameters were configured with a rankr=16r=16, scale factorα=32\\alpha=32, and targeted projection matrices in the self\-attention layers:Wq,Wk,Wv,WoW\_\{q\},W\_\{k\},W\_\{v\},W\_\{o\}\. The learning rate was set to2×10−42\\times 10^\{\-4\}with a cosine annealing schedule down to1×10−51\\times 10^\{\-5\}, optimized via the AdamW optimizer with weight decay of 0\.01\. Training was performed for 3 epochs with an effective batch size of 64 \(micro batch size 4 and gradient accumulation steps of 4 across 4 GPUs\), utilizing DeepSpeed ZeRO\-2 optimization to fit within standard multi\-GPU setups\.

The inference pipeline is strictly managed through the Hugging Face Transformers library\. Environmental datasets are standardized into JSON format, and all multimodal observations are systematically tokenized using the LLaMA\-3 native Byte\-Pair Encoding \(BPE\) tokenizer\.

For the prompt design, the system prompt injected into the LLM Actor is structured as follows:

> You are an autonomous agent operating in a partially observable environment\. You are given a Goal\. At each step, analyze your history and memory, formulate a reasoning trace \(Thought\), and select a single action \(Action\) in JSON format: \{"thought": "\.\.\.", "action": "\.\.\."\}\. Do not repeat previous failed actions\.

The Critique Model receives the following prompt:

> Evaluate the proposed action against the observation history and the overarching goal\. Output a JSON containing: "logic\_check" \(boolean: true if action is physically possible\), "safety\_check" \(boolean: true if action does not cause infinite loops/redundant state transitions\), "critique\_score" \(float: 0\.0 to 1\.0 representing overall validity\), and "reason" \(string: explanation of critique score\)\.

Crucial hyperparameters were determined through extensive grid search during the preliminary phase of the project\. To balance exploration with logical determinism, the text generation temperature is strictly fixed atT=0\.2T=0\.2, while top\-p \(nucleus\) sampling is set to0\.900\.90\. The reinforcement learning discount factorγ\\gammais set to0\.950\.95to prioritize near\-term logical steps while still maintaining foresight\. The internal critique thresholdτ\\tauis established at0\.700\.70, and the reward balancing coefficientλ\\lambdais set to0\.50\.5\. Maximum context length is strictly truncated at 4,096 tokens; if an agent’s memory graph exceeds this limit, a specialized summarization sub\-routine aggressively compresses the oldest historical interactions\.

## 4Experiments and Results

### 4\.1Experimental Setup

To rigorously and empirically evaluate the proposed Reward\-Driven LLM Agent Workflow \(RLAW\) under standardized conditions, we established a comprehensive, tightly controlled experimental protocol\. To ensure that our findings are broadly applicable and not overfit to a single domain, we selected two widely recognized, challenging public benchmarks that require fundamentally different cognitive capabilities: ALFWorld and WebShop\.

Dataset 1: ALFWorld \(Embodied Simulation\):ALFWorld\[[3](https://arxiv.org/html/2607.17038#bib.bib3)\]is a complex, simulated embodied text environment built upon the continuous TextWorld engine\. It requires an autonomous agent to perform complex multi\-room navigation and intricate object manipulation to achieve high\-level, abstract household goals \(e\.g\., ”wash a dirty apple and place it on the kitchen table”\)\. The environment is inherently partially observable; the agent cannot see the contents of a drawer until it executes an ”open drawer” action\. This dataset specifically tests the agent’s capacity for spatial reasoning, long\-horizon planning, and structural memory retention\. The state spaceSSis highly dimensional, representing the physical coordinates and states of all objects, while the observation spaceΩ\\Omegaconsists of textual descriptions returned after each action\.

ALFWorld contains six distinct household task categories: Pick and Place, Clean and Place, Heat and Place, Cool and Place, Examine in Light, and Stack and Place\. Each task requires the agent to navigate through different rooms \(e\.g\., kitchen, bathroom, living room, bedroom\), locate relevant receptacles \(e\.g\., fridge, stove, sink, microwave, drawer\), retrieve objects \(e\.g\., apple, mug, key, paper\), perform a state\-transforming action \(e\.g\., washing an apple in the sink, heating a mug in the microwave\), and place it in the target location\. The average trajectory length required for a perfect oracle is approximately 12\.8 steps, making any deviation or error highly penalizing\. To simulate realistic deployment conditions, we split the ALFWorld scenarios into two sets: seen environments \(where the layout of the rooms is known, testing the agent’s capability\) and unseen environments \(where the layouts are completely novel, testing the agent’s generalization capability\)\. Our evaluation focuses strictly on the unseen environments to ensure rigorous validation of the agent’s cognitive flexibility\.

Dataset 2: WebShop \(Online Web Navigation\):To evaluate the agent’s ability to navigate dynamic, real\-world digital interfaces, we utilized WebShop\[[4](https://arxiv.org/html/2607.17038#bib.bib4)\]\. WebShop is an intricate, large\-scale web navigation benchmark containing over 1\.1 million real\-world products scraped from Amazon\. The agent is provided with a multi\-attribute user query \(e\.g\., ”Find me a red, noise\-canceling wireless headphone under $50”\) and must autonomously navigate through search bars, product category pages, and attribute selection menus to purchase the exact matching item\. WebShop introduces significant stochasticity \(e\.g\., search results may change and pagination is required\), making it an ideal testbed for evaluating the robustness of our internal Critique Module\.

The WebShop benchmark contains 1,180,810 real\-world products, 298,111 user search queries, and 4,120 human navigation trajectories\. WebShop models a browser\-based interaction where the agent has access to 5 basic actions:search\[query\],click\[button\],click\[link\],select\[attribute\], andbuy\. Button actions must target specific attributes like size, color, or model, which are dynamically populated based on search queries\. The rewardRe​n​vR\_\{env\}returned by the WebShop environment is a continuous value between 0\.0 and 1\.0, calculated as the overlap between the selected product attributes and the target query parameters\. A score of 1\.0 is awarded only if the agent purchases the exact item requested with all matching attributes\. Our testing was conducted over 500 tasks from the WebShop test split, requiring multi\-step search query formulation and attribute refinement\.

Evaluation Metrics:To objectively quantify agent performance across these diverse environments, we established a rigorous set of evaluation metrics\.

- •Task Success Rate \(SR %\):The primary, absolute metric\. It measures the percentage of evaluation episodes in which the agent successfully achieves the overarching goal within the maximum allowed step count \(H=50H=50\)\. Partial successes are scored as 0 to ensure strict evaluation\.
- •Average Steps \(AvgS\):The secondary metric, calculated strictly over successful episodes\. It indicates trajectory efficiency and planning quality\. A lower AvgS indicates that the agent navigated to the goal without redundant actions or infinite loops\.
- •Hallucination Error Rate \(%\):A custom metric developed for this study\. It tracks the percentage of episodes where the agent attempted to execute a mathematically or physically impossible action \(e\.g\., ”take apple from fridge” when the fridge is closed\)\.

Baseline Configurations:To ensure a fair, rigorous, and completely objective comparison, we benchmark our proposed RLAW architecture against two established classic models\. The first is a standard zero\-shot Base\-LLM \(LLaMA\-3\-8B\-Instruct\)\. This represents pure, isolated Language Intelligence without any recursive agentic loop; it simply predicts the entire sequence of actions in a single pass based on the initial observation\. The second baseline is the Standard ReAct\[[2](https://arxiv.org/html/2607.17038#bib.bib2)\]framework, using the same underlying 8B\-parameter model\. ReAct represents the current mainstream agent standard, interleaving Chain\-of\-Thought reasoning with environmental actions, but critically lacking our proposed POMDP routing mechanism and internal reward\-driven critique\. All models were evaluated over 500 randomly sampled, highly complex episodes from the unseen test splits of both datasets to guarantee statistical significance\.

### 4\.2Result Analysis

Main Experimental Results:The comparative data across 500 simulated evaluation episodes is presented in Table[4](https://arxiv.org/html/2607.17038#S4.T4)\.

Table 4:Performance comparison of the proposed intelligent workflow against baselines on ALFWorld and WebShop datasets\. Best results are in bold\.The empirical results, strictly quantified across 500 evaluation episodes, conclusively demonstrate that our proposed RLAW method delivers highly significant performance improvements over all baselines\. On the physically simulated ALFWorld benchmark, the RLAW architecture achieves an unprecedented 78\.6% absolute success rate\. This represents a massive 24\.5% absolute increase over the Standard ReAct baseline \(54\.1%\) and completely eclipses the zero\-shot Base\-LLM \(22\.4%\)\. Furthermore, the average number of steps required to successfully complete a task \(AvgS\) is dramatically reduced from 12\.1 in ReAct to just 9\.4 in RLAW\. This dual improvement—higher success combined with fewer steps—proves that the agent is not simply brute\-forcing solutions through infinite loops, but rather executing highly optimal, direct trajectories\.

Similarly, on the highly stochastic WebShop benchmark, RLAW achieves a 65\.8% success rate compared to ReAct’s 42\.3%\. The hallucination rate is perhaps the most striking metric: RLAW suppresses hallucinations down to 12\.4% in ALFWorld \(compared to the Base\-LLM’s catastrophic 45\.1%\)\. This improvement is directly attributable to the mathematical rigor of the Critique mechanism, which serves as a logical filter, preventing the agent from exploring redundant or physically impossible subtrees in the state space\.

The results shown in Table[4](https://arxiv.org/html/2607.17038#S4.T4)illustrate a clear hierarchy of performance across the three configurations\. The Base\-LLM \(Zero\-Shot\) model performs poorly on both benchmarks\. On ALFWorld, it achieves a Success Rate of only 22\.4%, and on WebShop, 18\.6%\. Because the model must output the entire trajectory in a single zero\-shot pass, it cannot adapt to environmental feedback\. It frequently generates actions that target non\-existent objects or assumes state transitions that have not occurred, leading to an extremely high Hallucination Error Rate \(45\.1% on ALFWorld and 48\.3% on WebShop\)\. The Standard ReAct framework significantly improves upon the Base\-LLM by introducing a step\-by\-step reasoning\-action loop\. It achieves a 54\.1% success rate on ALFWorld and 42\.3% on WebShop\. However, because ReAct lacks an internal validation mechanism, it is highly sensitive to early errors\. Once the model generates a flawed reasoning trace or attempts an invalid action, the subsequent environment observations \(often containing error messages like ‘Nothing happens‘ or ‘Invalid command‘\) are appended to the context\. Standard LLMs often fail to recover from these error messages, instead falling into repetitive loops \(e\.g\., repeatedly executing ‘open cabinet‘ when it is already open\) or generating redundant search queries\.

Our proposed RLAW architecture successfully overcomes these issues\. The key to this success is the dramatic reduction in the Hallucination Rate, which drops to 12\.4% on ALFWorld and 14\.1% on WebShop\. By filtering out invalid candidate actions during the Critique Phase, the agent only interacts with the environment using highly viable trajectories\. This prevents the environment history from becoming cluttered with error messages, allowing the self\-attention mechanism to focus on high\-quality reasoning traces\.

Ablation Experiment:While the holistic performance of RLAW is highly superior, rigorous academic analysis requires us to isolate and verify the independent, marginal contribution of our individual improved modules\. To achieve this, we conducted a systematic ablation study\. We established two degraded configurations: \(1\) removing the Critique Module entirely \(which effectively reverts the execution mechanism to a standard open\-loop system where actions are trusted implicitly\) and \(2\) removing the Graph Memory Module \(which reverts the agent to utilizing a standard, highly volatile FIFO context window\)\.

Table 5:Ablation study of RLAW components on the ALFWorld dataset, detailing the impact on success rate, efficiency, and rejection frequency\.The ablation results, detailed in Table[5](https://arxiv.org/html/2607.17038#S4.T5), provide mathematical confirmation of our core hypothesis: the self\-correcting Critique Module is the primary and indispensable contributor to the observed performance gains\. Removing the Critique Module causes a massive performance regression, dropping the success rate from 78\.6% down to 61\.2% \(an absolute penalty of 17\.4%\)\. Furthermore, without the critic, the average step count inflates to 11\.5, indicating that the agent wastes significant time recovering from its own unchecked hallucinations\.

The data explicitly tracks ”Critique Rejections”—the average number of times per episode the internal reward model vetoed a proposed action\. At 2\.1 rejections per successful episode, the data proves that even a highly capable 8B model natively proposes fatal actions approximately twice per task; without the critic intercepting these, failure is almost guaranteed\. Removing the Graph Memory Module also imposes a severe penalty \(success drops to 68\.4%\), as the agent frequently ”forgets” the state of previously explored rooms, forcing it to backtrack recursively\. Together, these ablations highlight the absolute necessity of integrating internal reward mechanisms and structured memory within generative Language Intelligence models\.

To qualitatively illustrate the differences between Standard ReAct and the proposed RLAW, we conduct a detailed case study of an ALFWorld task instance\. The goal is:’find a clean sponge and place it on the vanity table’\. In the Standard ReAct run, the agent successfully navigates to the bathroom and locates the sink\. However, it proposes the actiontake sponge from sinkbefore washing it\. The environment returns:The sponge is dirty\. The agent then attemptsplace sponge on vanity, forgetting that the task required acleansponge\. The agent has failed the task\. In contrast, the RLAW execution follows a different trajectory\. At step 4, the Actor proposestake sponge from sink\. The Critique Module intercepts this proposal, evaluates it against the goal \(which requires acleansponge\), and notes that the sponge is currently dirty\. The Critic returns a score of 0\.35 \(FAIL\) and outputs the feedback:’The sponge is dirty\. You must turn on the faucet to clean the sponge before taking it\.’The Actor receives this feedback, updates its reasoning trace, and generates the corrected action:clean sponge with faucet\. After execution, the agent successfully cleans the sponge, takes it, and places it on the vanity table, achieving task success\.

Table 6:Inference latency, hardware resource utilization, and efficiency metrics per reasoning step on an RTX 4090 GPU\.While accuracy and success rate are paramount, industrial deployment of Agent Intelligence requires strict adherence to hardware resource constraints\. Table[6](https://arxiv.org/html/2607.17038#S4.T6)presents a detailed profiling of inference latency, VRAM utilization, hardware power draw, and context token consumption per reasoning step, benchmarked on a standard NVIDIA RTX 4090 GPU\.

Because the RLAW architecture enforces a ”System 2” dual\-loop \(propose, critique, potentially regenerate, execute\), it inherently incurs a computational overhead\. The average latency per step for RLAW is 2\.15 seconds, representing a roughly1\.8×1\.8\\timesslowdown compared to the Standard ReAct baseline \(1\.20s\)\. Correspondingly, VRAM usage marginally increases to 9\.4 GB, and the active power draw peaks at 210W due to the sustained utilization of the tensor cores during the continuous critique regeneration loops\. Furthermore, by maintaining a persistent Graph Memory structure alongside the full critique history, RLAW consumes an average of 3,500 tokens per context window, approaching the base model’s architectural limits\. However, this increased latency and resource cost is an entirely acceptable, highly strategic trade\-off\. In complex, real\-world deployment scenarios \(e\.g\., executing financial transactions, modifying system databases, or controlling physical robotics\), a slower but guaranteed\-safe action \(via the 2\.15s RLAW step\) is infinitely superior to a fast but hallucinated action \(via the 1\.20s ReAct step\) that could irreparably corrupt the environment\.

### 4\.3Visualizations and Learning Dynamics

To provide a deeper, dynamic understanding of the learning and execution process over time, we present a series of detailed visualizations capturing reward convergence, hallucination reduction, and parameter scaling laws\.

05510101515202025253030353540404545505002020404060608080Training/Evaluation EpisodesCumulative Expected RewardAgent Reward Convergence AnalysisBase\-LLM \(Zero\-Shot\)Standard ReActRLAW \(w/o Memory\)RLAW \(Ours Full\)Figure 2:Cumulative reward curves across 50 episodes\. The inclusion of the Critique Module and Graph Memory accelerates convergence and prevents the early plateauing seen in ReAct and Base models\.Figure[2](https://arxiv.org/html/2607.17038#S4.F2)shows the cumulative reward curves across 50 episodes\. The inclusion of the Critique Module and Graph Memory accelerates convergence and prevents the early plateauing seen in ReAct and Base models\. Additionally, Figure[3](https://arxiv.org/html/2607.17038#S4.F3)details the ablation on perception modalities across different datasets\.

Text OnlyVision OnlyText\+VisionRLAW \(Full\)020204040606045\.145\.141\.241\.226\.526\.512\.412\.448\.348\.343\.843\.829\.129\.114\.114\.1Hallucination Error Rate \(%\)Cross\-Dataset Hallucination Reduction via Multimodal FusionALFWorldWebShopFigure 3:Detailed ablation on perception modalities across two distinct datasets\. The grouped bar chart demonstrates that integratingMultimodal Intelligenceconsistently reduces hallucination errors\. The full RLAW architecture, leveraging internal critique, achieves the lowest hallucination rate in both physical simulation \(ALFWorld\) and web navigation \(WebShop\) tasks\.11338813133030707002020404060608080100100ReAct 70B Baseline \(75%\)Model Parameter Size \(Billions\)Task Success Rate \(%\)Scaling Laws: Model Parameters vs\. Success RateBase\-LLM \(Zero\-Shot\)Standard ReActRLAW \(Ours\)Figure 4:Scaling behavior of agent architectures on the ALFWorld dataset\. RLAW not only scales favorably but exhibits exceptional parameter efficiency: an 8B parameter RLAW model \(78\.6%\) explicitly outperforms the standard ReAct architecture running on a massive 70B parameter model \(75%\)\. Zero\-shot capabilities plateau early, highlighting the necessity of agentic loops\.

## 5Discussion

### 5\.1In\-depth Result Interpretation

Applying advanced theories drawn symmetrically from Language Intelligence and Reinforcement Learning, we can rigorously analyze the exact mathematical and cognitive mechanisms behind the observed empirical results\. The primary reason our RLAW architecture drastically outperforms standard LLMs is its systematic resolution of ”short\-sighted autoregressive generation\.” Standard generative models decode text purely by maximizing the immediate conditional probabilityP​\(wt∣w<t\)P\(w\_\{t\}\\mid w\_\{<t\}\)\. In complex environments, this greedy strategy frequently leads to locally optimal but globally catastrophic actions\.

By formally embedding the generative process within a mathematically bounded POMDP framework, and by heavily utilizing the self\-correcting Critique mechanism, we force the generative agent to actively consider the long\-term, delayed viability of its actions\. The internal Critique mechanism essentially functions as an approximated, learned Value FunctionVπ​\(s\)V^\{\\pi\}\(s\)derived from the Bellman equations\. Rather than blindly executing an action that merely sounds grammatically correct, the agent explicitly prunes logically impossible, dead\-end trajectories before they materialize in the external environment\. As visually confirmed in Figure[4](https://arxiv.org/html/2607.17038#S4.F4), this paradigm shift completely rewrites established scaling laws\. Our 8B parameter RLAW model explicitly outperforms a massive 70B parameter standard ReAct model, proving conclusively that architectural ingenuity and formal self\-correction are significantly more critical to Agent Intelligence than mere raw parameter count\.

A key theoretical insight from our experiments is the role of the Critique Module in mimicking human metacognition\. When humans solve complex problems, they do not simply output a stream of consciousness; they internally simulate and critique candidate actions before speaking or acting\. RLAW replicates this ’System 2’ cognitive process by separating action proposal \(Actor\) from action evaluation \(Critic\)\. This separation allows the Actor to remain creative and exploratory, while the Critic ensures safety and goal alignment\. This division of labor is highly effective in constrained environments where the cost of error is high\.

### 5\.2Research Advantages and Limitations

This research contributes several key methodological advantages to the field of Agent Intelligence\. Primarily, it successfully bridges the historically separate domains of symbolic, multi\-step logical planning with continuous, neural text generation\. It establishes a highly reliable, deterministic fail\-safe mechanism within a notoriously stochastic generative process\.

However, despite these profound architectural advantages, rigorous scientific inquiry demands a transparent acknowledgment of the inherent limitations of the current methodology\. The most prominent limitation is the substantially increased inference latency\. Because the agent is computationally required to generate, mathematically critique, and frequently regenerate actions before environmental execution, the system incurs roughly a 1\.8x to 2\.5x increase in computational overhead per temporal step compared to the baseline ReAct framework\. While this is highly acceptable for asynchronous, high\-stakes offline tasks, it currently precludes the use of RLAW in real\-time, low\-latency control systems \(such as high\-speed drone navigation\)\.

Additionally, the underlying Language Intelligence engine remains fundamentally constrained by the Transformer’s maximum context window limit\. Even with aggressive summarization sub\-routines, maintaining the entire Graph Memory structure and the extensive history of critique rejections results in severe ”attention dilution\.” Over extremely long episodic horizons \(e\.g\.,H\>100H\>100steps\), the agent occasionally exhibits a ”lost in the middle” phenomenon, in which it fails to effectively retrieve crucial constraints established early in the episode, leading to an eventual breakdown of the POMDP assumption\. Furthermore, the system depends heavily on the quality of the Critique Model itself\. If the Critic is under\-trained or suffers from its own hallucinations, it may reject valid actions \(false positives\) or approve invalid actions \(false negatives\)\. In our experiments, we observed that a weak Critic can cause the agent to become stuck in infinite regeneration loops, in which the Actor continuously proposes actions that the Critic rejects without providing useful feedback\. Future work should focus on training more robust, self\-calibrating Critique models that dynamically adjust their thresholds based on task difficulty\.

### 5\.3Design Insights and Lessons Learned

The process of designing, implementing, and theoretically justifying RLAW yields several critical engineering insights\. Initially, the main technical challenge was bridging the disconnect between the deterministic dynamics of Reinforcement Learning and the highly stochastic text outputs of Generative Language Models\. Resolving this required structured JSON Schema enforcement and constrained decoding to ensure strict state\-action space mapping\. Furthermore, we observed that separate Actor\-Critic modules partition creative exploration from critical logic verification, which mimics cognitive dual\-process theory\.

Looking forward, future iterations of this research must aggressively push towards full\-spectrum, deep integration of Multimodal Intelligence\. By actively processing real\-time, continuous visual streams \(leveraging Visual Intelligence\) seamlessly alongside textual state descriptions, and utilizing advanced Graph Intelligence networks to explicitly map complex 3D spatial relationships, we can transcend text\-based simulations and facilitate true, embodied Agent Intelligence\. Such intricately synthesized, multi\-module technologies hold massive, transformative potential for complex industrial applications, ranging from autonomous robotics and smart manufacturing to sophisticated, self\-correcting software engineering assistants\.

## 6Conclusion

This extensive academic study proposed, rigorously implemented, and empirically evaluated a highly optimized, Reward\-Driven LLM Agent Workflow \(RLAW\), grounded in the theoretical framework of POMDP routing and reinforcement learning\. By fundamentally combining the generative cognitive capabilities of Language Intelligence with the mathematically formal, self\-correcting reward mechanisms drawn from Reinforcement Intelligence, this research directly addressed and mitigated the most critical limitations of mainstream large language models—specifically, their propensity for cascading hallucinations, ungrounded actions, and severely short\-sighted planning trajectories\.

Through exhaustive quantitative evaluations across both the physically simulated ALFWorld environment and the highly stochastic WebShop digital benchmark, the proposed architecture demonstrated a massive 24\.5% absolute improvement in task success rate while simultaneously enhancing trajectory efficiency by reducing unnecessary environmental interactions\. Comprehensive ablation studies definitively confirmed that the internal, ”System 2” self\-correction critique module is the primary driver of these remarkable performance gains, proving that structural verification is inherently superior to blind generative scaling\. Ultimately, this research successfully translates these foundational modules into a highly cohesive, practical, and parameter\-efficient artificial intelligence implementation\. The resulting RLAW architecture not only establishes a highly robust, parameter\-efficient basis for current digital agents but also provides a vital theoretical stepping stone toward deploying highly reliable, fully autonomous systems in complex real\-world environments\.

## References

- \[1\]Vaswani A, Shazeer N, Parmar N, et al\. Attention is all you need \[C\]//Advances in Neural Information Processing Systems \(NeurIPS\), 2017\.
- \[2\]Yao S, Zhao J, Yu D, et al\. ReAct: Synergizing reasoning and acting in language models \[C\]//International Conference on Learning Representations \(ICLR\), 2023\.
- \[3\]Shridhar M, Yuan X, Côté M A, et al\. ALFWorld: Aligning text and embodied environments for interactive learning \[C\]//International Conference on Learning Representations \(ICLR\), 2021\.
- \[4\]Yao S, Chen H, Yang J, et al\. WebShop: Towards scalable real\-world web interaction with grounded language agents \[C\]//Advances in Neural Information Processing Systems \(NeurIPS\), 2022\.
- \[5\]Sutton R S, Barto A G\. Reinforcement learning: An introduction \[M\]\. MIT Press, 2018\.
- \[6\]Wei J, Wang X, Schuurmans D, et al\. Chain\-of\-thought prompting elicits reasoning in large language models \[C\]//Advances in Neural Information Processing Systems \(NeurIPS\), 2022\.
- \[7\]Goodfellow I, Bengio Y, Courville A\. Deep Learning \[M\]\. MIT Press, 2016\.
- \[8\]Brown T, Mann B, Ryder N, et al\. Language models are few\-shot learners \[C\]//Advances in Neural Information Processing Systems \(NeurIPS\), 2020\.
- \[9\]Kipf T N, Welling M\. Semi\-supervised classification with graph convolutional networks \[C\]//International Conference on Learning Representations \(ICLR\), 2017\.
- \[10\]Dosovitskiy A, Beyer L, Kolesnikov A, et al\. An image is worth 16x16 words: Transformers for image recognition at scale \[C\]//International Conference on Learning Representations \(ICLR\), 2021\.
- \[11\]Schick T, Dwivedi\-Yu J, Dessì R, et al\. Toolformer: Language models can teach themselves to use tools \[C\]//Advances in Neural Information Processing Systems \(NeurIPS\), 2023\.
- \[12\]Shen Y, Song K, Tan X, et al\. HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face \[C\]//Advances in Neural Information Processing Systems \(NeurIPS\), 2023\.
- \[13\]Shinn N, Cassano F, Gopinath A, et al\. Reflexion: Language agents with verbal reinforcement learning \[C\]//Advances in Neural Information Processing Systems \(NeurIPS\), 2023\.
- \[14\]Kaelbling L P, Littman M L, Cassandra A R\. Planning and acting in partially observable stochastic domains \[J\]\. Artificial Intelligence, 1998, 101\(1\-2\): 99\-134\.
- \[15\]AI M\. Llama 3 model card \[Z\]\. Meta AI, 2024\.
- \[16\]Ouyang L, Wu J, Jiang X, et al\. Training language models to follow instructions with human feedback \[C\]//Advances in Neural Information Processing Systems \(NeurIPS\), 2022\.
- \[17\]Richards T\. Auto\-GPT: An autonomous GPT\-4 experiment \[Z\]\. GitHub repository, 2023\.
- \[18\]Yao S, Yu D, Zhao J, et al\. Tree of thoughts: Deliberate problem solving with large language models \[C\]//Advances in Neural Information Processing Systems \(NeurIPS\), 2024\.
- \[19\]Besta M, Blach N, Kubicek A, et al\. Graph of thoughts: Solving elaborate problems with large language models \[C\]//Proceedings of the AAAI Conference on Artificial Intelligence, 2024\.
- \[20\]Hu E J, Shen Y, Wallis P, et al\. LoRA: Low\-rank adaptation of large language models \[C\]//International Conference on Learning Representations \(ICLR\), 2022\.

## Appendix AAppendix: Experimental Code Snippets and Auxiliary Data

### A\.1Reward\-Driven Prompt Structure

To enforce the POMDP\-based reasoning loop, the LLM is prompted with a structured JSON format to separate thinking and acting\.

Algorithm 1System Prompt for RLAW Critique Phase1:Input:Current Observation

oto\_\{t\}, Proposed Action

ata\_\{t\}
2:Instruction:Evaluate if

ata\_\{t\}contributes to the overall goal based on environmental constraints\.

3:Output Format:JSON with keys"logic\_check"and"safety\_check"\(booleans\),

4:"critique\_score"\(0\.0 to 1\.0\), and"decision"\(PASS or FAIL\)\.

5:ifcritique\_score

<0\.7<0\.7then

6:Return FAIL and provide natural language regeneration feedback\.

7:else

8:Return PASS to execute the proposed action\.

9:endif