AI-Driven Digital Twin Framework for Warehouse Logistics Automation Optimization

# AI-Driven Digital Twin Framework for Warehouse Logistics Automation Optimization
Source: [https://arxiv.org/html/2610.38101](https://arxiv.org/html/2610.38101)
## AI\-Driven Digital Twin Framework for Warehouse Logistics Automation OptimizationThanks:\*Equal Contribution

‡Md Mahinur Alam\*[https://orcid.org/0009-0006-5453-443X](https://orcid.org/0009-0006-5453-443X),†Kanita Jerin Tanha\*[https://orcid.org/0009-0006-5453-443X](https://orcid.org/0009-0006-5453-443X),§Md Tanjin Alam[https://orcid.org/](https://orcid.org/),φShadman Sakib[https://orcid.org/0009-0006-5453-443X](https://orcid.org/0009-0006-5453-443X),‡Min Gyu Kim[https://orcid.org/](https://orcid.org/)Affiliation:†Kumoh National Institute of Technology, Gumi, South Korea, ‡Georgia Southern University, GA, USA, φUniversity of Maryland, Baltimore County, MD, USA,§VosuAI, kanitajerin17@kumoh\.ac\.kr, piash\.tanjin@gmail\.com, \(malam4, ssakib1\)@umbc\.edu, mkim@georgiasouthern\.eduAffiliation:

###### Abstract

In recent years, warehouse logistics optimization has increasingly focused on intelligent, data\-driven systems to enhance operational efficiency\. Conventional static product placement strategies are being progressively replaced by dynamic slotting approaches that can adapt to real\-time variations in warehouse conditions\. In this work, we present a novel framework that employs a reinforcement learning agent integrated within a FlexSim\-based Digital Twin environment for real\-time product placement optimization\. The proposed agent continuously learns from streaming operational data, enabling it to reduce picking time while improving overall system throughput\. The simulation environment models a four\-zone warehouse layout with forklift\-based material handling and leverages the Asset Administration Shell framework to ensure seamless interaction between the learning agent, physical asset, and the digital twin\. Experimental evaluation demonstrates substantial performance gains, with the proposed approach achieving a 29\.66% reduction in average picking time, 35\.42% increase in throughput, and 18\.54% decrease in travel distance\.

###### Index Terms:

Automation, digital twin, reinforcement learning \(RL\), warehouse logistics\.

## IIntroduction

Warehouse logistics increasingly face volatile demand, labor shortages, and tight service\-level constraints, making static slotting and rule\-based placement strategies inadequate for sustaining low picking time and high throughput\. Digital twin \(DT\) technology addresses this limitation by providing a high\-fidelity, continuously updatable virtual replica of warehouse operations that supports real\-time monitoring and safe evaluation of alternative decisions without interrupting physical workflows\. To ensure interoperable and consistent interactions across heterogeneous assets \(e\.g\., racks, transporters, conveyors\), the Asset Administration Shell \(AAS\)\[[1](https://arxiv.org/html/2610.38101#bib.bib2)\]further standardizes asset identity, state, and control interfaces, enabling reliable data exchange between the DT environment and intelligent optimization modules\[[2](https://arxiv.org/html/2610.38101#bib.bib1)\]\.

![Refer to caption](https://arxiv.org/html/2610.38101v1/front_page.png)Fig\. 1:Proposed AI\-Driven Digital Twin Framework for Smart Warehouse Logistics, Physical\-to\-Digital Synchronization via Asset Administration Shell with Reinforcement Learning Based Dynamic Reslotting Optimization\.Dynamic warehouse optimization has been explored through operations research, rule\-based heuristics, and learning\-based methods\[[3](https://arxiv.org/html/2610.38101#bib.bib4)\]\. Classical approaches include MILP\-based hierarchical frameworks for multi\-AGV scheduling, which achieve notable reductions in task completion time, but depend on static formulations that limit real\-time adaptability\[[4](https://arxiv.org/html/2610.38101#bib.bib9)\]\. Similarly, genetic\-algorithm variants such as ARGA\[[5](https://arxiv.org/html/2610.38101#bib.bib10)\]and multi\-agent pathfinding decomposition methods\[[6](https://arxiv.org/html/2610.38101#bib.bib11)\]offer strong coordination capabilities, but incur significant computational overhead and do not adapt dynamically to evolving warehouse conditions\.

Reinforcement learning \(RL\) has gained prominence as a more adaptive alternative for storage assignment and routing\[[7](https://arxiv.org/html/2610.38101#bib.bib8)\]\. Proximal Policy Optimization \(PPO\) has demonstrated a 6\.3% reduction in transport costs through adaptive slot selection, outperforming classical ABC heuristics\[[8](https://arxiv.org/html/2610.38101#bib.bib5)\]\. Deep Q\-Network \(DQN\) methods have been applied to task assignment and shelf reallocation, reducing travel time and improving throughput over heuristic\-based planning\[[9](https://arxiv.org/html/2610.38101#bib.bib6)\]\. Other deep RL approaches using The Asprilo environment has minimized travel distance, outperforming GA, ACO, and VRP heuristics by 1\.76% across various warehouse layouts\[[10](https://arxiv.org/html/2610.38101#bib.bib7)\]\. Despite this advances, most RL\-based solutions operate in isolated simulation environments without integration into high\-fidelity digital twin systems, limiting their ability to continuously reflect real warehouse states and hinder practical deployment\.

To bridge this gap, we present an integrated framework that couples a DQN\-based RL agent with a FlexSim\-based Digital Twin through an AAS middleware layer, enabling real\-time perception–action loops for dynamic slotting optimization\. The key contributions of this paper are:\(1\)We propose a DQN\-based dynamic slotting framework that learns demand\-aware zone assignment policies from real\-time warehouse states, integrating action masking, experience replay, andϵ\\epsilon\-greedy exploration to jointly optimize picking time, travel distance, and throughput\.\(2\)We design a hybrid CNN–BiLSTM–Attention Q\-network architecture that captures spatial feature correlations and temporal dependencies in warehouse state sequences, and validate its effectiveness through ablation studies against alternative architectures\.\(3\)We implement a high\-fidelity FlexSim\-based Digital Twin integrated with the AAS, enabling standardized asset representation, seamless bidirectional data exchange, and safe shadow evaluation of slotting policies prior to physical deployment\.

![Refer to caption](https://arxiv.org/html/2610.38101v1/Figures/Simulation_digital_twin.png)Fig\. 2:Digital Twin Warehouse Logistics Simulation in FlexSim Four Zone Layout with Forklift\-Based Product Handling\.
## IIProposed Framework

The proposed framework tightly couples a reinforcement learning agent with a FlexSim\-based Digital Twin through an Asset Administration Shell middleware layer, as illustrated in Fig\.[1](https://arxiv.org/html/2610.38101#S1.F1)\. IoT\-instrumented physical warehouse assets are mirrored digitally via AAS, which standardizes asset identification, state representation, and control interfaces, ensuring bidirectional synchronization between the physical facility and its simulated counterpart\. The RL agent operates exclusively within the DT environment, issuing dynamic slotting decisions that are evaluated in simulation before deployment as adaptive reslotting plans\.

### II\-ARL Agent Formulation

At each decision steptt, the warehouse state is encoded asst=\[order\_id,rack\_num,pick\_time,…,stock\_level\]s\_\{t\}=\[\\text\{order\\\_id\},\\ \\text\{rack\\\_num\},\\ \\text\{pick\\\_time\},\\ \\ldots,\\ \\text\{stock\\\_level\}\], and the agent selects a zone assignmentat∈\{A,B,C,D\}a\_\{t\}\\in\\\{A,B,C,D\\\}for a candidate item\. Invalid actions are masked at each step by setting their Q\-values to−∞\-\\inftybefore action selection, preventing assignments to zones that have reached rack capacity or whose current stock levels exceed a predefined threshold, thereby constraining exploration to operationally feasible placements\. The scalar rewardrt=Δ​Tsaved−λ⋅Cmove\+μ⋅Δ​KPIr\_\{t\}=\\Delta T\_\{\\text\{saved\}\}\-\\lambda\\cdot C\_\{\\text\{move\}\}\+\\mu\\cdot\\Delta\\text\{KPI\}balances picking time improvementΔ​Tsaved\\Delta T\_\{\\text\{saved\}\}, relocation costCmoveC\_\{\\text\{move\}\}, and temporal change in zone\-level performanceΔ​KPI=KPIzone​\(t\+1\)−KPIzone​\(t\)\\Delta\\text\{KPI\}=\\text\{KPI\}\_\{\\text\{zone\}\}\(t\{\+\}1\)\-\\text\{KPI\}\_\{\\text\{zone\}\}\(t\), whereKPIitem=∑iwi​x~i\\text\{KPI\}\_\{\\text\{item\}\}=\\sum\_\{i\}w\_\{i\}\\tilde\{x\}\_\{i\}aggregates min\-max normalized fulfillment rate \(w1=0\.35w\_\{1\}\{=\}0\.35\), stockout rate \(w2=0\.25w\_\{2\}\{=\}0\.25\), turnover \(w3=0\.25w\_\{3\}\{=\}0\.25\), and holding cost \(w4=0\.15w\_\{4\}\{=\}0\.15\)\. We setλ=0\.3\\lambda\{=\}0\.3andμ=0\.2\\mu\{=\}0\.2via grid search over\{0\.1,0\.2,0\.3,0\.5\}\\\{0\.1,0\.2,0\.3,0\.5\\\}, prioritizing picking time as the dominant signal\. The agent remains robust across moderate ranges; performance degrades only when relocation penalty dominates \(λ=0\.5\\lambda\{=\}0\.5, restricting beneficial moves\) or KPI weight is excessive \(μ=0\.5\\mu\{=\}0\.5, causing unstable early training from noisyΔ​KPI\\Delta\\text\{KPI\}estimates\)\.

The agent maximizes the expected discounted returnGt=𝔼⁡\[∑k=0∞γk​rt\+k\]G\_\{t\}=\\mathbb\{E\}\\\!\\left\[\\sum\_\{k=0\}^\{\\infty\}\\gamma^\{k\}r\_\{t\+k\}\\right\]via a parameterized action\-value networkQ⁡\(s,a,θ\)Q\(s,a;\\theta\)\. A target networkQt​\(s,a,θ−\)Q\_\{t\}\(s,a;\\theta^\{\-\}\)is periodically synchronized to stabilize training\. For each minibatch\{\(sj,aj,rj,sj\+1\)\}j=1N\\\{\(s\_\{j\},a\_\{j\},r\_\{j\},s\_\{j\+1\}\)\\\}\_\{j=1\}^\{N\}, the temporal difference target isyj=rjy\_\{j\}=r\_\{j\}ifsj\+1s\_\{j\+1\}is terminal andyj=rj\+γ​Qt​\(sj\+1,a′,θ−\)y\_\{j\}=r\_\{j\}\+\\gamma\\,Q\_\{t\}\(s\_\{j\+1\},a^\{\\prime\};\\theta^\{\-\}\)otherwise, with mean\-squared lossℒ⁡\(θ\)=1N​∑j=1N\(yj−Q⁡\(sj,aj,θ\)\)2\\mathcal\{L\}\(\\theta\)=\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}\(y\_\{j\}\-Q\(s\_\{j\},a\_\{j\};\\theta\)\)^\{2\}\. Training uses Adam \(η=3×10−4\\eta\{=\}3\{\\times\}10^\{\-4\},γ=0\.99\\gamma\{=\}0\.99, batch size 64\), a10510^\{5\}\-transition replay buffer with 2,000\-step warm\-up, hard target updates everyτ=1000\\tau\{=\}1000steps, andε\\varepsilon\-greedy exploration decaying linearly from 1\.0 to 0\.05 over 50,000 steps\. Gradient clipping \(norm=10\{=\}10\) and weight decay \(10−510^\{\-5\}\) ensure stability across 2,000 episodes\.

### II\-BDeep Q\-Network

The Q\-network employs a hybrid CNN–BiLSTM–Attention architecture designed to capture both spatial feature correlations across warehouse state variables and long\-range temporal dependencies across decision steps\. The statests\_\{t\}is first passed through a 1D\-CNN, where each layer computesxi\(n\)=ReLU⁡\(Wn⋅xi−1\+bn\)x\_\{i\}^\{\(n\)\}=\\mathrm\{ReLU\}\(W\_\{n\}\\cdot x\_\{i\-1\}\+b\_\{n\}\), to extract local spatial patterns among co\-located features such as rack occupancy, stock levels, and picking frequency\. The resulting feature maps are fed into a BiLSTM, producing forward and backward hidden representationsH→t\\overrightarrow\{H\}\_\{t\}andH←t\\overleftarrow\{H\}\_\{t\}that encode both past operational trends and anticipated future demand signals with standard gating mechanisms \(ftf\_\{t\},iti\_\{t\},CtC\_\{t\},oto\_\{t\}\)\.

To enable the agent to focus on the most operationally critical time steps, an additive Bahdanau attention mechanism\[[11](https://arxiv.org/html/2610.38101#bib.bib3)\]computes alignment scoresei=vT​tanh⁡\(Wa​hi\+ba\)e\_\{i\}=v^\{T\}\\tanh\(W\_\{a\}h\_\{i\}\+b\_\{a\}\), normalized via softmax to obtain attention weightsαi=exp⁡\(ei\)/∑k=1Texp⁡\(ek\)\\alpha\_\{i\}=\\exp\(e\_\{i\}\)/\\sum\_\{k=1\}^\{T\}\\exp\(e\_\{k\}\)\. The context vectorc=∑i=1Tαi​hic=\\sum\_\{i=1\}^\{T\}\\alpha\_\{i\}h\_\{i\}is then projected through a fully connected layer to produce Q\-valuesQ⁡\(st,a,θ\)Q\(s\_\{t\},a;\\theta\)for all four zone actions\. This selective attention allows the network to weight warehouse state transitions where demand shifts or congestion events most strongly influence slotting decisions, directly supporting the framework’s objective of demand\-aware zone assignment\.

### II\-CDigital Twin Closed\-Loop Integration

As depicted in Fig\.[1](https://arxiv.org/html/2610.38101#S1.F1), the agent\-environment interaction follows a structured closed loop: the DQN agent issues a zone\-assignment actionata\_\{t\}, which the action executor translates into FlexSim simulator commands via AAS\. FlexSim advances the epoch and returns operational traces \(picking durations, travel distances, queue states\), from which the reward module computesrtr\_\{t\}and the next statest\+1s\_\{t\+1\}\. The transition tuple\(st,at,rt,st\+1\)\(s\_\{t\},a\_\{t\},r\_\{t\},s\_\{t\+1\}\)is stored in the response buffer for minibatch sampling and network updates\. This perception decision execution loop supports online iterative policy refinement in a high\-fidelity environment, while the AAS layer permits shadow evaluation of candidate slotting policies before any physical deployment\.

TABLE I:FlexSim Digital Twin Warehouse Simulation Asset Configuration Parameters\.![Refer to caption](https://arxiv.org/html/2610.38101v1/Figures/abi2.png)Fig\. 3:Ablation study comparing average picking time across Q\-network architecture variants\.![Refer to caption](https://arxiv.org/html/2610.38101v1/result_all.png)Fig\. 4:FlexSim dashboard comparison between the proposed framework DQN\-Agent and Static Slotting baseline across per\-transporter travel distance, utilization state distribution, and warehouse throughput\.TABLE II:Performance comparison averaged over 5 independent runs with different random seeds\.
### II\-DWarehouse Logistics Digital Twin Environment

The warehouse environment is instantiated as a high\-fidelity 3D discrete\-event simulation in FlexSim, serving as the interactive execution substrate for the DQN agent\. The simulated facility encompasses four storage zones \(A–D\), each equipped with four double\-sided racks, yielding 16 racks total, accommodating a 500\-SKU product catalog\. Zones are connected through navigable aisle networks that govern transporter routing and forklift trajectories, as visualized in the digital twin layout Fig\.[2](https://arxiv.org/html/2610.38101#S1.F2)\. Product flow is managed by four autonomous transporters: two Picker units \(v = 1\.25 m/s, a = 1\.75 m/s²\) responsible for retrieval to delivery movement, and two Placer units \(v = 1\.0 m/s, a = 1\.25 m/s²\) handling replenishment operations\. Two inbound and two outbound queues \(capacity: 100 units each\) buffer product flows at the zone interfaces, while dedicated conveyors operating at 1\.75 m/s inbound and 1\.25 m/s outbound link storage areas to the dispatch sink\. Two human operator agents and one sink node complete the material handling circuit, as summarized in Table[I](https://arxiv.org/html/2610.38101#S2.T1)\. Each simulation epoch is triggered by the DQN agent’s action executor, which translates zone assignment decisions into FlexSim object commands via the AAS interface layer\. Upon epoch completion, the simulator emits operational traces picking durations, transporter trajectories, and queue states that are consumed by the reward computation module to close the agent\-environment loop\. This architecture enables safe shadow evaluation of slotting policies prior to any physical deployment, while maintaining continuous synchronization between the agent’s learned model and the evolving simulated state\.

### II\-EExperimental Setup

All experiments are conducted on a workstation equipped with an Intel Core i9\-13900K CPU, 64 GB RAM, and an NVIDIA RTX 4090 GPU\. The digital twin environment runs on FlexSim 25\.1, with the DQN agent implemented in Python 3\.10 using PyTorch 2\.1\. The agent is trained for 2,000 episodes with each episode comprising a full order cycle across the four\-zone warehouse\. Evaluation metrics are averaged over several independent runs with distinct random seeds to ensure reproducibility\.

## IIIPerformance Analysis

### III\-AOverall DQN Agent Performance

As reported in Table[II](https://arxiv.org/html/2610.38101#S2.T2), our framework demonstrates consistent gains over the static slotting baseline \(fixed zone assignment by SKU category\) across all three KPIs, averaged over five runs with distinct random seeds\. The agent reduces average picking time by 29\.66% \(169\.22 to 118\.32 s\), improves throughput by 35\.42% \(23\.35 to 31\.62 orders/hour\), and decreases travel distance by 18\.54%, reflecting its learned ability to prioritize high\-frequency items near dispatch zones while internalizing spatial efficiency\. Low standard deviations across all metrics confirm policy stability under varying initializations\.

### III\-BQ\-Network Architecture Ablation

To validate our architectural choices, we compare the proposed CNN\-BiLSTM\-Attention Q\-network against four ablated variants, as shown in Fig\.[3](https://arxiv.org/html/2610.38101#S2.F3)\. The full architecture achieves the lowest median picking time with the tightest inter\-quartile range, confirming that each component is essential: without the CNN, learning variance increases significantly, substituting GRU for BiLSTM reduces temporal modeling fidelity, and removing attention weakens the agent’s ability to identify critical state transitions, and replacing CNN with MLP yields the poorest overall performance\.

### III\-CRuntime Operational Analysis

The FlexSim dashboard results in Fig\.[4](https://arxiv.org/html/2610.38101#S2.F4)provide deeper operational insight into how our framework’s slotting decisions propagate through the warehouse system at runtime\. In terms of travel distance, the optimized configuration reduces per\-transporter travel by approximately 19% across all four units, a uniform improvement that confirms the agent learns globally efficient zone assignments rather than benefiting only specific transporters or routes\. The transporter utilization results are particularly revealing\. Under static slotting, transporters spend up to 40% of their time idle or repositioning without a load, a direct consequence of poorly placed inventory forcing long empty traversals\. The DQN\-Agent resolves this inefficiency by aligning product locations with real\-time demand patterns, pushing all four transporters above 98% productive utilization\. This near\-elimination of idle time represents a fundamental shift in how the warehouse allocates its material handling resources, transforming transporters from underutilized assets into continuous productive units\.

These efficiency gains compound naturally into throughput, where the framework raises source throughput from 6,629 to 8,543 units and sink throughput from 5,709 to 7,512 units\. This outcome underscores a key finding of our work: intelligent slotting does not merely improve individual metrics in isolation but creates a cascading effect across the entire material flow pipeline, from storage assignment through transporter routing to final order delivery\.

## IVConclusion

This study introduces a dynamic slotting optimization framework based on DQN, integrated with a FlexSim Digital Twin for smart warehouse logistics\. By unifying reinforcement learning with a high\-fidelity simulation environment and the standardized AAS, the framework enables real\-time, adaptive decision\-making while maintaining consistent and interoperable asset modeling\. The proposed approach is validated through extensive simulation, where the DQN agent consistently outperforms traditional static slotting methods by learning demand\-aware and distance\-efficient placement strategies\. The results demonstrate significant improvements in operational efficiency, including faster picking processes, increased system throughput, and reduced travel distances\. These gains are driven by the agent’s ability to prioritize high\-demand items, minimize unnecessary transporter movements, and reduce idle and non\-productive states\.

## References

- \[1\]\(2025\)FlexSim\-based digital twin framework for smart warehouse monitoring and performance optimization\.Proceedings of the Korean Institute of Communication Sciences Conference,pp\. 362–363\.Cited by:[§I](https://arxiv.org/html/2610.38101#S1.p1.1)\.
- \[2\]M\. M\. Alam, R\. A\. N\. Ugli, K\. J\. Tanha, and T\. Jun\(2026\)AI\-enhanced digital twin systems for warehouse logistics optimization: a review of challenges with solutions and future directions\.ICT Express\.Cited by:[§I](https://arxiv.org/html/2610.38101#S1.p1.1)\.
- \[3\]Y\. Jia, W\. Chen, T\. Gu, H\. Zhang, H\. Yuan, Y\. Lin, W\. Yu, and J\. Zhang\(2017\)A dynamic logistic dispatching system with set\-based particle swarm optimization\.IEEE Transactions on Systems, Man, and Cybernetics: Systems48\(9\),pp\. 1607–1621\.Cited by:[§I](https://arxiv.org/html/2610.38101#S1.p2.1)\.
- \[4\]E\. Hu, J\. He, and S\. Shen\(2023\)A dynamic integrated scheduling method based on hierarchical planning for heterogeneous agv fleets in warehouses\.Frontiers in neurorobotics16,pp\. 1053067\.Cited by:[§I](https://arxiv.org/html/2610.38101#S1.p2.1)\.
- \[5\]W\. Cheng and W\. Meng\(2023\)An efficient genetic algorithm for multi agv scheduling problem about intelligent warehouse\.Robotic Intelligence and Automation43\(4\),pp\. 382–393\.Cited by:[§I](https://arxiv.org/html/2610.38101#S1.p2.1)\.
- \[6\]B\. Li and H\. Ma\(2023\)Double\-deck multi\-agent pickup and delivery: multi\-robot rearrangement in large\-scale warehouses\.IEEE Robotics and Automation Letters8\(6\),pp\. 3701–3708\.Cited by:[§I](https://arxiv.org/html/2610.38101#S1.p2.1)\.
- \[7\]M\. R\. Islam, S\. M\. Ali, A\. M\. Fathollahi\-Fard, and G\. Kabir\(2021\)A novel particle swarm optimization\-based grey model for the prediction of warehouse performance\.Journal of Computational Design and Engineering8\(2\),pp\. 705–727\.Cited by:[§I](https://arxiv.org/html/2610.38101#S1.p3.1)\.
- \[8\]C\. Waubert de Puiseau, D\. T\. Nanfack, H\. Tercan, J\. Löbbert\-Plattfaut, and T\. Meisen\(2022\)Dynamic storage location assignment in warehouses using deep reinforcement learning\.Technologies10\(6\),pp\. 129\.Cited by:[§I](https://arxiv.org/html/2610.38101#S1.p3.1)\.
- \[9\]S\. Wu, W\. Chiu, and C\. Wu\(2024\)Deep reinforcement learning for task assignment and shelf reallocation in smart warehouses\.IEEE Access12,pp\. 58915–58926\.Cited by:[§I](https://arxiv.org/html/2610.38101#S1.p3.1)\.
- \[10\]S\. Li, Z\. Zhao, D\. Wang, K\. Li, G\. Liu, and Q\. Wang\(2025\)A reinforcement learning framework for efficient task allocation among agvs in smart warehouse\.IEEE Internet of Things Journal12\(11\),pp\. 16947–16961\.Cited by:[§I](https://arxiv.org/html/2610.38101#S1.p3.1)\.
- \[11\]S\. Abinaya, M\. Deepak, and A\. S\. Alphonse\(2024\)Enhanced image captioning using bahdanau attention mechanism and heuristic beam search algorithm\.IEEE Access12,pp\. 100991–101003\.Cited by:[§II\-B](https://arxiv.org/html/2610.38101#S2.SS2.p2.1)\.