Quantum Frog: Emergent Cooperation and Difficulty Scaling in a Quantized-Time Cooperative Game

# Emergent Cooperation and Difficulty Scaling in a Quantized-Time Cooperative Game
Source: [https://arxiv.org/html/2605.23930](https://arxiv.org/html/2605.23930)
###### Abstract

We introduce*Quantum Frog*, a two\-player cooperative game built on a novel*quantized\-time*mechanic in which the environment advances only when a player acts\. Inspired by the classic arcade game Frogger, Quantum Frog requires two frogs to cross an 8×\\times8 grid of traffic and reach the far side together\. We use reinforcement learning \(RL\) as an analytical lens to answer four design questions: \(1\) how does game difficulty scale with traffic density, \(2\) what is the optimal single\-agent policy and why, \(3\) how large is the cooperation gap between independent and cooperative two\-agent play, and \(4\) what joint strategy emerges when agents are incentivised to cooperate? We train agents through five escalating stages, Tabular Q\-Learning, Deep Q\-Network \(dqn\), Independentdqn\(idqn\), and Multi\-Agent Proximal Policy Optimisation \(mappowith a centralised critic\), evaluating each against traffic densities of one to six cars\. Our key findings are: \(i\) the quantized\-time mechanic makes a*rush strategy*\(moving directly upward at every step\) universally optimal, as time exposure to traffic is minimised; \(ii\) adding an uncoordinated second player is harder than sextupling the traffic for a single expert player; \(iii\) cooperative training recovers \+32–34 percentage points of joint success rate relative to independent agents and reduces episode length from∼\\sim90 to∼\\sim6 steps; and \(iv\) the emergent cooperative strategy is synchronised rushing, not complex positional coordination, illustrating that shared incentives alone suffice to align agents in time\-critical cooperative tasks\. These findings provide concrete, empirically grounded guidance for the commercial design of Quantum Frog and offer broader insights into the role of environment mechanics in shaping multi\-agent learning dynamics\.

###### Contents

1. [1Introduction](https://arxiv.org/html/2605.23930#S1)
2. [2Related Work](https://arxiv.org/html/2605.23930#S2)
3. [3Environment](https://arxiv.org/html/2605.23930#S3)1. [3\.1Game Description](https://arxiv.org/html/2605.23930#S3.SS1) 2. [3\.2State Representation](https://arxiv.org/html/2605.23930#S3.SS2) 3. [3\.3Action Space](https://arxiv.org/html/2605.23930#S3.SS3) 4. [3\.4Reward Function](https://arxiv.org/html/2605.23930#S3.SS4) 5. [3\.5Episode Dynamics](https://arxiv.org/html/2605.23930#S3.SS5) 6. [3\.6Implementation](https://arxiv.org/html/2605.23930#S3.SS6)
4. [4Methods](https://arxiv.org/html/2605.23930#S4)1. [4\.1Tabular Q\-Learning \(Stages 1–2\)](https://arxiv.org/html/2605.23930#S4.SS1) 2. [4\.2Deep Q\-Network \(Stage 3\)](https://arxiv.org/html/2605.23930#S4.SS2) 3. [4\.3Independent DQN \(Stage 4\)](https://arxiv.org/html/2605.23930#S4.SS3) 4. [4\.4Multi\-Agent PPO with Centralised Critic \(Stage 5\)](https://arxiv.org/html/2605.23930#S4.SS4) 5. [4\.5Evaluation Protocol](https://arxiv.org/html/2605.23930#S4.SS5)
5. [5Results](https://arxiv.org/html/2605.23930#S5)1. [5\.1Single\-Agent Performance \(Stages 1–3\)](https://arxiv.org/html/2605.23930#S5.SS1) 2. [5\.2Multi\-Agent Without Cooperation \(Stage 4: IDQN\)](https://arxiv.org/html/2605.23930#S5.SS2) 3. [5\.3Cooperative Multi\-Agent \(Stage 5: MAPPO\)](https://arxiv.org/html/2605.23930#S5.SS3)
6. [6Discussion](https://arxiv.org/html/2605.23930#S6)1. [6\.1The Rush Strategy as Emergent Optimality](https://arxiv.org/html/2605.23930#S6.SS1) 2. [6\.2The Cooperation Gap and Its Shape](https://arxiv.org/html/2605.23930#S6.SS2) 3. [6\.3Emergent Cooperation is Synchronised Rushing, Not Tactical Coordination](https://arxiv.org/html/2605.23930#S6.SS3) 4. [6\.4Implications for Game Design](https://arxiv.org/html/2605.23930#S6.SS4) 5. [6\.5Broader Implications for Cooperative MARL](https://arxiv.org/html/2605.23930#S6.SS5) 6. [6\.6Limitations](https://arxiv.org/html/2605.23930#S6.SS6)
7. [7Conclusion](https://arxiv.org/html/2605.23930#S7)
8. [References](https://arxiv.org/html/2605.23930#bib)
9. [AHyperparameters](https://arxiv.org/html/2605.23930#A1)

## 1Introduction

Designing a multiplayer game is fundamentally a question about incentive structures: what mechanics encourage players to communicate, to coordinate, and to find elegant solutions to difficult problems? Answering such questions analytically is notoriously hard, human playtesters provide subjective and noisy signal, and exhaustive human trials are expensive\. Reinforcement learning agents offer an alternative: a policy trained to optimise a reward function will expose the true incentive landscape of a game far more systematically than human play, revealing what behaviours the game actually rewards\.

#### The game\.

Quantum Frog is a two\-dimensional, turn\-based game played on an 8×\\times8 grid\. Two frogs start at the bottom row and must cross to the top while avoiding a stream of horizontally moving cars\. The game’s defining mechanic is*quantized time*: the environment is frozen between player decisions, advancing by exactly one tick whenever a frog acts\. This departs from continuous\-time variants \(such as the original Frogger\(Konami,[1981](https://arxiv.org/html/2605.23930#bib.bib22)\)\) in a fundamental way, players can deliberate without penalty and car positions are perfectly predictable\. The game is cooperative: both frogs must survive and reach the far side, requiring players to communicate and act jointly\. Figure[1](https://arxiv.org/html/2605.23930#S1.F1)illustrates the board layout and one complete time step\.

![Refer to caption](https://arxiv.org/html/2605.23930v1/game.png)Figure 1:Quantum Frogs Game Representation\. Both frogs \(represented as boy and girl charachters\) begin at the bottom row \(row 7\); four cars move horizontally at speed different speeds and directions\. The entire board is frozen while the players choose their actions\.
#### Motivation\.

The quantized\-time mechanic is understudied in game design research\. It transforms an arcade reflex challenge into a combinatorial planning problem, sharply changing the optimal strategy and the character of difficulty scaling\. Understanding these effects empirically, before the game reaches a commercial audience, is precisely the kind of analysis that RL agents can provide\. Beyond the specific game, this work touches on a fundamental question in cooperative multi\-agent RL: when agents have shared objectives, what form does the emergent coordination take, and how much does the choice of learning algorithm determine whether cooperation appears at all?

#### Research questions\.

We formulate four concrete research questions that guide the experimental design:

1. RQ1\.Difficulty scaling\.How does the win rate of an optimal single\-agent policy change as a function of traffic density \(number of cars\)?
2. RQ2\.Optimal single\-agent strategy\.What policy does a converged agent adopt, and how does the quantized\-time rule shape it?
3. RQ3\.Cooperation gap\.How much does joint success rate improve when agents are trained cooperatively \(shared reward, centralised critic\) versus independently \(separate rewards\)?
4. RQ4\.Emergent joint strategy\.What coordination behaviour do cooperative agents discover, and does it resemble complex tactical cooperation or a simpler structure?

#### Contributions\.

- •A novel cooperative game environment, Quantum Frog, implemented with the Gymnasium API\(Towerset al\.,[2023](https://arxiv.org/html/2605.23930#bib.bib18)\)and open\-sourced\.
- •A five\-stage empirical study spanning tabular Q\-Learning through cooperativemappo, providing a clean curriculum across algorithm families\.
- •Quantitative answers to all four research questions, including the first measurement of the cooperation gap in a quantized\-time setting\.
- •Concrete game\-design recommendations grounded in RL analysis\.

## 2Related Work

#### Deep reinforcement learning for games\.

Mnihet al\.\([2015](https://arxiv.org/html/2605.23930#bib.bib1)\)demonstrated that a Deep Q\-Network \(dqn\) trained directly from pixel observations could achieve human\-level performance on 49 Atari games, establishing RL as a viable tool for game analysis\.Silveret al\.\([2016](https://arxiv.org/html/2605.23930#bib.bib2)\)extended this to the combinatorial planning domain with AlphaGo, showing that RL agents can uncover strategies invisible to human experts\. Our work is closer in spirit toMnihet al\.\([2015](https://arxiv.org/html/2605.23930#bib.bib1)\): we use RL not to beat human players, but to characterise the game’s reward landscape\.

#### Difficulty scaling and game design\.

Smith and Whitehead \([2010](https://arxiv.org/html/2605.23930#bib.bib21)\)formalised procedural difficulty generation\.Hunickeet al\.\([2004](https://arxiv.org/html/2605.23930#bib.bib20)\)introduced the Mechanics–Dynamics–Aesthetics framework for analysing how game rules produce player experience\. Our work contributes an empirical, RL\-based methodology to this tradition: rather than reasoning analytically about mechanics, we let agents discover the implied difficulty function directly\.

#### Multi\-agent reinforcement learning\.

The cooperative MARL problem has been studied extensively sinceBusoniuet al\.\([2008](https://arxiv.org/html/2605.23930#bib.bib12)\)\. A central challenge is*non\-stationarity*: as agents learn simultaneously, each agent’s effective environment shifts, violating the stationarity assumption required for single\-agent RL convergence guarantees\(Hernandez\-Lealet al\.,[2017](https://arxiv.org/html/2605.23930#bib.bib13)\)\.*Independent Q\-Learning*\(Tan,[1993](https://arxiv.org/html/2605.23930#bib.bib11)\)ignores this problem and trains each agent with its own reward and replay buffer; it remains a strong baseline despite its theoretical limitations\.

#### Centralised training with decentralised execution \(CTDE\)\.

The CTDE paradigm\(Oliehoek and Amato,[2016](https://arxiv.org/html/2605.23930#bib.bib14)\)addresses non\-stationarity by allowing richer information during training while preserving decentralised execution\.Loweet al\.\([2017](https://arxiv.org/html/2605.23930#bib.bib8)\)applied actor\-critic methods under CTDE \(MADDPG\)\.Rashidet al\.\([2018](https://arxiv.org/html/2605.23930#bib.bib7)\)introducedqmix, a value\-decomposition method that uses a monotonic mixing network to ensure that individually greedy actions are jointly optimal \(the IGM principle\)\.Yuet al\.\([2022](https://arxiv.org/html/2605.23930#bib.bib6)\)showed that a straightforward extension of Proximal Policy Optimisation\(Schulmanet al\.,[2017](https://arxiv.org/html/2605.23930#bib.bib4)\)to multi\-agent settings,mappo, is competitive with or superior to more specialised cooperative algorithms on the StarCraft Multi\-Agent Challenge\(Samvelyanet al\.,[2019](https://arxiv.org/html/2605.23930#bib.bib15)\)\. We adoptmappoas our primary cooperative algorithm, motivated by its stability and strong empirical performance\.

#### Cooperative game environments\.

The StarCraft Multi\-Agent Challenge \(SMAC\)\(Samvelyanet al\.,[2019](https://arxiv.org/html/2605.23930#bib.bib15)\), Overcooked\(Carrollet al\.,[2019](https://arxiv.org/html/2605.23930#bib.bib16)\), and the Cooperative Navigation task\(Loweet al\.,[2017](https://arxiv.org/html/2605.23930#bib.bib8)\)are standard cooperative MARL benchmarks\. Our environment is simpler and more interpretable than these, making it suitable for isolating the effect of the quantized\-time mechanic from confounding environmental complexity\.

## 3Environment

### 3\.1Game Description

Quantum Frog is played on a discrete 8×\\times8 grid\. One or two frogs start at the bottom row \(row 7\) and must reach the top row \(row 0\)\. Horizontally moving cars occupy rows 1–6; row 0 \(goal\) and row 7 \(start\) are car\-free\. Cars wrap around when they leave the grid boundary\.

#### Quantized\-time rule\.

Unlike continuous\-time traffic games, the environment advances by exactly one simulation tick each time a frog callsstep\(\)\. Between steps, all agents and cars are frozen\. This rule eliminates reaction\-time pressure and makes the game a fully observable, deterministic planning problem: at every decision point the player has complete information and unlimited deliberation time\.

### 3\.2State Representation

The state is a three\-channel 8×\\times8 integer array𝐬∈ℤ3×8×8\\mathbf\{s\}\\in\\mathbb\{Z\}^\{3\\times 8\\times 8\}:

𝐬​\[0,r,c\]\\displaystyle\\mathbf\{s\}\[0,r,c\]∈\{0,1,2\}\\displaystyle\\in\\\{0,1,2\\\}frog positions \(1 = frog A, 2 = frog B\)𝐬​\[1,r,c\]\\displaystyle\\mathbf\{s\}\[1,r,c\]∈\{0,1\}\\displaystyle\\in\\\{0,1\\\}car presence𝐬​\[2,r,c\]\\displaystyle\\mathbf\{s\}\[2,r,c\]∈ℤ\\displaystyle\\in\\mathbb\{Z\}signed car velocityThe flat observation vector has dimensionality3×8×8=1923\\times 8\\times 8=192\. This representation is compatible with both multi\-layer perceptron \(MLP\) and convolutional network policies, and encodes all information needed for optimal play\.

### 3\.3Action Space

Each frog chooses from five primitive actions: \{up,down,left,right,stay\}\. In single\-frog experiments the action space isDiscrete​\(5\)\\text\{Discrete\}\(5\)\. In two\-frog experiments we useMultiDiscrete​\(\[5,5\]\)\\text\{MultiDiscrete\}\(\[5,5\]\)for joint actions, equivalent to 25 joint action combinations\.

### 3\.4Reward Function

The reward function balances terminal outcomes with dense shaping:

rt=\{\+100frog reaches row 0 \(goal\)−100frog occupies a car cell \(collision\)\+1frog advances upward one row−1otherwise \(step cost\)r\_\{t\}=\\begin\{cases\}\+100&\\text\{frog reaches row 0 \(goal\)\}\\\\ \-100&\\text\{frog occupies a car cell \(collision\)\}\\\\ \+1&\\text\{frog advances upward one row\}\\\\ \-1&\\text\{otherwise \(step cost\)\}\\end\{cases\}\(1\)The step cost−1\-1penalises deliberation and encourages efficient paths\. The progress shaping\+1\+1provides a dense gradient before the sparse terminal reward is reached\. In two\-frog cooperative training \(mappo\), the team reward isrteam=rA\+rBr\_\{\\text\{team\}\}=r\_\{A\}\+r\_\{B\}, shared equally by both agents\.

### 3\.5Episode Dynamics

An episode terminates when \(a\) both frogs reach row 0 \(success\), or \(b\) any frog collides with a car \(failure\)\. If neither condition is met within 200 steps, the episode is truncated\. Car speeds are drawn uniformly from a configurable set \(e\.g\.\{1\}\\\{1\\\}or\{1,2\}\\\{1,2\\\}squares per step\) at the start of each episode\.

### 3\.6Implementation

The environment is implemented following the Gymnasium API\(Towerset al\.,[2023](https://arxiv.org/html/2605.23930#bib.bib18)\), exposingreset\(\),step\(action\), andrender\(\)methods\. Training uses vectorised environments \(up to 32 parallel instances\) to improve throughput\. All experiments were run on a single node with four NVIDIA GPUs; each seed occupies one GPU\.

## 4Methods

We train agents through five escalating stages, each adding complexity to the environment and algorithm\. Table[1](https://arxiv.org/html/2605.23930#S4.T1)summarises the experimental design\.

Table 1:Experimental stages\. Each stage trains until convergence before evaluation\.### 4\.1Tabular Q\-Learning \(Stages 1–2\)

We implement one\-step Q\-Learning\(Watkins and Dayan,[1992](https://arxiv.org/html/2605.23930#bib.bib3)\)with anε\\varepsilon\-greedy policy and multiplicativeε\\varepsilon\-decay:

Q​\(s,a\)←Q​\(s,a\)\+α​\[r\+γ​maxa′⁡Q​\(s′,a′\)−Q​\(s,a\)\]Q\(s,a\)\\leftarrow Q\(s,a\)\+\\alpha\\bigl\[r\+\\gamma\\max\_\{a^\{\\prime\}\}Q\(s^\{\\prime\},a^\{\\prime\}\)\-Q\(s,a\)\\bigr\]\(2\)The Q\-table is a hash map keyed onobs\.tobytes\(\), mapping each observed state to a length\-5 value vector initialised to zero\. Parameters:α=0\.1\\alpha=0\.1,γ=0\.99\\gamma=0\.99,ε0=1\.0\\varepsilon\_\{0\}=1\.0,εmin=0\.01\\varepsilon\_\{\\min\}=0\.01, decay=0\.9995=0\.9995per episode\. Training runs for 20 000 episodes \(Stage 1\) and 50 000 episodes \(Stage 2\)\.

### 4\.2Deep Q\-Network \(Stage 3\)

To generalise beyond the discrete state space accessible to a tabular method, Stage 3 replaces the Q\-table with a multi\-layer perceptron \(MLP\):

Qθ​\(s,⋅\):ℝ192→ℝ5Q\_\{\\theta\}\(s,\\cdot\):\\;\\mathbb\{R\}^\{192\}\\to\\mathbb\{R\}^\{5\}\(3\)with two hidden layers of width 256 and ReLU activations\. We follow thedqnalgorithm ofMnihet al\.\([2015](https://arxiv.org/html/2605.23930#bib.bib1)\)with two stabilising mechanisms:

- •Experience replay\.Transitions\(s,a,r,s′,done\)\(s,a,r,s^\{\\prime\},\\text\{done\}\)are stored in a ring buffer of capacity 100 000\. Each gradient step samples a uniformly random mini\-batch of 128 transitions, breaking temporal correlation\.
- •Target network\.A frozen copyQθ−Q\_\{\\theta^\{\-\}\}of the online network provides the regression target, updated to matchQθQ\_\{\\theta\}every 1 000 steps\. This prevents the target from shifting with every gradient update\.

The loss is:

ℒ​\(θ\)=𝔼\(s,a,r,s′\)∼ℬ​\[\(r\+γ​maxa′⁡Qθ−​\(s′,a′\)−Qθ​\(s,a\)\)2\]\\mathcal\{L\}\(\\theta\)=\\mathbb\{E\}\_\{\(s,a,r,s^\{\\prime\}\)\\sim\\mathcal\{B\}\}\\Bigl\[\\bigl\(r\+\\gamma\\max\_\{a^\{\\prime\}\}Q\_\{\\theta^\{\-\}\}\(s^\{\\prime\},a^\{\\prime\}\)\-Q\_\{\\theta\}\(s,a\)\\bigr\)^\{2\}\\Bigr\]\(4\)Exploration uses a linearε\\varepsilon\-schedule from 1\.0 to 0\.05 over 30% of training\. We train for 150 000 environment steps with learning rate10−310^\{\-3\}and train frequency 4 \(one gradient step per 4 environment steps\)\. Four independent runs with different random seeds are conducted per stage, with one run per GPU\.

### 4\.3Independent DQN \(Stage 4\)

Stage 4 introduces the second frog using Independent Q\-Learning\(Tan,[1993](https://arxiv.org/html/2605.23930#bib.bib11)\): each agent maintains its owndqn, replay buffer, target network, and reward signal\. Agent A trains onrAr\_\{A\}only; Agent B trains onrBr\_\{B\}only\. Both agents observe the full global state𝐬\\mathbf\{s\}\(including each other’s position\) but never explicitly coordinate\.

ℒA\\displaystyle\\mathcal\{L\}\_\{A\}=𝔼​\[\(rA\+γ​max⁡QA​\(s′\)−QA​\(s,aA\)\)2\]\\displaystyle=\\mathbb\{E\}\\bigl\[\\bigl\(r\_\{A\}\+\\gamma\\max Q\_\{A\}\(s^\{\\prime\}\)\-Q\_\{A\}\(s,a\_\{A\}\)\\bigr\)^\{2\}\\bigr\]\(5\)ℒB\\displaystyle\\mathcal\{L\}\_\{B\}=𝔼​\[\(rB\+γ​max⁡QB​\(s′\)−QB​\(s,aB\)\)2\]\\displaystyle=\\mathbb\{E\}\\bigl\[\\bigl\(r\_\{B\}\+\\gamma\\max Q\_\{B\}\(s^\{\\prime\}\)\-Q\_\{B\}\(s,a\_\{B\}\)\\bigr\)^\{2\}\\bigr\]\(6\)From each agent’s perspective the environment is non\-stationary, since the other agent’s evolving policy changes the effective transition dynamics\(Hernandez\-Lealet al\.,[2017](https://arxiv.org/html/2605.23930#bib.bib13)\)\. This violates the convergence conditions of Q\-Learning and is expected to produce instability\. We use 200 000 training steps, 32 parallel environments, and the same network architecture and hyperparameters as Stage 3\.

### 4\.4Multi\-Agent PPO with Centralised Critic \(Stage 5\)

Stage 5 adoptsmappo\(Yuet al\.,[2022](https://arxiv.org/html/2605.23930#bib.bib6)\), a Centralised Training, Decentralised Execution \(CTDE\) algorithm\(Oliehoek and Amato,[2016](https://arxiv.org/html/2605.23930#bib.bib14)\)\.

#### Architecture\.

Two actor networksπA​\(aA∣s;ϕA\)\\pi\_\{A\}\(a\_\{A\}\\mid s;\\,\\phi\_\{A\}\)andπB​\(aB∣s;ϕB\)\\pi\_\{B\}\(a\_\{B\}\\mid s;\\,\\phi\_\{B\}\)output categorical distributions over 5 actions\. A single centralised criticVψ​\(s\)V\_\{\\psi\}\(s\)estimates the joint state value\. All three networks share the MLP architecture \(192→\\to256→\\to256→\\tooutput\)\.

#### Data collection\.

At each update, we collect a rollout ofT=128T=128steps acrossN=32N=32parallel environments, yielding 4 096 joint transitions\. Both actors act simultaneously; the team rewardrteam=rA\+rBr\_\{\\text\{team\}\}=r\_\{A\}\+r\_\{B\}is assigned to both agents\.

#### Advantage estimation\.

We compute Generalised Advantage Estimation \(GAE\)\(Schulmanet al\.,[2016](https://arxiv.org/html/2605.23930#bib.bib5)\)withλ=0\.95\\lambda=0\.95:

A^t=∑k=0T−t−1\(γ​λ\)k​δt\+k,δt=rt\+γ​Vψ​\(st\+1\)−Vψ​\(st\)\\hat\{A\}\_\{t\}=\\sum\_\{k=0\}^\{T\-t\-1\}\(\\gamma\\lambda\)^\{k\}\\delta\_\{t\+k\},\\qquad\\delta\_\{t\}=r\_\{t\}\+\\gamma V\_\{\\psi\}\(s\_\{t\+1\}\)\-V\_\{\\psi\}\(s\_\{t\}\)\(7\)

#### PPO update\.

For each agenti∈\{A,B\}i\\in\\\{A,B\\\}, the clipped surrogate objective is:

ℒactori=−𝔼t​\[min⁡\(ρti​A^t,clip​\(ρti,1−ϵ,1\+ϵ\)​A^t\)\],ρti=πi​\(ati∣st\)πiold​\(ati∣st\)\\mathcal\{L\}\_\{\\text\{actor\}\}^\{i\}=\-\\mathbb\{E\}\_\{t\}\\Bigl\[\\min\\\!\\Bigl\(\\rho\_\{t\}^\{i\}\\hat\{A\}\_\{t\},\\;\\text\{clip\}\\\!\\bigl\(\\rho\_\{t\}^\{i\},1\{\-\}\\epsilon,1\{\+\}\\epsilon\\bigr\)\\hat\{A\}\_\{t\}\\Bigr\)\\Bigr\],\\quad\\rho\_\{t\}^\{i\}=\\frac\{\\pi\_\{i\}\(a\_\{t\}^\{i\}\\mid s\_\{t\}\)\}\{\\pi\_\{i\}^\{\\text\{old\}\}\(a\_\{t\}^\{i\}\\mid s\_\{t\}\)\}\(8\)with clipping thresholdϵ=0\.2\\epsilon=0\.2\. The total loss combines both actor losses, a critic MSE loss, and an entropy bonus:

ℒ=ℒactorA\+ℒactorB\+0\.5​ℒcritic−0\.01​\(ℋ​\[πA\]\+ℋ​\[πB\]\)\\mathcal\{L\}=\\mathcal\{L\}\_\{\\text\{actor\}\}^\{A\}\+\\mathcal\{L\}\_\{\\text\{actor\}\}^\{B\}\+0\.5\\,\\mathcal\{L\}\_\{\\text\{critic\}\}\-0\.01\\bigl\(\\mathcal\{H\}\[\\pi\_\{A\}\]\+\\mathcal\{H\}\[\\pi\_\{B\}\]\\bigr\)\(9\)Parameters are updated for 4 epochs per rollout using mini\-batches of 512\. Training runs for 300 000 environment steps with learning rate3×10−43\\times 10^\{\-4\}and gradient norm clipping at 0\.5\.

#### How cooperation emerges\.

The team reward propagates consequences across agents: if agent A’s action leads to agent B being hit, agent A’s return decreases\. The centralised criticVψ​\(s\)V\_\{\\psi\}\(s\)encodes the long\-term value of the joint configuration, guiding both actors toward trajectories that benefit both\. Coordination is not produced by explicit messaging but by shared incentives and a globally informed value function\.

### 4\.5Evaluation Protocol

After each stage, the learned policy is evaluated deterministically \(ε=0\\varepsilon=0, greedy actor outputs\) over 200 episodes per traffic density \(1–6 cars\)\. We report:

- •Win rate: fraction of episodes in which the success condition is met \(single frog reaches top; or both frogs reach top in two\-agent stages\)\.
- •Average episode length: mean steps per episode\.
- •Individual win rates\(two\-agent stages\): fraction in which frog A or frog B individually reaches the top, regardless of partner\.
- •Seed variance: standard deviation of win rate across 4 independent training runs \(seeds\)\.

## 5Results

### 5\.1Single\-Agent Performance \(Stages 1–3\)

Table 2:Stage 3dqnwin rate and average episode length at evaluation, mean±\\pmstandard deviation across 4 seeds\. Trained on 4 cars \(speeds 1–2\); evaluated on 1–6 cars\.Tabular Q\-Learning \(Stages 1–2\) achieves 94\.2% win rate with 2 cars \(speed 1\) and 58\.5% with 4 cars \(mixed speeds\)\. Stage 3dqn, trained on 4 cars, generalises smoothly across all densities \(Table[2](https://arxiv.org/html/2605.23930#S5.T2)\), losing approximately 7–10 percentage points per additional car, with diminishing returns above 5 cars\.

The most striking result is the average episode length: all converged single\-agent policies solve the game in 6–7 steps, close to the theoretical minimum of 7 steps required to traverse 7 rows upward\. This indicates agents discovered a*rush strategy*: move directly upward at every step without lateral evasion\.

### 5\.2Multi\-Agent Without Cooperation \(Stage 4: IDQN\)

Table 3:Stage 4idqnevaluation results, mean across 4 seeds\. Individual win rates indicate the fraction of episodes in which that frog alone reached the top\.Table[3](https://arxiv.org/html/2605.23930#S5.T3)reveals three findings\. First,idqnachieves only 43\.0% joint success at 1 car, lower than Stage 3dqnat 6 cars \(58\.8%\)\. A second uncoordinated player is more damaging than quintupling the traffic for an expert single agent\. This confirms that the difficulty of requiring*both*frogs to succeed is geometrically harder than the marginal difficulty of each individual crossing\.

Second, the variance across seeds is extraordinary: at 1 car, win rates span 10\.5% to 79\.0% \(seed 102 vs\. seed 103\), a 7×\\timesspread\. The best seed performs near\-expert; the worst barely exceeds random\. This reflects non\-stationary multi\-agent learning dynamics: small differences in early exploration cause one agent to converge faster, which shifts the environment for the other and creates a compounding feedback loop\.

Third, Frog B consistently outperforms Frog A \(73\.4% vs\. 56\.1% at 1 car\)\. Both frogs start in different grid columns but otherwise face symmetric conditions\. As we show in Section[5\.3](https://arxiv.org/html/2605.23930#S5.SS3),mappoachieves identical win rates for both frogs, confirming this asymmetry is a training artifact rather than a positional advantage\.

Average episode length is 74–90 steps at low car counts, 12×\\timeslonger than the single\-agentdqn\. Agents have not discovered the rush strategy and are wandering or hedging\.

### 5\.3Cooperative Multi\-Agent \(Stage 5: MAPPO\)

Table 4:Stage 5mappoevaluation results \(all four seeds converged identically\), compared to Stage 4idqnmean\.Δ\\Deltadenotes absolute percentage\-point improvement ofmappooveridqn\.mappoimproves joint success rate by 10\.8–34\.1 percentage points across all traffic densities \(Table[4](https://arxiv.org/html/2605.23930#S5.T4)\)\. The improvement is largest at 1–2 cars \(\+32–34 pp\), where the task is tractable but independent agents fail due to training instability, and smallest at 6 cars \(\+11 pp\), where even cooperative play is limited by traffic density\.

Episode length collapses from∼\\sim90 \(IDQN\) to∼\\sim6 steps \(MAPPO\)\. Critically, individual win rates are identical for both frogs \(both 75\.0% at 1 car, both 58\.5% at 2 cars, etc\.\), eliminating the asymmetry observed inidqn\. The agents discovered the same synchronised rush strategy as the single\-agentdqn\.

Perhaps most notably, all fourmapposeeds converged to*identical*policies\. The seed variance that dominatedidqnresults is eliminated: the centralised critic’s global value signal breaks the non\-stationary feedback loop and produces deterministic convergence\.

## 6Discussion

### 6\.1The Rush Strategy as Emergent Optimality

Every algorithm that converged, tabular Q\-Learning,dqn, andmappo, discovered the same policy: move directly upward at every step\.idqnis the only exception, averaging 74–90 steps per episode instead of 6–7\.

The rush strategy is not a coincidence\. It is the direct, logical consequence of the quantized\-time mechanic combined with the step cost\. Since the environment advances exactly one tick per player action, a frog that takes fewer steps exposes itself to fewer total car movements\. A frog crossing in 7 steps gives the traffic 7 ticks to reach it; one crossing in 90 steps gives the traffic 90 ticks\. The time cost and the collision risk point in the same direction: act fast\.

This is a meaningful design finding\. The quantized\-time rule does not simply slow down the game, it*structurally rewards decisive, direct play*and penalises deliberation\. A player who pauses to strategise is not thinking more effectively; they are giving their adversary \(time\) an advantage\. The mechanic creates a clear, learnable optimal strategy that novice players are unlikely to discover intuitively, providing natural progression as players improve\.

### 6\.2The Cooperation Gap and Its Shape

The cooperation gap \(\+32\+32to\+34\+34pp at 1–2 cars\) is large in absolute terms and reveals that independent agents fail not because the task is impossible but because they cannot converge\. Theidqnbest seed achieves 79% joint success at 1 car, nearly matchingmappo’s 75%, demonstrating that cooperative play is within reach of the environment\. The problem is the reliability of learning it\.

The gap is largest at intermediate traffic density \(2 cars\) and narrows at both extremes\. At very low density \(1 car\), uncoordinated agents can occasionally succeed by chance, inflating theidqnbaseline\. At very high density \(5–6 cars\), even perfectly coordinated agents fail frequently, as the game itself becomes near\-unwinnable, pushing both methods toward zero\. The regime of maximal cooperation value is 2–4 cars: hard enough to require coordination but tractable enough for coordination to help\.

### 6\.3Emergent Cooperation is Synchronised Rushing, Not Tactical Coordination

A natural hypothesis is that cooperative agents would learn complex joint tactics: one frog waits while the other advances, they take different lanes, or one acts as a decoy\. The data contradicts this\.mappo’s emergent policy is*both frogs rush upward simultaneously*, identical to the single\-agent policy\.

This has a clear explanation: the team reward and centralised critic teach each actor that its return is maximised when both complete the crossing quickly\. The simplest way to achieve this is for both to use the individually optimal \(rush\) strategy\. Complex coordination, one frog waiting for the other, would slow one agent down, increasing team exposure to traffic\. In a quantized\-time environment, speed dominates positioning\.

The broader implication for cooperative MARL is notable: shared incentives alone can induce cooperation without explicit communication channels, and the emergent cooperative strategy need not be more complex than the corresponding single\-agent strategy\. The difficulty of multi\-agent problems may sometimes be a learning problem \(non\-stationarity, credit assignment\) rather than a strategic complexity problem\.

### 6\.4Implications for Game Design

Our findings translate into four actionable design recommendations for a commercial version of Quantum Frog:

1. 1\.Quantized time is the central mechanic; preserve it\.It creates a clear optimal strategy \(rush\), makes difficulty analytically tunable via car count, and is the property that distinguishes Quantum Frog from Frogger\. Softening it \(e\.g\. partial time advance between turns\) would dilute these properties\.
2. 2\.The optimal difficulty range for cooperative play is 2–4 cars\.Below 2 cars, even uncoordinated players can succeed; the game is too easy\. Above 5 cars, even coordinated play struggles \(<30%<30\\%\); the game becomes frustrating\. The 2–4 car range is where communication skill most directly determines outcome, creating a meaningful learning curve for teams\.
3. 3\.Communication between players is necessary, not optional\.The 7×\\timesseed variance inidqnshows that players who do not communicate will have wildly inconsistent outcomes\. Game interfaces should actively facilitate synchronisation, a countdown, simultaneous reveal, or explicit ”ready” signal, rather than relying on players to coordinate implicitly\.
4. 4\.The per\-step cost is the key shaping tool\.The−1\-1/step penalty is what makes the rush strategy dominant\. Reducing it \(e\.g\. to−0\.1\-0\.1\) would make deliberation less costly and might encourage richer lateral\-avoidance strategies\. Increasing it \(e\.g\. to−5\-5\) would make any policy that is not a straight\-line rush uncompetitive\. Designers can tune the character of optimal play by adjusting this single parameter\.

### 6\.5Broader Implications for Cooperative MARL

Our results are consistent with recent findings in the cooperative MARL literature\.Yuet al\.\([2022](https://arxiv.org/html/2605.23930#bib.bib6)\)showedmappocompetitive with specialised algorithms onsmac; we corroborate this on a simpler, more interpretable environment\. Our contribution is the clean isolation of one environmental property, quantized time, as the dominant force shaping both optimal policy*and*the character of emergent cooperation\.

Theidqninstability we observe is a textbook instance of non\-stationarity\(Hernandez\-Lealet al\.,[2017](https://arxiv.org/html/2605.23930#bib.bib13)\)\. The fact that even the bestidqnseed occasionally matchesmappoperformance suggests the strategy space is simple enough that independent agents can accidentally converge; the problem is reliability\. This points to a practical design principle: in time\-critical cooperative tasks, the value of CTDE methods lies less in finding better policies and more in finding the same good policy reliably\.

### 6\.6Limitations

This study has several limitations\. First, the environment assumes full observability: both agents see the complete 8×\\times8 grid at all times\. Human players have limited attention and imperfect recall, making partial observability a natural extension\. Second, our implementation ofmappouses a centralised critic that sees the*same*observation as individual agents \(since the global state is already fully encoded in the single\-agent obs\)\. A richer global\-state critic, e\.g\. one that receives both agents’ observations concatenated, may perform better in settings with truly partial observability\. Third, we did not implementqmix, which is listed in the original design as an alternative for Stages 5–6\. Stage 6 \(6\+ cars\) remains untrained; given our difficulty curve, this setting is likely to reveal the limits of the current approach\. Finally, our results are based on a fixed 8×\\times8 grid; larger grids may change the character of difficulty scaling\.

## 7Conclusion

We presented Quantum Frog, a cooperative two\-player game with a*quantized\-time*mechanic, and used multi\-agent reinforcement learning to answer four design questions about the game\. Across five training stages and four algorithms, we found that:

- •The quantized\-time rule makes the rush strategy universally optimal: every converged algorithm traverses the grid in the theoretical minimum of∼\\sim7 steps\.
- •Adding a second uncoordinated player is harder than multiplying traffic five\-fold for an expert single agent; the joint success requirement is geometrically more demanding than marginal single\-agent difficulty\.
- •Cooperative training \(mappo\) closes the joint\-success gap by 10–34 percentage points relative to independent agents \(idqn\), with the largest gains at intermediate traffic densities \(2–4 cars\)\.
- •The emergent cooperative strategy is synchronised rushing, not complex positional coordination, demonstrating that shared incentives suffice to align agents in time\-critical cooperative tasks\.
- •mappoeliminates the 7×\\timesseed variance seen inidqn, confirming that the value of centralised training is convergence reliability rather than discovery of qualitatively different policies\.

These findings answer the game’s foundational design questions and provide empirical grounding for commercial development decisions\. They also contribute a controlled, interpretable environment to the cooperative MARL literature in which the connection between environment mechanics and emergent strategy is unusually transparent\.

Future work includes implementingqmixfor comparison on Stage 5–6, extending to partial observability with recurrent policies, and exploring whether mixed\-speed traffic \(speeds 1–3\) breaks the rush\-strategy dominance and induces richer avoidance behaviour\.

## References

- L\. Busoniu, R\. Babuska, and B\. De Schutter \(2008\)A comprehensive survey of multiagent reinforcement learning\.IEEE Transactions on Systems, Man, and Cybernetics, Part C38\(2\),pp\. 156–172\.Cited by:[§2](https://arxiv.org/html/2605.23930#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Carroll, R\. Shah, M\. K\. Ho, T\. Griffiths, S\. Seshia, P\. Abbeel, and A\. Dragan \(2019\)On the utility of learning about humans for human\-AI coordination\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.32\.Cited by:[§2](https://arxiv.org/html/2605.23930#S2.SS0.SSS0.Px5.p1.1)\.
- P\. Hernandez\-Leal, B\. Kartal, and M\. E\. Taylor \(2017\)A survey of learning in multiagent environments: dealing with non\-stationarity\.arXiv preprint arXiv:1707\.09183\.Cited by:[§2](https://arxiv.org/html/2605.23930#S2.SS0.SSS0.Px3.p1.1),[§4\.3](https://arxiv.org/html/2605.23930#S4.SS3.p1.4),[§6\.5](https://arxiv.org/html/2605.23930#S6.SS5.p2.1)\.
- R\. Hunicke, M\. LeBlanc, and R\. Zubek \(2004\)MDA: a formal approach to game design and game research\.Proceedings of the AAAI Workshop on Challenges in Game AI4\(1\),pp\. 1722\.Cited by:[§2](https://arxiv.org/html/2605.23930#S2.SS0.SSS0.Px2.p1.1)\.
- Konami \(1981\)Frogger\.Note:Arcade gameCited by:[§1](https://arxiv.org/html/2605.23930#S1.SS0.SSS0.Px1.p1.1)\.
- R\. Lowe, Y\. I\. Wu, A\. Tamar, J\. Harb, P\. Abbeel, and I\. Mordatch \(2017\)Multi\-agent actor\-critic for mixed cooperative\-competitive environments\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.30\.Cited by:[§2](https://arxiv.org/html/2605.23930#S2.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2605.23930#S2.SS0.SSS0.Px5.p1.1)\.
- V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. A\. Rusu, J\. Veness, M\. G\. Bellemare, A\. Graves, M\. Riedmiller, A\. K\. Fidjeland, G\. Ostrovski,et al\.\(2015\)Human\-level control through deep reinforcement learning\.Nature518\(7540\),pp\. 529–533\.Cited by:[§2](https://arxiv.org/html/2605.23930#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2605.23930#S4.SS2.p1.4)\.
- F\. A\. Oliehoek and C\. Amato \(2016\)A concise introduction to decentralized POMDPs\.Springer\.Cited by:[§2](https://arxiv.org/html/2605.23930#S2.SS0.SSS0.Px4.p1.1),[§4\.4](https://arxiv.org/html/2605.23930#S4.SS4.p1.1)\.
- T\. Rashid, M\. Samvelyan, C\. Schroeder, G\. Farquhar, J\. Foerster, and S\. Whiteson \(2018\)QMIX: monotonic value function factorisation for deep multi\-agent reinforcement learning\.InInternational Conference on Machine Learning \(ICML\),pp\. 4295–4304\.Cited by:[§2](https://arxiv.org/html/2605.23930#S2.SS0.SSS0.Px4.p1.1)\.
- M\. Samvelyan, T\. Rashid, C\. S\. de Witt, G\. Farquhar, N\. Nardelli, T\. G\. J\. Rudner, C\. Lyu, P\. H\. S\. Torr, J\. Foerster, and S\. Whiteson \(2019\)The StarCraft multi\-agent challenge\.InInternational Conference on Autonomous Agents and Multi\-Agent Systems \(AAMAS\),pp\. 2186–2188\.Cited by:[§2](https://arxiv.org/html/2605.23930#S2.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2605.23930#S2.SS0.SSS0.Px5.p1.1)\.
- J\. Schulman, P\. Moritz, S\. Levine, M\. Jordan, and P\. Abbeel \(2016\)High\-dimensional continuous control using generalized advantage estimation\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§4\.4](https://arxiv.org/html/2605.23930#S4.SS4.SSS0.Px3.p1.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§2](https://arxiv.org/html/2605.23930#S2.SS0.SSS0.Px4.p1.1)\.
- D\. Silver, A\. Huang, C\. J\. Maddison, A\. Guez, L\. Sifre, G\. Van Den Driessche, J\. Schrittwieser, I\. Antonoglou, V\. Panneershelvam, M\. Lanctot,et al\.\(2016\)Mastering the game of Go with deep neural networks and tree search\.Nature529\(7587\),pp\. 484–489\.Cited by:[§2](https://arxiv.org/html/2605.23930#S2.SS0.SSS0.Px1.p1.1)\.
- G\. Smith and J\. Whitehead \(2010\)Launchpad: a game description language for procedural content generation\.InFoundations of Digital Games,pp\. 43–50\.Cited by:[§2](https://arxiv.org/html/2605.23930#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Tan \(1993\)Multi\-agent reinforcement learning: independent vs\. cooperative agents\.pp\. 330–337\.Cited by:[§2](https://arxiv.org/html/2605.23930#S2.SS0.SSS0.Px3.p1.1),[§4\.3](https://arxiv.org/html/2605.23930#S4.SS3.p1.3)\.
- M\. Towers, J\. K\. Terry, A\. Kwiatkowski, J\. U\. Balis, G\. de Cola, T\. Deleu, M\. Goulão, A\. Kallinteris, A\. KG, M\. Krimmel,et al\.\(2023\)Gymnasium: a standard interface for reinforcement learning environments\.InarXiv preprint arXiv:2407\.17032,Cited by:[1st item](https://arxiv.org/html/2605.23930#S1.I2.i1.p1.1),[§3\.6](https://arxiv.org/html/2605.23930#S3.SS6.p1.1)\.
- C\. J\. C\. H\. Watkins and P\. Dayan \(1992\)Q\-learning\.Machine Learning8\(3–4\),pp\. 279–292\.Cited by:[§4\.1](https://arxiv.org/html/2605.23930#S4.SS1.p1.2)\.
- C\. Yu, A\. Velu, E\. Vinitsky, J\. Gao, Y\. Wang, A\. Bayen, and Y\. Wu \(2022\)The surprising effectiveness of PPO in cooperative multi\-agent games\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.35,pp\. 24611–24624\.Cited by:[§2](https://arxiv.org/html/2605.23930#S2.SS0.SSS0.Px4.p1.1),[§4\.4](https://arxiv.org/html/2605.23930#S4.SS4.p1.1),[§6\.5](https://arxiv.org/html/2605.23930#S6.SS5.p1.1)\.

## Appendix AHyperparameters

Table 5:Full hyperparameter settings for all algorithms\.ParameterQ\-LearningDQN / IDQNMAPPOLearning rateα\\alpha/ lr0\.110−310^\{\-3\}3×10−43\\times 10^\{\-4\}Discountγ\\gamma0\.990\.990\.99ε\\varepsilonstart1\.01\.0, \(entropy\)ε\\varepsilonend0\.010\.05,ε\\varepsilonschedule×0\.9995\\times 0\.9995/eplinear 30%,Replay buffer size,100 0004 096 \(rollout\)Batch size,128512Target update interval,1 000 steps,Train frequencyevery stepevery 4 stepsevery rolloutRollout stepsTT,,128PPO clipϵ\\epsilon,,0\.2GAEλ\\lambda,,0\.95PPO epochs,,4Value coefficient,,0\.5Entropy coefficient,,0\.01Gradient norm clip,10\.00\.5Parallel environments13232Network architectureTableMLP 256×\\times2MLP 256×\\times2Total training steps20k–50k ep150k–200k300 000Seeds per stage144