Uncertainty-Aware RL-Controlled Adaptive 3D Mapping

# Uncertainty-Aware RL-Controlled Adaptive 3D Mapping
Source: [https://arxiv.org/html/2610.00188](https://arxiv.org/html/2610.00188)
UnRL: Adaptive 3D Mapping

###### Abstract

Voxel\-based volumetric mapping is fundamental to 3D reconstruction, yet fixed\-resolution grids remain inherently inefficient – wasting memory in uniform regions and losing detail in complex ones\. Existing adaptive methods, such as MAP\-ADAPT, partially address this by varying resolution based on geometry and user\-defined semantic class lists, but these heuristics require expert tuning, lack generalization to unseen objects, and provide no explicit mechanism to control memory usage\. We propose an adaptive framework that refines voxels based on semantic entropy, which captures label uncertainty, together with geometric curvature and texture richness as scene complexity cues, yielding principled resolution allocation without reliance on semantic taxonomies\. To make the accuracy\-memory trade\-off explicit and user\-controlled, we further introduce a reinforcement learning agent that learns voxel subdivision policies under a user\-specified target memory budget, replacing hand\-tuned thresholds with a single intuitive control parameter\. The resulting multi\-resolution TSDF achieves higher geometric accuracy, better semantic consistency, and improved memory\-accuracy trade\-offs compared to MAP\-ADAPT and fixed\-resolution baselines on both synthetic and real\-world datasets\. Our code and models are available at[https://github\.com/alpayozkan/UnRL](https://github.com/alpayozkan/UnRL)\.

††email:aoezkan@inf\.ethz\.ch††email:tunc\.aydin@inf\.ethz\.ch††email:marc\.pollefeys@inf\.ethz\.ch††email:jelena\.trisovic@inf\.ethz\.ch††email:danielbela\.barath@inf\.ethz\.ch††affiliation:ETH Zurich
Zurich, Switzerland††affiliation:Microsoft
Zurich, Switzerland††affiliation:ETH AI Center
Zurich, SwitzerlandEqual supervision

## 1Introduction

Autonomous systems rely on 3D maps that capture geometry, color, and semantics to perceive, navigate, and interact with their environment\. Volumetric representations based on Truncated Signed Distance Fields \(TSDFs\)\[[Newcombe et al\.\(2011\)Newcombe, Izadi, Hilliges, Molyneaux, Kim, Davison, Kohli, Shotton, Hodges, and Fitzgibbon](https://arxiv.org/html/2610.00188#bib.bibx17),[Nießner et al\.\(2013\)Nießner, Zollhöfer, Izadi, and Stamminger](https://arxiv.org/html/2610.00188#bib.bibx18),[Oleynikova et al\.\(2017\)Oleynikova, Taylor, Fehr, Siegwart, and Nieto](https://arxiv.org/html/2610.00188#bib.bibx19)\]have become the standard for dense reconstruction, offering accurate geometry and consistent integration over time\. However, conventional TSDF pipelines employ a*fixed voxel resolution*, which leads to redundant storage in large homogeneous regions \(*e\.g*, walls or floors\) and loss of detail in complex areas\.

![Refer to caption](https://arxiv.org/html/2610.00188v1/UnRLTeaser.png)Figure 1:Given RGB\-D input frames, our method computes semantic entropy, geometric complexity, and texture complexity as voxel\-level subdivision signals\. Semantic entropy highlights object boundaries where multiple labels compete; geometric complexity responds to corners and curved surfaces \(*e\.g*, sofa armrests\); and texture complexity activates on visually detailed regions such as carpets\. These complexity cues drive selective voxel refinement, while a reinforcement learning agent uses a user\-specified target memory budget to balance reconstruction fidelity and storage cost\. The resulting multi\-resolution TSDF mesh preserves high\-quality geometry, semantics, and appearance where it matters most\.Recent work has therefore shifted toward*adaptive mapping*, where voxel resolution is adjusted according to scene complexity\. MAP\-ADAPT\[[Zheng et al\.\(2024\)Zheng, Barath, Pollefeys, and Armeni](https://arxiv.org/html/2610.00188#bib.bibx31)\]represents an important step in this direction, combining geometric cues and manually defined semantic classes to control voxel subdivision\. By allocating higher resolution to semantically meaningful regions, it achieves compact multi\-resolution 3D voxel maps while maintaining fidelity\. However, its refinement policy remains tied to user\-specified semantic classes and fixed geometric rules, requiring dataset\- or task\-specific design choices and providing no class\-agnostic measure of semantic ambiguity or texture complexity\. Moreover, these deterministic rules provide no principled mechanism to manage the trade\-off between reconstruction quality and limited storage capacity\.

We address these limitations by reformulating adaptive mapping as a*data\-driven refinement problem*guided by semantic uncertainty and geometric/texture complexity\. Each voxel maintains a semantic class distribution whose entropy quantifies label uncertainty, complemented by geometric curvature and texture richness scores that capture local scene complexity\. Subdivision is triggered when semantic uncertainty is high or when scene complexity indicates that coarse resolution cannot faithfully represent the local geometry or appearance, ensuring that memory is devoted to informative regions rather than predefined object categories\. This data\-driven refinement produces adaptive maps that respond dynamically to scene complexity, semantic ambiguity, and visual richness\.

While these signals provide a principled refinement criterion, the deterministic variant relies on threshold selection and provides no direct mechanism to target a desired memory operating point\. To overcome this, we introduce a*reinforcement learning agent*that learns to control subdivision under a user\-defined*target storage budget*\. The agent observes local and global voxel statistics and learns subdivision policies that maximize reconstruction accuracy while penalizing budget overuse \(see[Figure1](https://arxiv.org/html/2610.00188#S1.F1)\)\. The target budget corresponds directly to a physical memory amount \(*e\.g*, in megabytes\), providing an intuitive interface for resource\-aware mapping\.

Our main contributions are as follows:

- •Unified adaptive mapping framework:We integrate geometric, semantic, and texture complexity signals into a multi\-resolution TSDF representation that generalizes beyond class\-based heuristics and scales with available memory\.
- •Uncertainty\- and complexity\-driven adaptivity:We trigger voxel refinement using semantic entropy as a measure of label uncertainty, complemented by geometric curvature and texture richness as scene complexity signals, providing a unified and data\-driven subdivision criterion\.
- •Reinforcement\-learned subdivision policy:We train an RL agent using Proximal Policy Optimization \(PPO\)\[[Schulman et al\.\(2017\)Schulman, Wolski, Dhariwal, Radford, and Klimov](https://arxiv.org/html/2610.00188#bib.bibx24)\]to decide when and where to subdivide, learning budget\-aware policies that trade map fidelity against storage cost\.

## 2Related Work

Volumetric Mapping\.Volumetric 3D reconstruction based on Truncated Signed Distance Fields \(TSDFs\) is widely used for dense scene mapping\[[Newcombe et al\.\(2011\)Newcombe, Izadi, Hilliges, Molyneaux, Kim, Davison, Kohli, Shotton, Hodges, and Fitzgibbon](https://arxiv.org/html/2610.00188#bib.bibx17),[Nießner et al\.\(2013\)Nießner, Zollhöfer, Izadi, and Stamminger](https://arxiv.org/html/2610.00188#bib.bibx18),[Oleynikova et al\.\(2017\)Oleynikova, Taylor, Fehr, Siegwart, and Nieto](https://arxiv.org/html/2610.00188#bib.bibx19)\]\. These methods integrate RGB\-D observations into voxel grids, providing continuous surface representations and free\-space reasoning\. To mitigate the memory footprint of dense grids, hierarchical data structures such as voxel hashing\[[Nießner et al\.\(2013\)Nießner, Zollhöfer, Izadi, and Stamminger](https://arxiv.org/html/2610.00188#bib.bibx18),[Kähler et al\.\(2016\)Kähler, Prisacariu, Valentin, and Murray](https://arxiv.org/html/2610.00188#bib.bibx8)\]and octrees\[[Hornung et al\.\(2013\)Hornung, Wurm, Bennewitz, Stachniss, and Burgard](https://arxiv.org/html/2610.00188#bib.bibx6),[Vespa et al\.\(2019\)Vespa, Funk, Kelly, and Leutenegger](https://arxiv.org/html/2610.00188#bib.bibx27)\]have been proposed to allocate voxels only near surfaces\. While these systems achieve real\-time performance and scalable mapping, they typically rely on a*fixed voxel resolution*, which limits their ability to represent both fine local details and large\-scale environments efficiently\.*Neural implicit methods*such as NeRF\[[Mildenhall et al\.\(2020\)Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng](https://arxiv.org/html/2610.00188#bib.bibx15)\]and 3D Gaussian Splatting\[[Kerbl et al\.\(2023\)Kerbl, Kopanas, Leimkühler, and Drettakis](https://arxiv.org/html/2610.00188#bib.bibx9)\]target image rendering rather than explicit volumetric mapping, and do not produce TSDF grids, semantic labels, or free\-space representations required for planning and navigation\.

Semantic Mapping\.Integrating semantic understanding into 3D mapping has been central to advancing scene perception\. Early works like SemanticFusion\[[McCormac et al\.\(2017\)McCormac, Handa, Davison, and Leutenegger](https://arxiv.org/html/2610.00188#bib.bibx12)\]fused per\-frame semantic predictions into surfel\-based reconstructions, while Kimera\[[Rosinol et al\.\(2020\)Rosinol, Abate, Chang, and Carlone](https://arxiv.org/html/2610.00188#bib.bibx22)\]combined real\-time metric reconstruction with semantic fusion into a unified volumetric mesh\. Instance\- and panoptic\-level methods\[[McCormac et al\.\(2018\)McCormac, Clark, Bloesch, Davison, and Leutenegger](https://arxiv.org/html/2610.00188#bib.bibx13),[Grinvald et al\.\(2019\)Grinvald, Furrer, Novkovic, Chung, Cadena, Siegwart, and Nieto](https://arxiv.org/html/2610.00188#bib.bibx5),[Narita et al\.\(2019\)Narita, Seno, Ishikawa, and Kaji](https://arxiv.org/html/2610.00188#bib.bibx16),[Wu et al\.\(2021\)Wu, Wald, Tateno, Navab, and Tombari](https://arxiv.org/html/2610.00188#bib.bibx28)\]further extended these ideas by maintaining per\-object volumes or combining instance labels\. However, these pipelines generally operate at uniform spatial resolution, distributing computational and memory resources equally across all regions, independent of their semantic or geometric complexity\.

Adaptive Mapping\.To improve efficiency, adaptive techniques dynamically adjust voxel resolution based on scene content or task relevance\.*Geometry\-based adaptivity*refines regions of high curvature, surface density, or structural complexity\[[Steinbrücker et al\.\(2014\)Steinbrücker, Sturm, and Cremers](https://arxiv.org/html/2610.00188#bib.bibx25),[Stückler and Behnke\(2014\)](https://arxiv.org/html/2610.00188#bib.bibx26),[Vespa et al\.\(2019\)Vespa, Funk, Kelly, and Leutenegger](https://arxiv.org/html/2610.00188#bib.bibx27),[Zienkiewicz et al\.\(2016\)Zienkiewicz, Tsiotsios, Davison, and Leutenegger](https://arxiv.org/html/2610.00188#bib.bibx33)\]\. These approaches improve geometric fidelity near surfaces but lack semantic or visual awareness\.*Semantic\-based adaptivity*extends this idea by incorporating object information to guide subdivision\. Panoptic Multi\-TSDFs\[[Schmid et al\.\(2022\)Schmid, Delmerico, Schönberger, Nieto, Pollefeys, Siegwart, and Cadena](https://arxiv.org/html/2610.00188#bib.bibx23)\]reconstruct separate TSDF volumes per object instance, while MAP\-ADAPT\[[Zheng et al\.\(2024\)Zheng, Barath, Pollefeys, and Armeni](https://arxiv.org/html/2610.00188#bib.bibx31)\]combines geometric and semantic cues into a single multi\-resolution TSDF map\. MAP\-ADAPT demonstrates real\-time adaptive reconstruction by refining user\-specified semantic classes, but its reliance on manually defined rules limits generalization to unseen categories and neglects uncertainty in semantic or textural predictions that could indicate ambiguous or informative regions\.

Uncertainty and Learning\-Based Adaptivity\.Several works have incorporated uncertainty into mapping, active exploration, and reconstruction\. SemanticFusion\[[McCormac et al\.\(2017\)McCormac, Handa, Davison, and Leutenegger](https://arxiv.org/html/2610.00188#bib.bibx12)\]and Kimera\[[Rosinol et al\.\(2020\)Rosinol, Abate, Chang, and Carlone](https://arxiv.org/html/2610.00188#bib.bibx22)\]maintain probabilistic class distributions per voxel, though uncertainty is not used to influence map structure\. Information\-theoretic formulations\[[Charrow et al\.\(2015\)Charrow, Liu, Kumar, and Michael](https://arxiv.org/html/2610.00188#bib.bibx2),[Asgharivaskasi and Atanasov\(2023\)](https://arxiv.org/html/2610.00188#bib.bibx1)\]employ entropy and mutual information to guide exploration; notably, semantic entropy within a Bayesian Octree has been used to drive active exploration via mutual information maximization\[[Asgharivaskasi and Atanasov\(2023\)](https://arxiv.org/html/2610.00188#bib.bibx1)\]\. Reinforcement learning has similarly been applied to active exploration and mapping\[[Zhu et al\.\(2017\)Zhu, Mottaghi, Kolve, Lim, Gupta, Fei\-Fei, and Farhadi](https://arxiv.org/html/2610.00188#bib.bibx32),[Chen et al\.\(2019\)Chen, Bai, Shan, and Englot](https://arxiv.org/html/2610.00188#bib.bibx3),[Placed and Castellanos\(2020\)](https://arxiv.org/html/2610.00188#bib.bibx21)\], demonstrating that learned policies can balance performance and efficiency\. However, these exploration\-focused methods determine*where to move next*rather than*how to allocate reconstruction resolution*along a fixed trajectory\. In contrast, we formulate voxel subdivision as an*uncertainty\-driven*refinement problem and leverage reinforcement learning to learn adaptive, budget\-aware subdivision policies that directly regulate the trade\-off between reconstruction fidelity and storage capacity\.

![Refer to caption](https://arxiv.org/html/2610.00188v1/overviewUnRL.png)Figure 2:Overview\.\(a\) RGB\-D frames are fed into \(b\) a preprocessing stage that estimates camera poses via SLAM \(similar to\[[Zheng et al\.\(2024\)Zheng, Barath, Pollefeys, and Armeni](https://arxiv.org/html/2610.00188#bib.bibx31)\]\), performs 2D semantic segmentation\[[Lin et al\.\(2017\)Lin, Milan, Shen, and Reid](https://arxiv.org/html/2610.00188#bib.bibx11)\], and computes per\-voxel complexity cues \(semantic, geometric, and texture\) that highlight object boundaries, geometric curvature, and textured regions\. \(c\) Our adaptive mapping refines map resolution either through deterministic complexity thresholds \(*Un\-Map\-Adapt*\) or through a reinforcement\-learning policy \(*UnRL*\) allocating resolution under a user\-specified target memory budget\. The RL agent predicts subdivision thresholds for budget\-aware refinement\. \(d\) The final multi\-resolution TSDF map is converted into a high\-fidelity mesh\.
## 3Proposed Method

We propose an adaptive 3D semantic mapping framework that refines voxel resolution by explicitly targeting high\-uncertainty regions\. We build upon MAP\-ADAPT\[[Zheng et al\.\(2024\)Zheng, Barath, Pollefeys, and Armeni](https://arxiv.org/html/2610.00188#bib.bibx31)\], which operates on an adaptively subdivided voxel grid and controls subdivision via manually defined semantic classes and geometric complexity\. In contrast, we \(i\) reformulate voxel subdivision as an uncertainty\-driven refinement problem, using semantic, geometric, and textural complexity as proxies to trigger adaptive resolution allocation, and \(ii\) learn a budget\-aware, soft\-constrained subdivision policy with reinforcement learning \(RL\) that replaces hand\-tuned thresholds with a single intuitive parameter: the target storage budget\.

Preliminaries\.We consider a sequence of RGB\-D frames\{\(Ik,Dk\)\}k=1K\\\{\(I\_\{k\},D\_\{k\}\)\\\}\_\{k=1\}^\{K\}with camera poses\{Ck\}\\\{C\_\{k\}\\\}and per\-frame semantic predictions\{Pk\}\\\{P\_\{k\}\\\}, whereIkI\_\{k\}andDkD\_\{k\}are the input image and depth in thekk\-th frame\. The goal is to reconstruct an adaptive\-resolution TSDF mapℳ\\mathcal\{M\}that allocates fine resolution only where it meaningfully reduces uncertainty\. We adopt a hierarchical voxel hashing structure in the spirit of MAP\-ADAPT\[[Zheng et al\.\(2024\)Zheng, Barath, Pollefeys, and Armeni](https://arxiv.org/html/2610.00188#bib.bibx31)\], with three resolution levels \(coarse:88cm,medium:44cm,fine:11cm voxels\)\. Each voxelviv\_\{i\}storesvi=\{di,Si,gi,ti\}v\_\{i\}=\\\{d\_\{i\},S\_\{i\},g\_\{i\},t\_\{i\}\\\}, wheredi∈ℝd\_\{i\}\\in\\mathbb\{R\}is the TSDF value,Si∈ℝCS\_\{i\}\\in\\mathbb\{R\}^\{C\}denotes the semantic log\-scores overCCclasses,gi∈ℝg\_\{i\}\\in\\mathbb\{R\}a geometric complexity scalar, andti∈ℝt\_\{i\}\\in\\mathbb\{R\}a texture complexity scalar\. Semantic entropyHisemH\_\{i\}^\{\\text\{sem\}\}is derived on\-the\-fly from the normalized semantic distributionP^isem\\hat\{P\}\_\{i\}^\{\\text\{sem\}\}obtained fromSiS\_\{i\}, while the combined uncertainty proxyhih\_\{i\}used by the RL agent is computed from all three modalities as defined in[Section3\.3](https://arxiv.org/html/2610.00188#S3.SS3)\. The pipeline is shown in[Figure2](https://arxiv.org/html/2610.00188#S2.F2)\.

### 3\.1Uncertainty\-Driven Adaptive Mapping

We propose our uncertainty\-aware adaptive mapping pipeline Un\-Map\-Adapt\.

Voxel uncertainty and complexity measures\.Each voxelviv\_\{i\}maintains a semantic class distribution; geometric and texture complexity are fused as scalar attributes computed in the observation domain\. Semantic entropy captures label uncertainty across observations, motivating refinement at class boundaries; geometric curvature identifies edges and corners where coarse voxels lose structural detail; and texture richness flags visually dense regions where low resolution causes blurring\.

*Semantic entropy\.*In each voxel, we fuse per\-frame class probabilities into semantic log\-scoresSiS\_\{i\}using a Bayesian update similar to SemanticFusion\[[McCormac et al\.\(2017\)McCormac, Handa, Davison, and Leutenegger](https://arxiv.org/html/2610.00188#bib.bibx12)\]and Kimera\[[Rosinol et al\.\(2020\)Rosinol, Abate, Chang, and Carlone](https://arxiv.org/html/2610.00188#bib.bibx22)\]\. After fusion, the normalized class probabilities are as follows:

P^isem\(c\)=exp⁡\(Si​\(c\)\)∑c′=1Cexp⁡\(Si​\(c′\)\),c=1,…,C\.\\hat\{P\}\_\{i\}^\{\\text\{sem\}\}\(c\)=\\frac\{\\exp\(S\_\{i\}\(c\)\)\}\{\\sum\_\{c^\{\\prime\}=1\}^\{C\}\\exp\(S\_\{i\}\(c^\{\\prime\}\)\)\},\\quad c=1,\\dots,C\.\(1\)The semantic entropy is then as follows:

Hisem=−∑c=1CP^isem\(c\)logP^isem\(c\)\.H\_\{i\}^\{\\text\{sem\}\}=\-\\sum\_\{c=1\}^\{C\}\\hat\{P\}\_\{i\}^\{\\text\{sem\}\}\(c\)\\log\\hat\{P\}\_\{i\}^\{\\text\{sem\}\}\(c\)\.\(2\)
We use entropy as a class\-agnostic measure of predictive ambiguity: it increases when probability mass is distributed across competing labels and does not depend on which semantic class is most likely, thereby highlighting regions that may benefit from finer resolution\.

*Geometric complexity\.*We estimate geometric complexity using the change\-of\-curvature \(CoC\) descriptor\[[Pauly et al\.\(2002\)Pauly, Gross, and Kobbelt](https://arxiv.org/html/2610.00188#bib.bibx20)\], similar to\[[Zheng et al\.\(2024\)Zheng, Barath, Pollefeys, and Armeni](https://arxiv.org/html/2610.00188#bib.bibx31)\]\. For each depth frame, pixels are back\-projected to 3D at stride 2 for efficiency, producing a subsampled point cloud\. For each 3D pointpp, we find all neighbors within radiusr=0\.1r\{=\}0\.1m using a KD\-tree and compute the local covariance matrix as

𝐂⁡\(p\)=1n​∑i=1n\(pi−p¯\)​\(pi−p¯\)⊤,\\mathbf\{C\}\(p\)=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\(p\_\{i\}\-\\bar\{p\}\)\(p\_\{i\}\-\\bar\{p\}\)^\{\\top\},\(3\)wherep¯\\bar\{p\}is the local centroid andnnis the number of neighbors\. Letλ1≥λ2≥λ3≥0\\lambda\_\{1\}\\geq\\lambda\_\{2\}\\geq\\lambda\_\{3\}\\geq 0be the eigenvalues of𝐂⁡\(p\)\\mathbf\{C\}\(p\); the CoC is defined as:CoC​\(p\)=λ3/\(λ1\+λ2\+λ3\)∈\[0,0\.33\]\\text\{CoC\}\(p\)=\\lambda\_\{3\}/\(\\lambda\_\{1\}\+\\lambda\_\{2\}\+\\lambda\_\{3\}\)\\in\[0,0\.33\], whereCoC≈0\\text\{CoC\}\{\\approx\}0indicates a locally planar surface andCoC≈0\.33\\text\{CoC\}\{\\approx\}0\.33indicates an isotropic, maximally complex neighborhood\. The per\-point CoC values are stored at their original pixel coordinates\(u,v\)\(u,v\), producing a per\-frame 2D complexity map\. These per\-observation values are then fused into each voxelviv\_\{i\}as a*single scalar*gig\_\{i\}via confidence\-weighted accumulation \(weightwj=1/zj2w\_\{j\}=1/z\_\{j\}^\{2\}, wherezjz\_\{j\}is the observation depth, downweighting distant measurements\) as:

gi←wi​gi\+wj​CoC​\(pj\)wi\+wj\.g\_\{i\}\\leftarrow\\frac\{w\_\{i\}\\,g\_\{i\}\+w\_\{j\}\\,\\text\{CoC\}\(p\_\{j\}\)\}\{w\_\{i\}\+w\_\{j\}\}\.\(4\)Highgig\_\{i\}indicates geometrically complex or curved regions, such as edges and corners, that benefit from finer voxel resolution\.

*Texture complexity\.*We compute a per\-pixel texture complexity map for each RGB frame using local gradient entropy\. We apply3×33\{\\times\}3Sobel filters to obtain horizontal and vertical gradientsGxG\_\{x\},GyG\_\{y\}, and compute per\-pixel gradient magnitudesm⁡\(u,v\)=Gx​\(u,v\)2\+Gy​\(u,v\)2m\(u,v\)=\\sqrt\{G\_\{x\}\(u,v\)^\{2\}\+G\_\{y\}\(u,v\)^\{2\}\}\. For each pixel\(u,v\)\(u,v\), we extract a5×55\{\\times\}5neighborhood \(with reflect padding at borders\), yielding 25 gradient magnitude samples\. These are binned into a 20\-bin histogram spanning the local\[mmin,mmax\]\[m\_\{\\min\},m\_\{\\max\}\]range of the window, normalized to a probability distribution\{pj\}j=120\\\{p\_\{j\}\\\}\_\{j=1\}^\{20\}, and Shannon entropy is computed as:

Htex\(u,v\)=−∑j=120pjlog\(pj\+ε\),ε=10−6\.H^\{\\text\{tex\}\}\(u,v\)=\-\\sum\_\{j=1\}^\{20\}p\_\{j\}\\log\(p\_\{j\}\+\\varepsilon\),\\quad\\varepsilon=10^\{\-6\}\.\(5\)Intuitively, flat textureless regions concentrate gradients in a single bin \(low entropy\), while richly textured regions spread mass across bins \(high entropy\)\. The resulting entropy map is z\-normalized per image over valid pixels as:

t⁡\(u,v\)=Htex​\(u,v\)−μHσH,t\(u,v\)=\\frac\{H^\{\\text\{tex\}\}\(u,v\)\-\\mu\_\{H\}\}\{\\sigma\_\{H\}\},\(6\)whereμH\\mu\_\{H\}andσH\\sigma\_\{H\}are the mean and standard deviation computed over all valid pixels in the frame, making complexity values comparable across images with different overall texture levels\. Positive per\-pixel values are then projected into 3D and fused into each voxelviv\_\{i\}as a*single scalar*tit\_\{i\}via confidence\-weighted accumulation \(weightwj=1/zj2w\_\{j\}=1/z\_\{j\}^\{2\}\) asti←\(wi​ti\+wj​t​\(uj,vj\)\)/\(wi\+wj\)t\_\{i\}\\leftarrow\(w\_\{i\}\\,t\_\{i\}\+w\_\{j\}\\,t\(u\_\{j\},v\_\{j\}\)\)/\(w\_\{i\}\+w\_\{j\}\)\. Hightit\_\{i\}indicates visually rich regions like boundaries and textured surfaces that are prioritized for subdivision even when geometric complexity is low\.

Deterministic complexity\-threshold subdivision\.MAP\-ADAPT subdivides voxels according to manually specified semantic classes and geometric complexity thresholds\. We replace these class\-based rules with per\-modality complexity thresholds\. For a voxel at levelℓ∈\{coarse,medium\}\\ell\\in\\\{\\text\{coarse\},\\text\{medium\}\\\}, subdivision is triggered if*any*of the following holds:Hisem\>τℓsemH\_\{i\}^\{\\text\{sem\}\}\>\\tau\_\{\\ell\}^\{\\text\{sem\}\},gi\>τℓgeog\_\{i\}\>\\tau\_\{\\ell\}^\{\\text\{geo\}\},ti\>τℓtext\_\{i\}\>\\tau\_\{\\ell\}^\{\\text\{tex\}\}\. If subdivision is triggered, the parentviv\_\{i\}is replaced bynℓn\_\{\\ell\}of its children\{vi\(k\)\}k=1nℓ\\\{v\_\{i\}^\{\(k\)\}\\\}\_\{k=1\}^\{n\_\{\\ell\}\}, whose initial TSDF and semantic fields are interpolated from the parent and neighboring voxels\. This can be interpreted as refining voxels in which at least one modality exhibits high complexity\. The semantic threshold is shared across both levels, while the geometry and texture thresholds are level\-specific for the coarse\-to\-medium and medium\-to\-fine transitions\. Defaults areτsem=0\.10\\tau^\{\\text\{sem\}\}\{=\}0\.10,τlogeo=0\.05\\tau^\{\\text\{geo\}\}\_\{\\text\{lo\}\}\{=\}0\.05,τhigeo=0\.10\\tau^\{\\text\{geo\}\}\_\{\\text\{hi\}\}\{=\}0\.10,τlotex=1\.0\\tau^\{\\text\{tex\}\}\_\{\\text\{lo\}\}\{=\}1\.0,τhitex=2\.0\\tau^\{\\text\{tex\}\}\_\{\\text\{hi\}\}\{=\}2\.0, calibrated by inspecting complexity distributions on a held\-out training scene\. This deterministic variant removes dependence on predefined class lists from\[[Zheng et al\.\(2024\)Zheng, Barath, Pollefeys, and Armeni](https://arxiv.org/html/2610.00188#bib.bibx31)\], but it still requires hand\-tuning thresholds and is not budget\-aware\.[Table9](https://arxiv.org/html/2610.00188#A2.T9)analyzes sensitivity to these thresholds, showing that their selection jointly determines the accuracy–memory operating point\.

### 3\.2TSDF Integration and Field Updates

For each framekk, we lift pixels\(u,v\)\(u,v\)with depthzzinto 3D pointsp=π−1​\(u,v,z\)p=\\pi^\{\-1\}\(u,v,z\)and cast rays from the camera center toppinto the voxel grid\. Voxels whose centers fall within a truncation band\[z−μ,z\+μ\]\[z\-\\mu,z\+\\mu\]are updated\. Each observationjjcontributes with weightwj=1/zj2w\_\{j\}=1/z\_\{j\}^\{2\}\(downweighting distant measurements\)\. For a voxelviv\_\{i\}, the TSDF valuedid\_\{i\}and its weightwiw\_\{i\}are updated as

di←wi​di\+wj​djwi\+wj,wi←wi\+wj\.d\_\{i\}\\leftarrow\\frac\{w\_\{i\}d\_\{i\}\+w\_\{j\}d\_\{j\}\}\{w\_\{i\}\+w\_\{j\}\},\\qquad w\_\{i\}\\leftarrow w\_\{i\}\+w\_\{j\}\.\(7\)
#### Semantic fusion \(logit accumulation\)\.

Instead of directly accumulating probabilities, each voxel maintains a vector of*log\-scores*\(logits\)Si∈ℝCS\_\{i\}\\in\\mathbb\{R\}^\{C\}\. Given a per\-frame class probability vectorPj∈\[0,1\]CP\_\{j\}\\in\[0,1\]^\{C\}at the projected point \(with∑cPj​\(c\)=1\\sum\_\{c\}P\_\{j\}\(c\)=1\), we form an updated logit proposal as

Siprop​\(c\)=Si​\(c\)\+wj​log⁡\(Pj​\(c\)\+ε\),S\_\{i\}^\{\\text\{prop\}\}\(c\)\\;=\\;S\_\{i\}\(c\)\+w\_\{j\}\\,\\log\\\!\\big\(P\_\{j\}\(c\)\+\\varepsilon\\big\),\(8\)wherewjw\_\{j\}is the observation weight andε\\varepsilonis a small constant to avoidlog⁡0\\log 0\. We apply an exponential moving average:

Si​\(c\)←\(1−λ\)​Si​\(c\)\+λ​Siprop​\(c\),S\_\{i\}\(c\)\\;\\leftarrow\\;\(1\-\\lambda\)\\,S\_\{i\}\(c\)\\;\+\\;\\lambda\\,S\_\{i\}^\{\\text\{prop\}\}\(c\),\(9\)with fusion rateλ∈\(0,1\]\\lambda\\in\(0,1\]\. Settingλ=1\\lambda=1recovers simple additive accumulationSi←Si\+wj​log⁡PjS\_\{i\}\\leftarrow S\_\{i\}\+w\_\{j\}\\log P\_\{j\}\. To obtain a normalized class distribution for entropy or rendering, we compute the stable softmax as:

P^isem​\(c\)=exp⁡\(Si​\(c\)−mi\)∑c′exp⁡\(Si​\(c′\)−mi\),mi=maxc⁡Si​\(c\)\.\\hat\{P\}\_\{i\}^\{\\text\{sem\}\}\(c\)\\;=\\;\\frac\{\\exp\\\!\\big\(S\_\{i\}\(c\)\-m\_\{i\}\\big\)\}\{\\sum\_\{c^\{\\prime\}\}\\exp\\\!\\big\(S\_\{i\}\(c^\{\\prime\}\)\-m\_\{i\}\\big\)\},\\qquad m\_\{i\}\\;=\\;\\max\_\{c\}S\_\{i\}\(c\)\.\(10\)Finally, semantic entropy is obtained by:

Hisem=−∑c=1CP^isem\(c\)logP^isem\(c\)\.H\_\{i\}^\{\\text\{sem\}\}\\;=\\;\-\\sum\_\{c=1\}^\{C\}\\hat\{P\}\_\{i\}^\{\\text\{sem\}\}\(c\)\\,\\log\\hat\{P\}\_\{i\}^\{\\text\{sem\}\}\(c\)\.\(11\)

### 3\.3Budget\-Aware RL Subdivision Policy \(UnRL\)

Although the uncertainty formulation above is principled, it still requires manual threshold selection and provides no direct mechanism to target a desired global storage budget\. Thus, we cast adaptive subdivision as a sequential decision problem and train an RL agent to choose complexity thresholds per region while penalizing budget overuse\.

Spatial blocking and uncertainty proxy\.For each frame, we cluster voxels into spatial blocks of fixed size \(*e\.g*,0\.5​m0\.5\\,\\text\{m\}\) using a grid index\. Letℬ=\{1,…,B\}\\mathcal\{B\}=\\\{1,\\dots,B\\\}be the set of blocks and𝒱b\\mathcal\{V\}\_\{b\}the voxels assigned to blockbb\. For each voxelvi∈𝒱bv\_\{i\}\\in\\mathcal\{V\}\_\{b\}, we compute a normalized uncertainty proxy ashi=w~sem​H~isem\+w~geo​g~i\+w~tex​t~ih\_\{i\}=\\tilde\{w\}\_\{\\text\{sem\}\}\\tilde\{H\}\_\{i\}^\{\\text\{sem\}\}\+\\tilde\{w\}\_\{\\text\{geo\}\}\\tilde\{g\}\_\{i\}\+\\tilde\{w\}\_\{\\text\{tex\}\}\\tilde\{t\}\_\{i\}, whereH~isem=Hisem/log⁡C\\tilde\{H\}\_\{i\}^\{\\text\{sem\}\}=H\_\{i\}^\{\\text\{sem\}\}/\\log C, withCCthe number of semantic classes,g~i=min⁡\(gi,0\.333\)/0\.333\\tilde\{g\}\_\{i\}=\\min\(g\_\{i\},\\,0\.333\)/0\.333, andt~i=min⁡\(ti,3\.0\)/3\.0\\tilde\{t\}\_\{i\}=\\min\(t\_\{i\},\\,3\.0\)/3\.0denote the per\-voxel semantic entropy, geometric complexity, and texture complexity normalized to\[0,1\]\[0,1\], andw~sem=w~geo=w~tex=0\.33\\tilde\{w\}\_\{\\text\{sem\}\}=\\tilde\{w\}\_\{\\text\{geo\}\}=\\tilde\{w\}\_\{\\text\{tex\}\}=0\.33are fixed equal weights used across all datasets\.

For each blockbb, we compute its mean and standard deviation as:

μb=1\|𝒱b\|​∑i∈𝒱bhi,σb=1\|𝒱b\|​∑i∈𝒱b\(hi−μb\)2\.\\mu\_\{b\}=\\frac\{1\}\{\|\\mathcal\{V\}\_\{b\}\|\}\\sum\_\{i\\in\\mathcal\{V\}\_\{b\}\}h\_\{i\},\\quad\\sigma\_\{b\}=\\sqrt\{\\frac\{1\}\{\|\\mathcal\{V\}\_\{b\}\|\}\\sum\_\{i\\in\\mathcal\{V\}\_\{b\}\}\(h\_\{i\}\-\\mu\_\{b\}\)^\{2\}\}\.\(12\)We then importance\-sampleNNblocks with probabilities proportional toμb\\mu\_\{b\}, focusing the agent on informative regions over large uniform areas\.[Tables6](https://arxiv.org/html/2610.00188#A2.T6)and[7](https://arxiv.org/html/2610.00188#A2.T7)analyze the spatial block size and number of sampled blocks, respectively, showing stable performance around the default configuration\.

Observation space\.The agent observes the statistics ofNNsampled blocks and global summary variables\. The observation state is as follows:

𝐬=\[μ1,σ1,…,μN,σN,h¯,σ⁡\(h\),ρused,ρbudget\],\\mathbf\{s\}=\[\\mu\_\{1\},\\sigma\_\{1\},\\dots,\\mu\_\{N\},\\sigma\_\{N\},\\bar\{h\},\\sigma\(h\),\\rho\_\{\\text\{used\}\},\\rho\_\{\\text\{budget\}\}\],\(13\)whereh¯\\bar\{h\}andσ⁡\(h\)\\sigma\(h\)are the mean and standard deviation ofhih\_\{i\}over all voxels in the frame,ρused∈\[0,1\]\\rho\_\{\\text\{used\}\}\\in\[0,1\]is the fraction of the voxel budget currently consumed, andρbudget\\rho\_\{\\text\{budget\}\}is the fixed target budget ratio for the current agent\.

Action space and subdivision rule\.For each sampled blockbb, the agent predicts a pair of complexity thresholds\(τ0,b,τ1,b\)\(\\tau\_\{0,b\},\\tau\_\{1,b\}\)from a discrete set𝒯=\{0\.0,0\.025,0\.05,0\.1,0\.25,0\.5\}\\mathcal\{T\}=\\\{0\.0,0\.025,0\.05,0\.1,0\.25,0\.5\\\}\. The environment encodes this as a MultiDiscrete action over2​N2Nindices into𝒯\\mathcal\{T\}\. For voxels in blockbb, the predicted thresholds induce a three\-level decision:

level​\(vi\)=\{coarse,hi≤τ0,b,medium,τ0,b<hi≤τ1,b,fine,hi\>τ1,b\.\\text\{level\}\(v\_\{i\}\)=\\begin\{cases\}\\text\{coarse\},&h\_\{i\}\\leq\\tau\_\{0,b\},\\\\ \\text\{medium\},&\\tau\_\{0,b\}<h\_\{i\}\\leq\\tau\_\{1,b\},\\\\ \\text\{fine\},&h\_\{i\}\>\\tau\_\{1,b\}\.\\end\{cases\}\(14\)Each subdivision decision updates a*voxel budget counter*, with coarse, medium, and fine levels incurring different costs; the medium level contributes a fractionαmid∈\(0,1\)\\alpha\_\{\\text\{mid\}\}\\in\(0,1\)of the fine\-level budget cost, serving as a cost\-effective intermediate\.

Supervision losses and reward\.To provide a training signal, we precompute high\-resolution reference reconstructions at1​cm1\\,\\text\{cm\}voxel size using Voxblox\[[Oleynikova et al\.\(2017\)Oleynikova, Taylor, Fehr, Siegwart, and Nieto](https://arxiv.org/html/2610.00188#bib.bibx19)\]\. Each coarse \(88cm\) parent voxel maintains up to838^\{3\}child slots at the finest \(11cm\) resolution, populated from this reference; only children with nonzero observation weight contribute to the loss computation\. For each lower\-resolution voxel, we derive per\-voxel losses by comparing its semantic label, RGB color, and TSDF value to these aggregated child voxels\.

Concretely,ℒsem\\mathcal\{L\}\_\{\\text\{sem\}\}is the fraction of child voxels whose semantic label differs from the parent \(indicator\-based, range\[0,1\]\[0,1\]\);ℒtex\\mathcal\{L\}\_\{\\text\{tex\}\}is the mean L2 RGB distance between parent and children normalized by3\\sqrt\{3\}\(range\[0,1\]\[0,1\]\); andℒgeo\\mathcal\{L\}\_\{\\text\{geo\}\}is the mean L1 TSDF difference normalized by the truncation distance2​μ2\\mu\(range\[0,1\]\[0,1\]\)\. We denote these normalized losses byℒsem,ℒtex,ℒgeo∈\[0,1\]\\mathcal\{L\}\_\{\\text\{sem\}\},\\mathcal\{L\}\_\{\\text\{tex\}\},\\mathcal\{L\}\_\{\\text\{geo\}\}\\in\[0,1\]\. They are combined into a single reconstruction loss as follows:

ℒ=wsem​ℒsem\+wtex​ℒtex\+wgeo​ℒgeo,\\mathcal\{L\}=w\_\{\\text\{sem\}\}\\mathcal\{L\}\_\{\\text\{sem\}\}\+w\_\{\\text\{tex\}\}\\mathcal\{L\}\_\{\\text\{tex\}\}\+w\_\{\\text\{geo\}\}\\mathcal\{L\}\_\{\\text\{geo\}\},\(15\)with equal weightswsem=wtex=wgeo=0\.33w\_\{\\text\{sem\}\}=w\_\{\\text\{tex\}\}=w\_\{\\text\{geo\}\}=0\.33, fixed across all datasets and shared with the proxy weights used inhih\_\{i\}\.[Table8](https://arxiv.org/html/2610.00188#A2.T8)evaluates sensitivity to the reward\-weight configuration and shows that balanced settings yield similar performance\. For voxelviv\_\{i\}, we denote the resulting combined loss byℒi\\mathcal\{L\}\_\{i\}\.

At each step, we compute a per\-voxel reward that encourages subdivision where it reduces reconstruction loss while penalizing unnecessary refinement\. For a voxel assigned to levelℓ∈\{coarse,medium,fine\}\\ell\\in\\\{\\text\{coarse\},\\text\{medium\},\\text\{fine\}\\\}, we use:

ri\(ℓ\)=\{−ℒi,ℓ=coarse,αmid​\(ℒi−λsub\),ℓ=medium,ℒi−λsub,ℓ=fine,r\_\{i\}^\{\(\\ell\)\}=\\begin\{cases\}\-\\mathcal\{L\}\_\{i\},&\\ell=\\text\{coarse\},\\\\ \\alpha\_\{\\text\{mid\}\}\(\\mathcal\{L\}\_\{i\}\-\\lambda\_\{\\text\{sub\}\}\),&\\ell=\\text\{medium\},\\\\ \\mathcal\{L\}\_\{i\}\-\\lambda\_\{\\text\{sub\}\},&\\ell=\\text\{fine\},\\end\{cases\}\(16\)whereλsub\>0\\lambda\_\{\\text\{sub\}\}\>0is a subdivision penalty andαmid∈\(0,1\)\\alpha\_\{\\text\{mid\}\}\\in\(0,1\)downweights the effect of medium resolution relative to fine\. Intuitively, subdividing is only beneficial when the expected improvement in reconstruction outweighs the subdivision cost\. The step reward is obtained by averaging over all updated voxels and computing a temporal difference relative to the previous step to stabilize learning\.

Budget penalty\.LetNlargeN\_\{\\text\{large\}\}be the number of8​cm8\\,\\text\{cm\}voxels required to fill the bounding box of the scene, computed once at episode initialization from the known scene extent\. We define a per\-scene budget asNbudget=ρbudget​NlargeN\_\{\\text\{budget\}\}=\\rho\_\{\\text\{budget\}\}N\_\{\\text\{large\}\}, whereρbudget∈\(0,1\]\\rho\_\{\\text\{budget\}\}\\in\(0,1\]is the fixed budget ratio for the current agent\. Given the fixed storage footprintsvs\_\{v\}\(in bytes\) per voxel, this corresponds to a target memoryMtarget=Nbudget​svM\_\{\\text\{target\}\}=N\_\{\\text\{budget\}\}\\,s\_\{v\},*i\.e*, a target storage budget in MB or GB\. LetNsubN\_\{\\text\{sub\}\}denote the current effective number of subdivided voxels \(medium and fine levels, accounting for their relative cost\)\. We compute the used fraction asρused=min⁡\(Nsub/Nbudget,1\.0\)\\rho\_\{\\text\{used\}\}=\\min\\left\(N\_\{\\text\{sub\}\}/N\_\{\\text\{budget\}\},\\,1\.0\\right\)\. A penalty is applied whenρused\\rho\_\{\\text\{used\}\}exceeds an offsetρ0\\rho\_\{0\}\(*e\.g*,0\.70\.7of the allocated memory\) as:

rbudget=\{0,ρused≤ρ0,βbudget​\(ρused−ρ0\),ρused\>ρ0,r\_\{\\text\{budget\}\}=\\begin\{cases\}0,&\\rho\_\{\\text\{used\}\}\\leq\\rho\_\{0\},\\\\ \\beta\_\{\\text\{budget\}\}\\,\(\\rho\_\{\\text\{used\}\}\-\\rho\_\{0\}\),&\\rho\_\{\\text\{used\}\}\>\\rho\_\{0\},\\end\{cases\}\(17\)whereβbudget<0\\beta\_\{\\text\{budget\}\}<0controls the severity of overuse\. The total reward becomesrt=r¯t\+rbudgetr\_\{t\}=\\overline\{r\}\_\{t\}\+r\_\{\\text\{budget\}\}, withr¯t\\overline\{r\}\_\{t\}denoting the mean voxel\-level reward\. This formulation applies a*budget\-aware soft constraint*: rather than a formal hard constraint with satisfaction guarantees, the penalty makes sustained budget violations suboptimal during training, empirically yielding tight budget adherence at inference\.

Training\.We train the policy using PPO\[[Schulman et al\.\(2017\)Schulman, Wolski, Dhariwal, Radford, and Klimov](https://arxiv.org/html/2610.00188#bib.bibx24)\], following a reinforcement\-learning formulation in which each episode corresponds to a complete scene and the agent sequentially selects thresholds for blocks across frames\. We train separate agents for different fixed budget ratiosρbudget\\rho\_\{\\text\{budget\}\}, each targeting a different memory regime\.

Inspired by RL\-VO\[[Messikommer et al\.\(2024\)Messikommer, Cioffi, Gehrig, and Scaramuzza](https://arxiv.org/html/2610.00188#bib.bibx14)\], which uses cross\-attention for variable\-length RL observations, we adopt a PerceiverIO architecture\[[Jaegle et al\.\(2022\)Jaegle, Borgeaud, Alayrac, Doersch, Ionescu, Ding, Koppula, Zoran, Brock, Shelhamer, Hénaff, Botvinick, Zisserman, Vinyals, and Carreira](https://arxiv.org/html/2610.00188#bib.bibx7)\]for the variable number of sampled blocksNN\. Each block’s\(μb,σb\)\(\\mu\_\{b\},\\sigma\_\{b\}\)pair is a key–value input; four learned latent queries compress all block features into a fixed 256\-dim representation processed by an MLP to produce the action distribution over2​N2Nthreshold indices\. The value network additionally receives the global critic features\(h¯,σ⁡\(h\),ρused,ρbudget\)\(\\bar\{h\},\\,\\sigma\(h\),\\,\\rho\_\{\\text\{used\}\},\\,\\rho\_\{\\text\{budget\}\}\), focusing the critic on budget context\.

We train our reinforcement\-learning agent on 50 scenes randomly sampled from the ScanNet training split\[[Dai et al\.\(2017\)Dai, Chang, Savva, Halber, Funkhouser, and Nießner](https://arxiv.org/html/2610.00188#bib.bibx4)\], disjoint from all evaluation scenes\. Each scene provides a sequence of RGB\-D frames with corresponding camera poses and precomputed11cm Voxblox reference reconstructions used for reward computation\. Training is performed for a total of 10 million environment steps\. Each policy is trained separately for approximately 24 hours on a single consumer\-grade RTX 3080 \(10 GB\), using the precomputed 1 cm reference reconstructions for reward supervision\. At inference, these references are no longer required: the user simply selects a desired storage\-budget regime, and the corresponding policy predicts subdivision thresholds directly from the observed block uncertainty statistics\{μb,σb\}\\\{\\mu\_\{b\},\\sigma\_\{b\}\\\}\. Additional training details, including hardware and total training cost, are provided in[AppendixC](https://arxiv.org/html/2610.00188#A3)\.

## 4Experiments

We evaluate geometric reconstruction, semantic accuracy, texture fidelity, and efficiency on both synthetic and real\-world benchmarks, comparing fixed\-resolution baselines, MAP\-ADAPT, and our uncertainty\- and RL\-driven variants\.

Table 1:Accuracy and efficiency on ScanNet\.We report geometric reconstruction quality \(completion error and completion ratio within 5 cm\), semantic accuracy, and texture fidelity \(MS\-SSIM, LPIPS\), all obtained from mesh renderings in held\-out test views\. Efficiency metrics include total voxel count and peak memory usage in gigabytes\. Our complexity\- and uncertainty\-driven method \(Un\-Map\-Adapt \(Ours\)\\xspace\) outperforms MAP\-ADAPT inallaccuracy metrics while avoiding reliance on manually specified semantic class lists\. The RL\-based variants enable explicit control of memory through voxel\-budget constraints, offering competitive accuracy with reduced memory\.Table 2:Accuracy and efficiency on HSSD\.Evaluation on synthetic HSSD scenes, where the RL models are*not*trained on HSSD but transferred directly from ScanNet\. We measure geometric \(completion error and ratio<<5 cm\), semantic, and texture \(MS\-SSIM, LPIPS\) accuracy, together with efficiency \(voxel count, memory\)\. Our complexity\- and uncertainty\-driven method surpasses MAP\-ADAPT without requiring any dataset\-specific semantic class list\. The RL\-based variants maintain strong performance under strict memory budgets and generalize robustly to unseen synthetic environments\.Datasets\.Following MAP\-ADAPT\[[Zheng et al\.\(2024\)Zheng, Barath, Pollefeys, and Armeni](https://arxiv.org/html/2610.00188#bib.bibx31)\], we evaluate on 81 scenes \(all disjoint from the 50 ScanNet scenes used for RL training\): \(i\)HSSD\[[Khanna et al\.\(2024\)Khanna, Mao, Jiang, Haresh, Shacklett, Batra, Clegg, Undersander, Chang, and Savva](https://arxiv.org/html/2610.00188#bib.bibx10)\]provides photorealistic synthetic indoor scenes with noise\-free ground\-truth geometry, semantics, and textures\. \(ii\)ScanNet\[[Dai et al\.\(2017\)Dai, Chang, Savva, Halber, Funkhouser, and Nießner](https://arxiv.org/html/2610.00188#bib.bibx4)\]contains real\-world RGB\-D scans with measurement noise, missing depth, and partial semantic annotations, testing robustness under realistic conditions\.

Metrics\.All metrics are computed on meshes extracted from the adaptive TSDF grids\. \(i\)Geometry:mean completion error and completion ratio within 5 cm\. \(ii\)Semantics:vertex labels from the fused voxel distribution are rendered into held\-out test views via raycasting and compared to ground\-truth semantic projections, penalizing both label errors and geometric misalignment\. \(iii\)Texture:MS\-SSIM and LPIPS\[[Zhang et al\.\(2018\)Zhang, Isola, Efros, Shechtman, and Wang](https://arxiv.org/html/2610.00188#bib.bibx30)\]on raycasted color renderings\. \(iv\)Efficiency:total voxel count, peak memory, and per\-frame integration time\. All adaptive methods share the same TSDF integration and meshing pipelines\[[Zheng et al\.\(2024\)Zheng, Barath, Pollefeys, and Armeni](https://arxiv.org/html/2610.00188#bib.bibx31)\]\.

Baselines and Ablations\.

- •Voxblox \(8 cm, 4 cm, 1 cm\)\[[Oleynikova et al\.\(2017\)Oleynikova, Taylor, Fehr, Siegwart, and Nieto](https://arxiv.org/html/2610.00188#bib.bibx19)\]:Fixed\-resolution baselines at each of the three levels that the adaptive methods select among during subdivision\. The 1 cm grid provides a quality upper bound at prohibitive memory cost\.
- •Map\-Adapt\[[Zheng et al\.\(2024\)Zheng, Barath, Pollefeys, and Armeni](https://arxiv.org/html/2610.00188#bib.bibx31)\]:Uses geometric CoC\[[Pauly et al\.\(2002\)Pauly, Gross, and Kobbelt](https://arxiv.org/html/2610.00188#bib.bibx20)\]and a*manually selected semantic class list*, limiting generalization\. We use the official HSSD configuration; for ScanNet, we adopt the same list with unknown classes assigned to coarse resolution\.
- •Map\-Adapt\-Split\-Avg\[[Zheng et al\.\(2024\)Zheng, Barath, Pollefeys, and Armeni](https://arxiv.org/html/2610.00188#bib.bibx31)\]:MAP\-ADAPT averaged over 5 runs with randomly sampled semantic class lists, where each list specifies which class labels trigger fine\-resolution subdivision\. This evaluates sensitivity to class list selection when no dataset\-specific configuration is available\.
- •Un\-Map\-Adapt \(Ours\)\\xspace:Our method using semantic entropy, geometric complexity, and texture complexity as subdivision signals, without user\-defined class lists\.
- •UnRL \(Ours\)\\xspace:Budget\-aware RL refinement trained at 25% and 75% target voxel\-budget regimes, providing budget\-aware refinement at inference\.

We omit Multi\-TSDF\[[Schmid et al\.\(2022\)Schmid, Delmerico, Schönberger, Nieto, Pollefeys, Siegwart, and Cadena](https://arxiv.org/html/2610.00188#bib.bibx23)\], which has been shown to underperform MAP\-ADAPT\. We further evaluate robustness to the semantic segmentation backbone by replacing RefineNet with SegFormer;[Table5](https://arxiv.org/html/2610.00188#A1.T5)shows consistent accuracy–memory behavior with the alternative backbone\.

Results on ScanNet\.[Table1](https://arxiv.org/html/2610.00188#S4.T1)shows that our uncertainty\-driven refinement outperforms MAP\-ADAPT on almost all metrics\.Un\-Map\-Adapt \(Ours\)\\xspaceachieves lower completion error and higher*Compl\.<<5cm*than MAP\-ADAPT, and also improves semantic accuracy and photometric quality\. These improvements primarily arise from*automatically*identifying informative or uncertain regions for refinement, rather than relying on a preselected class list\. Importantly, this enables generalization to scenes containing object categories not explicitly known during system design\. Because our method additionally refines texture\-rich regions, which are ignored entirely by MAP\-ADAPT, the unconstrained variant naturally allocates more voxels, as expected\. This additional cost directly motivates the budget\-aware RL variant, which recovers the efficiency advantage while preserving the broader refinement criteria\.

The RL\-based variant learns predictable trade\-offs between memory and accuracy\. At 75% of the voxel budget, the reconstruction quality is close to our deterministic model, while memory is reduced by more than half\. These results confirm that the agent learns to allocate resources where they most reduce reconstruction loss\. At extreme budgets \(25%\), the policy prioritizes high uncertainty and complexity regions, maintaining key structural and semantic details, and gracefully reducing resolution elsewhere\.

Table 3:Ablation of refinement criteria on ScanNet\[[Dai et al\.\(2017\)Dai, Chang, Savva, Halber, Funkhouser, and Nießner](https://arxiv.org/html/2610.00188#bib.bibx4)\]\. Adding geometric complexity improves reconstruction completeness and semantic consistency, while incorporating texture complexity further enhances all metrics, demonstrating that all three refinement criteria contribute complementary information\.Results on HSSD\.[Table2](https://arxiv.org/html/2610.00188#S4.T2)reports results on the synthetic HSSD scenes\. Although the RL agent and all hyperparameters were trained on ScanNet, our uncertainty\-driven model again surpasses MAP\-ADAPT across geometric and texture metrics\. This demonstrates a key strength of our formulation: since voxel refinement is triggered by modality\-specific uncertainty and complexity signals rather than predefined semantic categories, the method naturally generalizes to scenes containing previously unseen objects, materials, and textures\.

The RL\-based models show the same trend observed on ScanNet\. With a moderate budget \(75%\), reconstruction accuracy remains nearly indistinguishable from the unconstrained version, while memory usage is substantially reduced\. Even under tight budgets \(25%\), the policy maintains stable reconstructions, consistently preserving fine geometric structures and semantically meaningful regions despite the reduced allocation of high\-resolution voxels\.

Runtime\.The average per\-frame integration times are shown in[Table4](https://arxiv.org/html/2610.00188#S4.T4)\. Voxblox with large voxels \(Voxblox \(8 cm\)\) achieves the highest throughput, but at the cost of substantially lower reconstruction accuracy\. The uniform fine\-resolution baseline \(Voxblox \(1 cm\)\) remains computationally efficient, but requires considerably higher memory\. Across all adaptive methods, runtime remains within a similar range of 19\-26 FPS\. This demonstrates that both heuristic and learned adaptive subdivision preserve real\-time performance\. The RL\-based approach achieves a favorable trade\-off: it adapts refinement to the available voxel budget, avoiding unnecessary refinement and maintaining consistent frame rates even under tight memory settings\.

Memory\.Uniform fine\-grid reconstruction \(Voxblox \(1 cm\)\) requires between 20–25 GB on larger HSSD scenes, making it impractical for memory\-constrained real\-time systems\. As shown in[Tables1](https://arxiv.org/html/2610.00188#S4.T1)and[2](https://arxiv.org/html/2610.00188#S4.T2), our deterministic uncertainty\-driven method \(Un\-Map\-Adapt\) prioritizes reconstruction quality by refining regions according to semantic uncertainty, geometric complexity, and texture complexity, but does not explicitly constrain memory usage\. The RL\-based variants address this limitation by targeting scene\-specific voxel budgets and providing explicit control over the memory–accuracy trade\-off\. Under the 25% setting,UnRLreduces memory by9\.2×9\.2\\timeson ScanNet and12\.6×12\.6\\timeson HSSD relative to the fine\-resolution Voxblox baseline, while trading some reconstruction accuracy for substantially lower memory usage\. This budget\-aware behavior makes the RL policies particularly well\-suited for systems operating under strict memory constraints, such as mobile or embedded platforms\.

Figure 3:Qualitative comparisonon HSSD \(top three rows\) and ScanNet \(bottom\)\. Columns show increasing reconstruction quality and detail from coarseVoxblox\(8 cm\) to the fine\-grained 1 cm baseline\. Our deterministic uncertainty\-driven method \(Un\-Map\-Adapt\) and our budget\-aware RL model \(UnRL\) recover sharper geometry, cleaner object boundaries, and more stable textures thanMap\-Adapt, without requiring a hand\-crafted semantic class list\. The RL variants \(25%, 75%\) adaptively preserve high uncertainty and complexity regions while respecting memory budgets, demonstrating controlled degradation under tight constraints\.Table 4:Per\-frame integration speed \(FPS\) on ScanNet\. All methods run in real time\.Ablation Study\.We further ablate the contribution of the proposed refinement criteria used for adaptive subdivision\. Specifically, we evaluate three variants of our approach: \(1\) using onlysemantic entropy\(Sem\), \(2\) combiningsemantic entropyandgeometric complexity\(Sem\+Geo\), and \(3\) the full model that additionally incorporatestexture complexity\(Sem\+Geo\+Tex\)\.

[Table3](https://arxiv.org/html/2610.00188#S4.T3)reports the results on ScanNet, where introducing geometric complexity improves completeness and semantic accuracy by focusing refinement on curved or structurally intricate geometry\. Adding texture complexity further enhances reconstruction fidelity and overall stability, as the model allocates fine voxels to visually rich regions\. These results confirm that integrating complementary refinement criteria leads to more balanced optimization between geometry, appearance, and semantics\.

Qualitative Results\.[Figure3](https://arxiv.org/html/2610.00188#S4.F3)presents qualitative reconstructions on both ScanNet and HSSD, illustrating the effect of adaptive subdivision on geometric detail and texture fidelity\. Coarse Voxblox \(8 cm\) produces heavily oversmoothed surfaces, while the fine 1 cm grid recovers sharp structure but at prohibitive memory cost \(as seen in[Tables1](https://arxiv.org/html/2610.00188#S4.T1)and[2](https://arxiv.org/html/2610.00188#S4.T2)\)\. Our deterministic uncertainty\-driven variant \(Un\-Map\-Adapt\) yields visibly sharper object contours, cleaner planar regions, and more stable texture reconstructions, particularly around edges where semantic entropy, geometric complexity, and texture complexity jointly signal the need for fine resolution\. The RL\-based policies further demonstrate controlled adaptation: with moderate budgets \(*e\.g*, 75%\), reconstructions closely match the unconstrained model, whereas under tight constraints \(*e\.g*, 25%\), the system preserves the most informative regions such as object boundaries, high\-curvature geometry, and textured surfaces, while simplifying homogeneous areas\. Across all settings, our method achieves a good balance between detail and compactness, and generalizes to HSSD despite the RL agent being trained only on ScanNet\.

## 5Conclusion

This work tackles the absence of explicit, user\-controllable accuracy–memory trade\-offs in current volumetric mapping pipelines\. MAP\-ADAPT offers limited control through manual class lists and class\-dependent subdivision levels, but this requires expert tuning, fails to generalize, and becomes impractical in scenes with many small or unseen objects\. Our method provides a principled and fully automatic alternative: uncertainty\- and complexity\-driven refinement removes dependence on semantic taxonomies, while a reinforcement\-learning policy enables direct specification of a desired memory budget, with the system allocating resolution accordingly\. Experiments on real and synthetic datasets show that our approach matches or surpasses MAP\-ADAPT in geometric, semantic, and texture accuracy, while delivering predictable and stable performance under varying storage constraints\. By making the target memory budget the primary user\-facing parameter, our framework provides a flexible and scalable approach to adaptive 3D mapping\. Our evaluation is currently limited to the indoor RGB\-D setting of ScanNet and HSSD\. Nevertheless, the semantic, geometric, and texture cues exploited by our approach are not intrinsically tied to indoor scene categories\. Extending and adapting the method to outdoor and larger\-scale mapping scenarios is an important direction for future work\.

## Appendix AExperiments with SegFormer

This section provides additional experiments and visualizations\. We evaluate our method using SegFormer\[[Xie et al\.\(2021\)Xie, Wang, Yu, Anandkumar, Alvarez, and Luo](https://arxiv.org/html/2610.00188#bib.bibx29)\], a stronger but heavier segmentation model than RefineNet\[[Lin et al\.\(2017\)Lin, Milan, Shen, and Reid](https://arxiv.org/html/2610.00188#bib.bibx11)\]used in the main paper\. We present extended qualitative comparisons, and analyze the behavior and generalization of the reinforcement learning \(RL\) agents\. Our goal is to demonstrate that \(i\) the proposed entropy\-driven framework is robust to changes in the semantic backbone, \(ii\) the accuracy\-memory trade\-off remains consistent under different segmentation signals, and \(iii\) methods relying on fixed semantic class heuristics, such as MAP\-ADAPT, are inherently brittle across datasets and segmentation models\.

### A\.1Qualitative Comparisons

[Figure4](https://arxiv.org/html/2610.00188#A1.F4)and[Figure5](https://arxiv.org/html/2610.00188#A1.F5)present visualizations on HSSD and ScanNet for RefineNet and SegFormer, respectively\. We visualize textured reconstructions and semantic renderings from multiple viewpoints and across all ablations\.

The coarse Voxblox baseline loses fine structure and produces over\-smoothed surfaces\. MAP\-ADAPT improves detail, but remains sensitive to its hand\-crafted class list, which often fails in the presence of new or ambiguous object categories\. Our entropy\-based variant yields crisp object boundaries and improved structural detail, while the RL models preserve high\-entropy regions even under strong memory constraints\. The 25% and 50% policies demonstrate controlled simplification, whereas the 75% policy closely matches the full\-resolution entropy\-driven method\.

Using SegFormer sharpens semantic boundaries and reveals a clearer separation between the methods: improvements in semantic quality are most visible forUn\-Map\-Adapt \(Ours\)\\xspaceand its RL variants, confirming the relevance of entropy\-driven subdivision\.

### A\.2ScanNet Benchmark with SegFormer

[Table5](https://arxiv.org/html/2610.00188#A1.T5)reports quantitative results on ScanNet with SegFormer\. Reconstruction and texture metrics remain consistent with the main paper, while semantic accuracy increases across all methods due to the improved segmentation model\. We additionally evaluate an RL agent with a 50% budget\. As expected, accuracy increases smoothly from 25% to 75%, showing that the learned policy produces a stable and monotonic accuracy – memory trade\-off\.

To study MAP\-ADAPT’s sensitivity to class heuristics, we introduce a random class\-to\-resolution assignment\. Performance degrades sharply across all metrics, underscoring the method’s reliance on user\-engineered class lists and highlighting its lack of adaptability\. As scenes, datasets, or segmentation backbones change, these heuristics may not transfer reliably\. In contrast, our uncertainty\- and complexity\-driven method is class agnostic and relies on observed semantic, geometric, and texture cues rather than predefined semantic class assignments, supporting transfer across the tested segmentation backbones\.

### A\.3Generalization of the RL Agents

A key question is whether the RL policies learned on RefineNet generalize to SegFormer\. The policy does not consume semantic class identities\. Instead, its semantic component is based on normalized prediction entropy, while geometric and texture complexity provide complementary class\-independent cues\. Specifically, the per\-voxel proxy used to compute the block\-level statistics observed by the policy is

hi=w~semH~isem\+w~geog~i\+w~text~i,H~isem=−1log⁡C∑c=1CP^isem\(c\)logP^isem\(c\)\.h\_\{i\}=\\tilde\{w\}\_\{\\mathrm\{sem\}\}\\tilde\{H\}\_\{i\}^\{\\mathrm\{sem\}\}\+\\tilde\{w\}\_\{\\mathrm\{geo\}\}\\tilde\{g\}\_\{i\}\+\\tilde\{w\}\_\{\\mathrm\{tex\}\}\\tilde\{t\}\_\{i\},\\qquad\\tilde\{H\}\_\{i\}^\{\\mathrm\{sem\}\}=\-\\frac\{1\}\{\\log C\}\\sum\_\{c=1\}^\{C\}\\hat\{P\}\_\{i\}^\{\\mathrm\{sem\}\}\(c\)\\log\\hat\{P\}\_\{i\}^\{\\mathrm\{sem\}\}\(c\)\.\(18\)Here,H~isem\\tilde\{H\}\_\{i\}^\{\\mathrm\{sem\}\}depends only on the shape of the predicted class\-probability distribution rather than the identity or ordering of the classes, whileg~i\\tilde\{g\}\_\{i\}andt~i\\tilde\{t\}\_\{i\}are independent of semantic class identities\. The reward similarly combines semantic, geometric, and texture reconstruction losses\. Consequently, the policy representation does not rely on a particular semantic class configuration\.

Results in[Table5](https://arxiv.org/html/2610.00188#A1.T5)confirm this: the RL policies trained entirely on RefineNet preserve their ordering, monotonicity, and accuracy–memory trade\-off when deployed with SegFormer, despite the two backbones differing substantially in architecture, capacity, and prediction characteristics\. This demonstrates that the agent learns a structural refinement strategy based on class\-agnostic uncertainty and complexity cues, rather than overfitting to the statistics of a particular pretrained network\.

### A\.4Qualitative Comparison: Semantic Maps

[Figure4](https://arxiv.org/html/2610.00188#A1.F4)provides semantic renderings for the same scenes and ablations as in Fig\. 3 of the main paper, which presents qualitative texture reconstruction comparisons\. HSSD contains fewer object categories dominated by background classes, which reduces the impact of semantic refinement\. Conversely, ScanNet contains a richer class distribution but is limited by RefineNet accuracy\. SegFormer alleviates this limitation, producing sharper maps and making the benefits of our adaptive strategy more apparent\.[Figure6](https://arxiv.org/html/2610.00188#A1.F6)presents the corresponding semantic renderings using SegFormer, where the improved backbone yields sharper class boundaries and a more visible separation between the adaptive methods\.

Figure 4:Semantic segmentationsrendered from the reconstructed scene meshes on HSSD \(top three rows\) and ScanNet \(bottom\) using RefineNet\[[Lin et al\.\(2017\)Lin, Milan, Shen, and Reid](https://arxiv.org/html/2610.00188#bib.bibx11)\]to provide per\-frame semantic segmentations\. Columns show increasing reconstruction quality and detail from coarseVoxblox\(8 cm\) to the fine\-grained 1 cm baseline\. Our deterministic entropy\-based method \(Un\-Map\-Adapt\) and our budget\-aware RL model \(UnRL\) recover sharper geometry, cleaner object boundaries, and more stable textures thanMap\-Adapt, without requiring a hand\-crafted semantic class list\. The RL variants \(25%, 75%\) adaptively preserve high\-uncertainty regions while respecting memory budgets, demonstrating controlled degradation under tight constraints\.Figure 5:Rendered imageson the ScanNet dataset from reconstructed meshes using SegFormer\[[Xie et al\.\(2021\)Xie, Wang, Yu, Anandkumar, Alvarez, and Luo](https://arxiv.org/html/2610.00188#bib.bibx29)\]for semantic labeling\. Columns show increasing reconstruction quality from coarseVoxblox\(8 cm\), through adaptive methods and a mid\-resolutionVoxblox\(4 cm\), to the fine\-grained 1 cm baseline\. Our deterministic entropy\-based method \(Un\-Map\-Adapt\) and the budget\-aware RL model \(UnRL\) recover sharper geometry, cleaner object boundaries, and more stable textures thanMap\-Adapt, without relying on hand\-crafted semantic class lists\. The RL variants \(25%, 50%, 75%\) demonstrate controlled degradation by prioritizing high\-uncertainty regions under increasing memory constraints\.Figure 6:Semantic segmentationson the ScanNet dataset rendered from reconstructed meshes using SegFormer\[[Xie et al\.\(2021\)Xie, Wang, Yu, Anandkumar, Alvarez, and Luo](https://arxiv.org/html/2610.00188#bib.bibx29)\]\. Columns show increasing reconstruction quality from coarseVoxblox\(8 cm\), through adaptive methods and a mid\-resolutionVoxblox\(4 cm\), to the fine\-grained 1 cm baseline\. Our deterministic entropy\-based method \(Un\-Map\-Adapt\) and the budget\-aware RL model \(UnRL\) recover cleaner object boundaries and more spatially consistent semantic regions thanMap\-Adapt, without relying on hand\-crafted semantic class lists\. The RL variants \(25%, 50%, 75%\) demonstrate controlled degradation by prioritizing high\-uncertainty regions under increasing memory constraints\.Table 5:Accuracy and efficiency on ScanNet with SegFormer\[[Xie et al\.\(2021\)Xie, Wang, Yu, Anandkumar, Alvarez, and Luo](https://arxiv.org/html/2610.00188#bib.bibx29)\]providing the per\-frame semantic segmentations\.We report geometric reconstruction quality \(completion error and completion ratio within 5 cm\), semantic accuracy, and texture fidelity \(MS\-SSIM, LPIPS\), all obtained from mesh renderings in held\-out test views\. Efficiency metrics include total voxel count and peak memory usage in gigabytes\. We also show MAP\-ADAPT with random semantic\-to\-resolution assignments, demonstrating its sensitivity to user\-defined class lists\. Our full entropy\-driven method \(Un\-Map\-Adapt \(Ours\)\\xspace\) outperforms MAP\-ADAPT inallaccuracy metrics while avoiding reliance on manually specified semantic class lists\. The RL\-based variants enable explicit control of memory through voxel\-budget constraints, offering competitive accuracy with reduced memory\.

## Appendix BSensitivity Analysis

We conduct a sensitivity analysis of the key hyperparameters for both the RL\-based variant \(UnRL\) and our deterministic method \(UN\-MAP\-ADAPT\)\. All ablations are evaluated on the ScanNet validation set using SegFormer as the segmentation backbone\. The UnRL ablations \([SectionB\.1](https://arxiv.org/html/2610.00188#A2.SS1)\) are averaged over 10 scenes, while the UN\-MAP\-ADAPT threshold sweep \([SectionB\.2](https://arxiv.org/html/2610.00188#A2.SS2)\) is averaged over 5 scenes\. We first ablate block size, number of blocks, and reward weight configuration for UnRL \([SectionB\.1](https://arxiv.org/html/2610.00188#A2.SS1)\), then sweep the subdivision threshold settings for UN\-MAP\-ADAPT \([SectionB\.2](https://arxiv.org/html/2610.00188#A2.SS2)\)\. Our goal is to demonstrate that both methods are robust to reasonable hyperparameter choices, and that the RL agent’s advantage lies precisely in eliminating the need for manual threshold tuning\.

### B\.1UNRL Parameter Sensitivity

TheUnRL \(Ours\)\\xspaceagent has three key design parameters: the spatial block sizess, the number of sampled blocksNNper frame, and the reward weight configuration\(wgeo,wsem,wtex\)\(w\_\{\\text\{geo\}\},w\_\{\\text\{sem\}\},w\_\{\\text\{tex\}\}\)that aggregates the three loss modalities into a single scalar reward\. We ablate each parameter independently while keeping all others at their default values\.

#### Block size\.

The block sizesscontrols the spatial granularity of the agent’s decisions: all voxels within a block share the same thresholds on the complexity proxy\.[Table6](https://arxiv.org/html/2610.00188#A2.T6)reports results fors∈\{0\.1,0\.25,0\.5,1\.0\}s\\in\\\{0\.1,0\.25,0\.5,1\.0\\\}m\. Very small blocks \(s=0\.1s\{=\}0\.1m\) underperform because each block contains too few voxels for reliable uncertainty and complexity statistics, leading to noisy observations and poor decisions\. Performance improves substantially from 0\.1 m to 0\.5 m and plateaus at 1\.0 m\. Althoughs=1\.0s\{=\}1\.0m is marginally better on completion metrics, we selects=0\.5s\{=\}0\.5m as the default because it provides finer spatial control: the agent can assign different thresholds to neighboring regions that a 1\.0 m block would merge\. The accuracy gap between the two \(<<0\.01 cm in completion error\) confirms that the method is not sensitive to this choice within a reasonable range\.

Table 6:Ablation on block sizess\. All other hyperparameters are fixed to default\.†\\daggerdenotes the default configuration\. Averaged over 10 ScanNet validation scenes\.
#### Number of blocks\.

The number of sampled blocksNNdetermines how many spatial regions the agent observes and acts on per frame\. Since the policy network uses a Perceiver\-based attention mechanism, computational cost scales quadratically withNN, which motivates keeping the observation compact\.[Table7](https://arxiv.org/html/2610.00188#A2.T7)shows that increasingNNfrom 10 to 50 yields only modest improvement, while increasing both training and inference cost\. We useN=25N\{=\}25as the default, which balances observation coverage with computational efficiency\. Notably, evenN=10N\{=\}10achieves competitive results, suggesting that the proxy\-weighted importance sampling effectively directs attention to the most informative blocks regardless of the total sample count\.

Table 7:Ablation on number of blocksNN\. All other hyperparameters are fixed to default\.†\\daggerdenotes the default configuration\. Averaged over 10 ScanNet validation scenes\.
#### Reward weights\.

The reward weights\(wgeo,wsem,wtex\)\(w\_\{\\text\{geo\}\},w\_\{\\text\{sem\}\},w\_\{\\text\{tex\}\}\)control how the agent aggregates the three loss modalities into a single scalar reward\. The combined loss used as the RL training signal is:

ℒ=wgeo⋅ℒgeo\+wsem⋅ℒsem\+wtex⋅ℒtex,\\mathcal\{L\}=w\_\{\\text\{geo\}\}\\cdot\\mathcal\{L\}\_\{\\text\{geo\}\}\+w\_\{\\text\{sem\}\}\\cdot\\mathcal\{L\}\_\{\\text\{sem\}\}\+w\_\{\\text\{tex\}\}\\cdot\\mathcal\{L\}\_\{\\text\{tex\}\},\(19\)whereℒgeo\\mathcal\{L\}\_\{\\text\{geo\}\},ℒsem\\mathcal\{L\}\_\{\\text\{sem\}\}, andℒtex\\mathcal\{L\}\_\{\\text\{tex\}\}denote the geometric, semantic, and texture reconstruction losses, respectively\.

[Table8](https://arxiv.org/html/2610.00188#A2.T8)shows that single\-modality rewards \(top three rows\) perform noticeably worse than any balanced combination, confirming that all three signals carry complementary information about where subdivision is beneficial\. Among the balanced configurations \(bottom four rows\), performance differences are minimal: completion error varies by only 0\.012 cm and semantic accuracy by 0\.15 percentage points\. This insensitivity is a desirable property; the weights serve primarily as an aggregation mechanism rather than a sensitive hyperparameter requiring careful tuning\. We adopt equal weights\(0\.33,0\.33,0\.33\)\(0\.33,0\.33,0\.33\)as the default for simplicity, noting that slightly emphasizing semantic loss\(0\.25,0\.50,0\.25\)\(0\.25,0\.50,0\.25\)yields virtually identical results\.

Table 8:Ablation on reward weight configuration\(wgeo,wsem,wtex\)\(w\_\{\\text\{geo\}\},\\,w\_\{\\text\{sem\}\},\\,w\_\{\\text\{tex\}\}\)\.†\\daggerdenotes the default configuration\. Averaged over 10 ScanNet validation scenes\.wgeow\_\{\\text\{geo\}\}wsemw\_\{\\text\{sem\}\}wtexw\_\{\\text\{tex\}\}Compl\. Err↓Compl\.<<5cm↑Sem\. Acc↑MS\-SSIM↑LPIPS↓\(cm\)\(%\)\(%\)1\.000\.000\.001\.43098\.2471\.320\.6210\.4110\.001\.000\.001\.51397\.8371\.310\.6200\.4130\.000\.001\.001\.51597\.3371\.040\.6180\.4210\.500\.250\.251\.37898\.8071\.680\.6250\.4030\.250\.250\.501\.37698\.7471\.570\.6250\.4040\.330\.330\.33†1\.36698\.9671\.710\.6260\.4030\.250\.500\.251\.36698\.9671\.720\.6260\.403

### B\.2UN\-MAP\-ADAPT Parameter Sensitivity

UN\-MAP\-ADAPT uses five hand\-tuned thresholds that control when a voxel is subdivided based on its semantic entropy \(τsem\\tau^\{\\mathrm\{sem\}\}\), texture complexity \(τlotex\\tau^\{\\mathrm\{tex\}\}\_\{\\mathrm\{lo\}\},τhitex\\tau^\{\\mathrm\{tex\}\}\_\{\\mathrm\{hi\}\}\), and geometric complexity \(τlogeo\\tau^\{\\mathrm\{geo\}\}\_\{\\mathrm\{lo\}\},τhigeo\\tau^\{\\mathrm\{geo\}\}\_\{\\mathrm\{hi\}\}\)\. Unlike UnRL, which learns a subdivision strategy end\-to\-end, UN\-MAP\-ADAPT requires these thresholds to be specified manually and jointly determines the accuracy–memory operating point\.

[Table9](https://arxiv.org/html/2610.00188#A2.T9)sweeps all five thresholds from loose \(top rows, fewer subdivisions, low memory\) to strict \(bottom rows, more subdivisions, high memory\)\. Loose settings dramatically reduce memory but at significant accuracy cost: the loosest configuration degrades completion error to 2\.22 cm and semantic accuracy to 71\.17%\. As thresholds tighten, accuracy improves rapidly up to the default configuration \(highlighted\), after which returns diminish: belowτsem=0\.10\\tau^\{\\mathrm\{sem\}\}\{=\}0\.10, further tightening yields less than 0\.03 cm improvement in completion error while continuing to increase memory from 5\.32 GB toward 5\.69 GB\.

This sweep illustrates the core motivation for our RL\-based approach\. The optimal threshold configuration depends on the target memory budget, scene complexity, and downstream task requirements, and must be re\-tuned whenever any of these factors change\.UnRL \(Ours\)\\xspacereplaces this manual search with a learned policy that automatically discovers the appropriate subdivision strategy for a given budget constraint, removing the need for per\-deployment threshold engineering\.

Table 9:Threshold sensitivity analysis across configurations, averaged over 5 ScanNet validation scenes\. The default configuration is highlighted in gray\. Best values per metric are shown inbold\.

## Appendix CTraining Hardware and Runtime

All experiments, including both 3D semantic scene reconstruction and RL policy training, were conducted on a single workstation equipped with an Intel Core i7\-14700K processor \(20 cores, 28 threads\) and an NVIDIA GeForce RTX 3080 GPU \(10 GB VRAM, CUDA 12\.2\)\.

For RL training, we used PPO with 4 parallel environments viaSubprocVecEnv, training for 10M timesteps per experiment\. Evaluation was performed every 50k steps on a held\-out validation set of 5 episodes, with model checkpoints saved every 250k steps\. Each training run took approximately 24 hours\.

The 3D reconstruction pipeline, which generates the RL dataset by processing ScanNet scenes into per\-frame voxel observations and reconstruction losses, was executed on the same machine\.

## References

- \[Asgharivaskasi and Atanasov\(2023\)\]Arash Asgharivaskasi and Nikolay Atanasov\.Semantic OcTree mapping and Shannon mutual information computation for robot exploration\.*IEEE TRO*, 2023\.
- \[Charrow et al\.\(2015\)Charrow, Liu, Kumar, and Michael\]Benjamin Charrow, Sikang Liu, Vijay Kumar, and Nathan Michael\.Information\-theoretic mapping using Cauchy\-Schwarz quadratic mutual information\.In*ICRA*, 2015\.
- \[Chen et al\.\(2019\)Chen, Bai, Shan, and Englot\]Fanfei Chen, Shi Bai, Tixiao Shan, and Brendan Englot\.Self\-learning exploration and mapping for mobile robots via deep reinforcement learning\.In*AIAA SciTech Forum*, 2019\.
- \[Dai et al\.\(2017\)Dai, Chang, Savva, Halber, Funkhouser, and Nießner\]Angela Dai, Angel X\. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner\.ScanNet: Richly\-annotated 3D reconstructions of indoor scenes\.In*CVPR*, 2017\.
- \[Grinvald et al\.\(2019\)Grinvald, Furrer, Novkovic, Chung, Cadena, Siegwart, and Nieto\]Margarita Grinvald, Fadri Furrer, Tonci Novkovic, Jen Jen Chung, Cesar Cadena, Roland Siegwart, and Juan Nieto\.Volumetric instance\-aware semantic mapping and 3D object discovery\.*IEEE RA\-L*, 2019\.
- \[Hornung et al\.\(2013\)Hornung, Wurm, Bennewitz, Stachniss, and Burgard\]Armin Hornung, Kai M\. Wurm, Maren Bennewitz, Cyrill Stachniss, and Wolfram Burgard\.OctoMap: an efficient probabilistic 3D mapping framework based on octrees\.*Auton\. Robots*, 2013\.
- \[Jaegle et al\.\(2022\)Jaegle, Borgeaud, Alayrac, Doersch, Ionescu, Ding, Koppula, Zoran, Brock, Shelhamer, Hénaff, Botvinick, Zisserman, Vinyals, and Carreira\]Andrew Jaegle, Sebastian Borgeaud, Jean\-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier J\. Hénaff, Matthew Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira\.Perceiver IO: A general architecture for structured inputs & outputs\.In*ICLR*, 2022\.
- \[Kähler et al\.\(2016\)Kähler, Prisacariu, Valentin, and Murray\]Olaf Kähler, Victor Prisacariu, Julien Valentin, and David Murray\.Hierarchical voxel block hashing for efficient integration of depth images\.*IEEE RA\-L*, 2016\.
- \[Kerbl et al\.\(2023\)Kerbl, Kopanas, Leimkühler, and Drettakis\]Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis\.3D Gaussian splatting for real\-time radiance field rendering\.*ACM TOG*, 2023\.
- \[Khanna et al\.\(2024\)Khanna, Mao, Jiang, Haresh, Shacklett, Batra, Clegg, Undersander, Chang, and Savva\]Mukul Khanna, Yongsen Mao, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X\. Chang, and Manolis Savva\.Habitat Synthetic Scenes Dataset \(HSSD\-200\): An analysis of 3D scene scale and realism tradeoffs for ObjectGoal navigation\.In*CVPR*, 2024\.
- \[Lin et al\.\(2017\)Lin, Milan, Shen, and Reid\]Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid\.RefineNet: Multi\-path refinement networks for high\-resolution semantic segmentation\.In*CVPR*, 2017\.
- \[McCormac et al\.\(2017\)McCormac, Handa, Davison, and Leutenegger\]John McCormac, Ankur Handa, Andrew Davison, and Stefan Leutenegger\.SemanticFusion: Dense 3D semantic mapping with convolutional neural networks\.In*ICRA*, 2017\.
- \[McCormac et al\.\(2018\)McCormac, Clark, Bloesch, Davison, and Leutenegger\]John McCormac, Ronald Clark, Michael Bloesch, Andrew Davison, and Stefan Leutenegger\.Fusion\+\+: Volumetric object\-level SLAM\.In*3DV*, 2018\.
- \[Messikommer et al\.\(2024\)Messikommer, Cioffi, Gehrig, and Scaramuzza\]Nico Messikommer, Giovanni Cioffi, Mathias Gehrig, and Davide Scaramuzza\.Reinforcement learning meets visual odometry\.In*ECCV*, 2024\.
- \[Mildenhall et al\.\(2020\)Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng\]Ben Mildenhall, Pratul P\. Srinivasan, Matthew Tancik, Jonathan T\. Barron, Ravi Ramamoorthi, and Ren Ng\.NeRF: Representing scenes as neural radiance fields for view synthesis\.In*ECCV*, 2020\.
- \[Narita et al\.\(2019\)Narita, Seno, Ishikawa, and Kaji\]Gaku Narita, Takashi Seno, Tomoya Ishikawa, and Yohsuke Kaji\.PanopticFusion: Online volumetric semantic mapping at the level of stuff and things\.In*IROS*, 2019\.
- \[Newcombe et al\.\(2011\)Newcombe, Izadi, Hilliges, Molyneaux, Kim, Davison, Kohli, Shotton, Hodges, and Fitzgibbon\]Richard A\. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J\. Davison, Pushmeet Kohli, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon\.KinectFusion: Real\-time dense surface mapping and tracking\.In*ISMAR*, 2011\.
- \[Nießner et al\.\(2013\)Nießner, Zollhöfer, Izadi, and Stamminger\]Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger\.Real\-time 3D reconstruction at scale using voxel hashing\.*ACM TOG*, 2013\.
- \[Oleynikova et al\.\(2017\)Oleynikova, Taylor, Fehr, Siegwart, and Nieto\]Helen Oleynikova, Zachary Taylor, Marius Fehr, Roland Siegwart, and Juan Nieto\.Voxblox: Incremental 3D Euclidean signed distance fields for on\-board MAV planning\.In*IROS*, 2017\.
- \[Pauly et al\.\(2002\)Pauly, Gross, and Kobbelt\]Mark Pauly, Markus Gross, and Leif P\. Kobbelt\.Efficient simplification of point\-sampled surfaces\.In*IEEE Vis\.*, 2002\.
- \[Placed and Castellanos\(2020\)\]Julio A Placed and José A Castellanos\.A deep reinforcement learning approach for active SLAM\.*Applied Sciences*, 2020\.
- \[Rosinol et al\.\(2020\)Rosinol, Abate, Chang, and Carlone\]Antoni Rosinol, Marcus Abate, Yun Chang, and Luca Carlone\.Kimera: an open\-source library for real\-time metric\-semantic localization and mapping\.In*ICRA*, 2020\.
- \[Schmid et al\.\(2022\)Schmid, Delmerico, Schönberger, Nieto, Pollefeys, Siegwart, and Cadena\]Lukas Schmid, Jeffrey Delmerico, Johannes L\. Schönberger, Juan Nieto, Marc Pollefeys, Roland Siegwart, and Cesar Cadena\.Panoptic multi\-TSDFs: a flexible representation for online multi\-resolution volumetric mapping and long\-term dynamic scene consistency\.In*ICRA*, 2022\.
- \[Schulman et al\.\(2017\)Schulman, Wolski, Dhariwal, Radford, and Klimov\]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov\.Proximal policy optimization algorithms\.*arXiv preprint arXiv:1707\.06347*, 2017\.
- \[Steinbrücker et al\.\(2014\)Steinbrücker, Sturm, and Cremers\]Frank Steinbrücker, Jürgen Sturm, and Daniel Cremers\.Volumetric 3D mapping in real\-time on a CPU\.In*ICRA*, 2014\.
- \[Stückler and Behnke\(2014\)\]Jörg Stückler and Sven Behnke\.Multi\-resolution surfel maps for efficient dense 3D modeling and tracking\.*JVCIR*, 2014\.
- \[Vespa et al\.\(2019\)Vespa, Funk, Kelly, and Leutenegger\]Emanuele Vespa, Nils Funk, Paul Kelly, and Stefan Leutenegger\.Adaptive\-resolution octree\-based volumetric SLAM\.In*3DV*, 2019\.
- \[Wu et al\.\(2021\)Wu, Wald, Tateno, Navab, and Tombari\]Shun\-Cheng Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, and Federico Tombari\.SceneGraphFusion: Incremental 3D scene graph prediction from RGB\-D sequences\.In*CVPR*, 2021\.
- \[Xie et al\.\(2021\)Xie, Wang, Yu, Anandkumar, Alvarez, and Luo\]Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M\. Alvarez, and Ping Luo\.SegFormer: Simple and efficient design for semantic segmentation with transformers\.In*NeurIPS*, 2021\.
- \[Zhang et al\.\(2018\)Zhang, Isola, Efros, Shechtman, and Wang\]Richard Zhang, Phillip Isola, Alexei A\. Efros, Eli Shechtman, and Oliver Wang\.The unreasonable effectiveness of deep features as a perceptual metric\.In*CVPR*, 2018\.
- \[Zheng et al\.\(2024\)Zheng, Barath, Pollefeys, and Armeni\]Jianhao Zheng, Daniel Barath, Marc Pollefeys, and Iro Armeni\.MAP\-ADAPT: Real\-time quality\-adaptive semantic 3D maps\.In*ECCV*, 2024\.
- \[Zhu et al\.\(2017\)Zhu, Mottaghi, Kolve, Lim, Gupta, Fei\-Fei, and Farhadi\]Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J\. Lim, Abhinav Gupta, Li Fei\-Fei, and Ali Farhadi\.Target\-driven visual navigation in indoor scenes using deep reinforcement learning\.In*ICRA*, 2017\.
- \[Zienkiewicz et al\.\(2016\)Zienkiewicz, Tsiotsios, Davison, and Leutenegger\]Jacek Zienkiewicz, Akis Tsiotsios, Andrew Davison, and Stefan Leutenegger\.Monocular, real\-time surface reconstruction using dynamic level of detail\.In*3DV*, 2016\.