NOML-NOML: hierarchical TD3 + anchor policy for flight control [P]

Reddit r/MachineLearning Tools
reinforcement-learning continuous-control flight-control hierarchical anchor-policy open-source

Summary

Introduced NOML, a custom reinforcement learning algorithm for continuous flight control that uses a hierarchical actor, anchor policy, and mirror learning to prevent oscillation and improve stability. The code is open-sourced on GitHub.

I built a custom RL algorithm for continuous flight control and open-sourced it. Sharing here in case the structural ideas are useful for anyone doing continuous control where one action axis dominates. I've been training continuous control on a 6-DoF flight sim (pitch/roll/yaw/throttle/brake/fire) and kept hitting the same wall: vanilla TD3 would peak, then collapse into pitch oscillation and never recover. I tried reward shaping for a while before concluding the problem was structural, not in the reward. NOML is what came out of that. Three structural changes on top of a standard TD3 skeleton: * **Anchor policy** — the action is `anchor + delta·gate`, where the anchor is a fixed safe action (wings level, MIL throttle). The policy literally cannot fully forget how to fly straight; the worst a collapsed policy can do is fall back to the anchor. * **Hierarchical actor** — three MLPs with independent optimizers (pitch → roll → rest), so a roll-side gradient update can't corrupt the pitch head. This is what actually killed the oscillation for me. * **Mirror learning** — left-right symmetry means every transition can be mirrored into a free second sample. 2× data when env steps are the bottleneck. One thing that surprised me and goes against the usual advice: my best results came with exploration noise effectively off. On this task adding Gaussian action noise mostly just shook the stick and hurt. The anchor+gate structure seems to provide enough of the "fall back to safe behavior" role that noise usually plays. Code (Apache 2.0), full writeup, and a test video are here: [https://github.com/9138noms/NOML](https://github.com/9138noms/NOML) [https://www.youtube.com/watch?v=ZNn6wo\_PX8Y](https://www.youtube.com/watch?v=ZNn6wo_PX8Y)
Original Article

Similar Articles

Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Control

arXiv cs.AI

This paper proposes a hierarchical multi-agent reinforcement learning framework that enforces hard safety constraints via a constraint manifold at the low level while enabling effective coordination through high-level policy learning, providing theoretical safety guarantees and achieving near-perfect safety rates with good generalization.

MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training

Hugging Face Daily Papers

MOPD proposes a multi-teacher on-policy distillation paradigm for LLM post-training, enabling efficient integration of multiple domain capabilities by distilling specialized RL teachers into a student model using its own rollouts. It outperforms existing methods like Mix-RL and Cascade RL, and has been deployed in industrial-scale models.