Experimenting with an adaptive memory governor for PyTorch on an 8GB GPU — would love some feedback

(Quick note: English is not my native language, so I used AI to help translate my thoughts clearly. The code, the 38 unit tests, and the experiments on my RTX 5060 Ti are my own work.) Hey everyone, Lately I've been trying to train small models locally on an RTX 5060 Ti (8GB), and like many people, I kept running into frustrating CUDA OOMs whenever memory pressure fluctuated mid-run. Instead of just sticking to a static batch size and hoping the run survives, I started experimenting with a lightweight runtime control layer around PyTorch called MEM Orchestrator: https://github.com/nobazzy/mem-llm-orchestrator The basic hypothesis I've been testing is: Can the training runtime monitor VRAM pressure and temporarily downshift things like micro-batch size and gradient accumulation before hitting an OOM, and then step back up when memory clears? I also added a local policy check to clamp parameter adjustments into safe bounds, plus atomic checkpointing so a crash doesn't corrupt saved files. At least in my own local tests on this 8GB card, the approach seemed to help keep runs alive: I managed to run an endurance test of 1M steps on a 130M model without crashes, and tried stress-testing a 255M model with injected memory spikes (like adding +1.2GB during a 50k FineWeb-Edu run), where the controller seemed to adapt and let training continue. That said, this is just based on experiments on my own machine, and I know GPU memory management in PyTorch and CUDA is full of tricky edge cases. I'm definitely not claiming this solves OOM or that it's the right way to handle training. I mainly wanted to share the code and see what people with more experience in CUDA allocators, DeepSpeed, or distributed training think of the concept. If anyone has time to take a look, I'd really appreciate thoughts on whether this approach makes sense, or what obvious blind spots I might be missing.