Reinforcement Learning for Code Optimization

Hugging Face Daily Papers Papers
reinforcement-learning code-optimization reward-design benchmark grpo

Summary

This paper addresses challenges in using reinforcement learning for code optimization by proposing three stages: improved testing with DMC-Optim, converting execution time into reward via correctness-speed composition and an offline simulator, and adapting GRPO for noisy timing rewards. The method achieves substantial improvements on code optimization benchmarks.

RL for code correctness is now established: have the model generate a program, run it against hidden test cases, and reward solutions that pass. Extending this to code optimization seems straightforward: just add execution time to the reward. But in practice, once timing drives the reward, small problems in measurement noise, reward sparsity, or GRPO instability overwhelm the signal and make RL fail: generated solutions are barely faster, and more of them can fail. We make execution time learnable through three stages: (1) how code is tested, by building DMC-Optim with large optimization tests and a calibrated sandbox; (2) how speed is turned into reward, by composing correctness and speed in the RL environment and using an offline simulator to predict the most promising configurations; and (3) how the model learns from that reward, by adapting GRPO and evaluation to the sparser, noisier timed-execution setting. On DMC-Optim, the strongest optimization-aware configurations improve strict top-50% pass@1 from 18.0% to 31.3% on Qwen 2.5 7B and from 30.7% to 50.4% on CWM 32B. These gains further increase at stricter percentiles such as top-30%, with 125% relative improvement for CWM 32B, while preserving pure-correctness scores. When the timing sandbox is degraded, robust optimization RL reaches 100% to 200% improvement over standard RLVR, depending on the evaluation criterion. On LCB, CWM 32B wins up to 83% of median-sample speed comparisons against standard RLVR. Relative to the fastest correct human submissions per problem, it reaches about half the human rate of complexity-class improvements (14% vs. 28%).
Original Article
View Cached Full Text

Cached at: 07/29/26, 07:54 PM

Paper page - Reinforcement Learning for Code Optimization

Source: https://huggingface.co/papers/2607.25970

Abstract

RLforcodecorrectnessisnowestablished:havethemodelgenerateaprogram,runitagainsthiddentestcases,andrewardsolutionsthatpass.Extendingthistocodeoptimizationseemsstraightforward:justaddexecutiontimetothereward.Butinpractice,oncetimingdrivesthereward,smallproblemsinmeasurementnoise,rewardsparsity,orGRPOinstabilityoverwhelmthesignalandmakeRLfail:generatedsolutionsarebarelyfaster,andmoreofthemcanfail.Wemakeexecutiontimelearnablethroughthreestages:(1)howcodeistested,bybuildingDMC-Optimwithlargeoptimizationtestsandacalibratedsandbox;(2)howspeedisturnedintoreward,bycomposingcorrectnessandspeedintheRLenvironmentandusinganofflinesimulatortopredictthemostpromisingconfigurations;and(3)howthemodellearnsfromthatreward,byadaptingGRPOandevaluationtothesparser,noisiertimed-executionsetting.OnDMC-Optim,thestrongestoptimization-awareconfigurationsimprovestricttop-50%[email protected]%to31.3%onQwen2.57Bandfrom30.7%to50.4%onCWM32B.Thesegainsfurtherincreaseatstricterpercentilessuchastop-30%,with125%relativeimprovementforCWM32B,whilepreservingpure-correctnessscores.Whenthetimingsandboxisdegraded,robustoptimizationRLreaches100%to200%improvementoverstandardRLVR,dependingontheevaluationcriterion.OnLCB,CWM32Bwinsupto83%ofmedian-samplespeedcomparisonsagainststandardRLVR.Relativetothefastestcorrecthumansubmissionsperproblem,itreachesabouthalfthehumanrateofcomplexity-classimprovements(14%vs.28%).

View arXiv pageView PDFAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.25970 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.25970 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.25970 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

RLPF: Reinforcement Learning from Performance Feedback for Code Generation

arXiv cs.LG

RLPF is a reinforcement learning method that trains code models to optimize runtime in addition to correctness, using staged rewards based on execution progress and relative efficiency. Fine-tuning Qwen3-32B with RLPF on PerfCodeBench raises correct-and-runnable solutions from 11.1% to 54.6% and improves relative efficiency from 8.1% to 38.6%.