How Fast Can Reward Models Score? A Systems Study of C++ and PyTorch Inference Runtimes for RLHF
Summary
This paper presents a systems study comparing C++ and PyTorch inference runtimes for reward model scoring in RLHF pipelines, finding that ONNXRuntime provides speedups on CPU while torch.compile leads on GPU, with batching strategy mattering more than language or runtime.
View Cached Full Text
Cached at: 07/29/26, 07:54 PM
Paper page - How Fast Can Reward Models Score? A Systems Study of C++ and PyTorch Inference Runtimes for RLHF
Source: https://huggingface.co/papers/2607.19712
Abstract
InRLHFpipelines,rewardscoringblockspolicyupdates.Slowscoringbottleneckstheentireloop,sincenoupdaterunsuntileveryrolloutgetsascore.AndyetmostsetupsjustdefaulttoPyTorcheagermodeortorch.compile,noonechecksifthat’sactuallyfastest.Scoringitselfissmall.RolloutgenerationeatsfarmoreofatypicalRLHFstep.ButscoringandgenerationfightoverthesameCPUandGPUresources,soafasterscoringenginedoesn’tshrinksteptimeonitsown.Itmainlyfreesupcapacitygenerationcanuseinstead.WebuiltanativeC++inferenceengineonONNXRuntime.Firststep:confirmcorrectness.OutputmatchedthePyTorchreferenceto5.7x10^-6onCPUand4.2x10^-3onGPU,closeenoughtotrust.ThenwetesteditagainstPyTorcheagermode,torch.compile,andFastAPI,onbothCPUandGPU.CPUwasdecisive.Ourenginebeateverybaseline,confidenceintervalsdidn’tevenoverlap.GPUgaveadifferentview:webeatPyTorchandFastAPI,buttorch.compilecameoutahead.FurthertestingtracedthespeeduptoONNXRuntimeitself,notC++asalanguage.Andbatchingstrategymatteredmorethaneitherthelanguageortheruntimechoice,morethanweexpected.Theresultsarefromrepeated,independentruns,sincesinglerunsjustaren’treliableenoughtotrust.
View arXiv pageView PDFProject pageGitHubAdd to collection
Get this paper in your agent:
hf papers read 2607\.19712
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.19712 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.19712 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.19712 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges
Survey introduces the Proxy Compression Hypothesis to explain how RLHF and related methods systematically induce reward hacking, deception, and oversight gaming in large language and multimodal models.
Benchmark: ONNX Runtime vs HF Transformers vs GGUF for Parakeet TDT 0.6B on CPU-only hardware [D]
A benchmark comparing ONNX Runtime, HF Transformers, and GGUF for the Parakeet TDT 0.6B ASR model on CPU-only hardware shows ONNX Runtime achieves 37% faster inference than HF Transformers bfloat16, while GGUF prioritizes memory efficiency.
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
OSReward introduces a standardized benchmark for evaluating VLM judges on computer-use agent trajectories, revealing that even state-of-the-art models have systematic leniency bias. The authors release OS-Shepherd-9B and 35B reward models that provide reliable, low-cost rewards for CUA tasks.
Rewriting model inference with CUDA kernels: the bottleneck was not just GEMM [P]
Author describes building FlashRT, a CUDA-first inference runtime that rewrites model inference paths with C++/CUDA kernels to address bottlenecks beyond GEMM for small-batch/realtime workloads, achieving significant latency improvements on Jetson Thor and RTX 5090. The article discusses lessons on precision (FP8 helpful, FP4 mixed) and the need to bypass generic runtimes for realtime inference.
Scaling laws for reward model overoptimization
OpenAI researchers empirically study how reward model overoptimization affects performance, establishing scaling laws that show the relationship between proxy reward optimization and ground truth performance varies by optimization method and scales predictably with model size.