EvalStop: Using World Feedback to Detect and Correct Reward Overoptimization in Multi-Tenant RLHF Platforms

# Using World Feedback to Detect and Correct Reward Overoptimization in Multi-Tenant RLHF Platforms
Source: [https://arxiv.org/html/2606.04145](https://arxiv.org/html/2606.04145)
###### Abstract

Background\.Cloud LLM fine\-tuning platforms increasingly serve RLHF workloads, where a learned reward model is optimized as a*proxy*for human quality\. AsGaoet al\.\([2023](https://arxiv.org/html/2606.04145#bib.bib10)\)showed, this proxy diverges from*world feedback*\(downstream eval metrics\) under sustained optimization pressure: the*reward overoptimization*phenomenon\.Limitations\.Existing platform schedulers ignore this divergence: non\-clairvoyant schedulers optimize JCT without any quality signal, SLAQ\-style quality\-aware schedulers use training loss \(a weaker proxy that drops monotonically through hacking\), and classical per\-job early stopping requires human monitoring and does not free shared GPUs\.Approach\.We proposeEvalStop, a composable scheduling primitive that terminates jobs onkkconsecutive eval\-score declines, releases GPUs, preserves the best checkpoint, and delegates to any base scheduler\. We frame scheduler\-level early stopping as a*detection*problem and evaluate it in a discrete\-event simulator whose RLHF workload mixes reward\-hacking and structurally healthy runs, with ground\-truth labels hidden from schedulers\.Results\.On RLHF\-heavy workloads \(80% RLHF, 64 GPUs\), EvalStop achieves precision 98% / recall 99% / FPR 1\.5% while improving JCT by 9% and cutting wasted compute by 22% over SRTF\-Est \(p<0\.05p\{<\}0\.05\)\. Trivial fixed\-progress and loss\-plateau competitors either incur 65% FPR or miss over half of true hacking cases\. Gains compose across base schedulers \(9–25% JCT\) and stay stable under eval noise \(σ≤0\.05\\sigma\{\\leq\}0\.05: precision≥\\geq91%\) and hacking base rate \(precision≥\\geq89% across 20–80%\)\.

![Refer to caption](https://arxiv.org/html/2606.04145v1/x1.png)Figure 1:EvalStop architecture\.*World Feedback*flows into the*Decline Detector*; onkkconsecutive eval drops the wrapper performs*Stop and Save*\(release GPUs, retain best checkpoint\), then delegates the updated cluster state to any*Base Scheduler*\. A non\-clairvoyant*Information Boundary*\(bottom band\) underlies the whole pipeline so detection quality is honestly measurable\. Each module maps 1:1 to a contribution in §[1](https://arxiv.org/html/2606.04145#S1): \{World Feedback, Information Boundary\}→\\toC1; \{Decline Detector, Stop and Save\}→\\toC2; \{Base Scheduler\}→\\toC3\.## 1Introduction

Background\.Cloud\-based LLM fine\-tuning platforms serve diverse workloads \(LoRA adapter tuning, DPO\(Rafailovet al\.,[2023](https://arxiv.org/html/2606.04145#bib.bib12)\), and RLHF\(Ouyanget al\.,[2022](https://arxiv.org/html/2606.04145#bib.bib9)\)\) from multiple tenants on shared GPU clusters, and as RLHF becomes the dominant method for LLM alignment\(Stiennonet al\.,[2020](https://arxiv.org/html/2606.04145#bib.bib11); Ouyanget al\.,[2022](https://arxiv.org/html/2606.04145#bib.bib9)\)the fraction of RLHF workloads on these platforms is growing rapidly\. RLHF differs structurally from supervised fine\-tuning: the policy is trained to maximize a learned reward model, but this reward is a*proxy*for human preferences and the policy’s true quality is measured only by downstream evaluation \(held\-out benchmarks, win\-rate on a held\-out preference set, task\-specific metrics\)\. A scheduler thus has access to three signals of decreasing proxiness: training loss \(proxy2, optimized by the RL algorithm\), reward model score \(proxy, what the policy directly optimizes\), and downstream eval score \(*world feedback*, grounded in task quality but delayed and noisy\)\.Gaoet al\.\([2023](https://arxiv.org/html/2606.04145#bib.bib10)\)showed that as optimization pressure increases the first two rise monotonically while the third peaks and then degrades\. This is the*reward overoptimization*phenomenon\(Skalseet al\.,[2022](https://arxiv.org/html/2606.04145#bib.bib19); Panet al\.,[2022](https://arxiv.org/html/2606.04145#bib.bib20)\); Figure[2](https://arxiv.org/html/2606.04145#S2.F2)\(§[2\.3](https://arxiv.org/html/2606.04145#S2.SS3)\) illustrates it on a representative run\.

Prior work and its limitations\.Three threads of prior work touch this setting but none address it directly\. \(i\)*Non\-clairvoyant ML schedulers*\(Tiresias\(Guet al\.,[2019](https://arxiv.org/html/2606.04145#bib.bib1)\), Pollux\(Qiaoet al\.,[2021](https://arxiv.org/html/2606.04145#bib.bib3)\), Gavel\(Narayananet al\.,[2020](https://arxiv.org/html/2606.04145#bib.bib4)\), Sia\(Jayaramet al\.,[2023](https://arxiv.org/html/2606.04145#bib.bib5)\)\) optimize JCT without using any quality signal and therefore cannot tell whether GPU minutes spent late in an RLHF job are productive or wasted\. \(ii\)*Quality\-aware schedulers*, the closest prior work, rely on training loss: SLAQ\(Zhanget al\.,[2017](https://arxiv.org/html/2606.04145#bib.bib2)\)preferentially allocates GPUs to jobs whose loss is improving fastest\. This works for convex losses where loss correlates with quality, but for RLHF the loss is precisely the proxy that decouples from world feedback during overoptimization, so a loss\-aware scheduler will keep allocating resources to a hacking job\. \(iii\)*Reward\-hacking mitigations*from the RL community\(Moskovitzet al\.,[2024](https://arxiv.org/html/2606.04145#bib.bib18); Gaoet al\.,[2023](https://arxiv.org/html/2606.04145#bib.bib10)\)operate*within*the training loop \(constrained RL, reward ensembles, SFT regularization\)\. They aim to prevent hacking but in production deployments imperfect mitigation is the norm, and these techniques offer no path to reclaiming GPUs from jobs that have already diverged\. Classical early stopping\(Prechelt,[1998](https://arxiv.org/html/2606.04145#bib.bib23)\)similarly requires per\-job human monitoring and does not translate into platform\-wide resource reallocation\.

Motivation\.We argue the right place to act on world feedback is the*scheduler*, not the trainer: the scheduler already controls GPU allocation and termination, and a single scheduler change applies uniformly across every tenant’s RLHF jobs\. Early experimentation with an eval\-aware*priority*scheduler \(EvalSched, §[4](https://arxiv.org/html/2606.04145#S4)\) revealed that simply deprioritizing post\-peak jobs is not enough: deprioritized jobs still occupy the system, and EvalSched’s JCT is 2\.2×\\timesworse than SRTF\-Est\. The fix is to*terminate*declining jobs rather than starve them\. But blunt termination \(e\.g\. stopping every RLHF job at a fixed progress\) would pay heavy false\-positive costs on the structurally healthy RLHF runs that exist in any real workload\. This reframes the problem as a*detection*task: the scheduler must discriminate hacking from healthy runs using only the signals it can legally observe \(eval scores at scheduled checkpoints\), without modifying training and without violating the non\-clairvoyant boundary\(Motwaniet al\.,[1994](https://arxiv.org/html/2606.04145#bib.bib21)\)\.

Contributions\.We proposeEvalStop, a composable scheduling primitive whose architecture \(Figure[1](https://arxiv.org/html/2606.04145#S0.F1)\) factors into five modules; each contribution maps to one or more of them\.

- •C1 — Detection framing\(*World Feedback*input \+*Information Boundary*\)\. We argue scheduler\-level early stopping for RLHF is a*detection*problem: the scheduler consumes world feedback and must report which jobs are hacking\. Prior framings that report only JCT/TTFUC miss this axis\. Our experimental design hides ground\-truth hacking labels behind a whitelist proxy \(SchedulerJobView, the Information Boundary\), so detector quality is honestly measurable\.
- •C2 — A composable detector that wins on the detection axis\(*Decline Detector*\+*Stop and Save*\)\. EvalStop tracks per\-job consecutive eval\-score declines and, onkkconsecutive drops, terminates the job and preserves its best checkpoint\. On RLHF\-heavy workloads \(80% RLHF, 64 GPUs\) it achieves precision 98% / recall 99% / FPR 1\.5%, dominating both a trivial fixed\-progress stop rule \(FPR 65%\) and a strong loss\-plateau detector \(recall 38%\)\.
- •C3 — Composable systems gains across base schedulers and workloads\(*Base Scheduler*delegation\)\. The wrapper hands the updated cluster state back to any scheduler \(FIFO, SJF\-Est, SRTF\-Est, LossAware\), inheriting its resource\-allocation policy\. EvalStop yields 9–25% JCT improvement and∼22%\{\\sim\}22\\%wasted\-compute reduction on top of every tested base scheduler, and its detection quality is stable under both eval noise \(σ≤0\.05\\sigma\{\\leq\}0\.05: precision≥\\geq91%\) and hacking base rate \(precision≥\\geq89% across 20–80%\)\.

## 2Background and Problem Setup

### 2\.1Job Model

A fine\-tuning jobjjhas type∈\{LoRA,DPO,RLHF\}\\in\\\{\\text\{LoRA\},\\text\{DPO\},\\text\{RLHF\}\\\}, GPU demand, an eval schedule \(progress fractions at which evaluation runs\), and a training curve mapping progress to \(loss, eval\_score\)\. Job types differ structurally: LoRA jobs are short \(10–60 min, 1–2 GPUs\) with monotonically improving eval; DPO jobs are medium \(30–120 min, 2–4 GPUs\) with diminishing returns; RLHF jobs are long \(60–360 min, 4–8 GPUs\) with eval that peaks then degrades\.

### 2\.2Eval\-Aware Metrics

Beyond JCT, we define: \(1\)*TTFUC*\(Time\-to\-First\-Useful\-Checkpoint\): time from arrival until first checkpoint exceeding quality thresholdτ\\tauand improving on the previous best by≥1%\{\\geq\}1\\%; \(2\)*Wasted Compute Fraction*: GPU\-minutes spent training after a job’s eval peak, divided by total GPU\-minutes; \(3\)*Saved Compute Fraction*: GPU\-minutes avoided via early stopping, divided by total planned GPU\-minutes\.

### 2\.3Proxy Signals vs World Feedback

Figure[2](https://arxiv.org/html/2606.04145#S2.F2)illustrates the core problem on a representative RLHF training run\. Three signals are available to a scheduler:

- •Training lossdecreases monotonically throughout training\. A SLAQ\-style\(Zhanget al\.,[2017](https://arxiv.org/html/2606.04145#bib.bib2)\)loss\-aware scheduler would interpret this as “the job is making good progress, keep running\.”
- •Reward model score\(normalised1−loss/loss01\-\\text\{loss\}/\\text\{loss\}\_\{0\}\) also increases monotonically, since the policy is directly optimizing this objective\.
- •Eval score\(downstream benchmark\) rises to a peak at∼\{\\sim\}55% progress, then degrades as reward hacking takes over\.

Only the eval score \(the*world feedback*signal\) reveals that the job has entered a regime of diminishing or negative returns\. A scheduler using proxy signals would allocate*more*resources to this job post\-peak \(steepest loss improvement\), while a world\-feedback\-aware scheduler would terminate it\.

![Refer to caption](https://arxiv.org/html/2606.04145v1/x2.png)Figure 2:Proxy vs world feedback in RLHF training\. Training loss \(proxy2, red\) and reward model score \(proxy, orange dashes\) both indicate continued improvement\. Only the eval score \(world feedback, blue\) reveals that quality has peaked and is degrading\. A scheduler using proxy signals would*increase*allocation; EvalStop uses world feedback to*terminate*\.This proxy–world divergence is well\-documented empirically\.Gaoet al\.\([2023](https://arxiv.org/html/2606.04145#bib.bib10)\)showed predictable overoptimization scaling laws for KL\-constrained RLHF policies\.Rafailovet al\.\([2024](https://arxiv.org/html/2606.04145#bib.bib17)\)extended these findings to DPO, and multiple works have proposed training\-level mitigations: constrained RL\(Moskovitzet al\.,[2024](https://arxiv.org/html/2606.04145#bib.bib18)\), reward model ensembles, and SFT regularization\. EvalStop is*complementary*to these approaches: rather than modifying the training algorithm, it uses world feedback as an external control signal at the scheduler level to terminate jobs that have diverged\.

## 3EvalStop: World\-Feedback\-Driven Early Stopping

Recall the architecture overview in Figure[1](https://arxiv.org/html/2606.04145#S0.F1)\. EvalStop is acomposable wrapperaround any base scheduling policy\. It monitors eval\-score trajectories \(the world feedback signal\) and early\-stops jobs when quality is irrecoverably declining\.

Algorithm 1EvalStop scheduling wrapper0:Base scheduler

𝒮\\mathcal\{S\}, decline thresholds

kRLHF=2k\_\{\\text\{RLHF\}\}\{=\}2,

kDPO=3k\_\{\\text\{DPO\}\}\{=\}3
1:State:per\-job consecutive decline count

d​\[j\]d\[j\]
2:on\_eval\_result\(j,score\)\(j,\\text\{score\}\):

3:if

score<prev\_score​\[j\]\\text\{score\}<\\text\{prev\\\_score\}\[j\]then

4:

d​\[j\]←d​\[j\]\+1d\[j\]\\leftarrow d\[j\]\+1
5:else

6:

d​\[j\]←0d\[j\]\\leftarrow 0
7:endif

8:if

d​\[j\]≥ktype​\(j\)d\[j\]\\geq k\_\{\\text\{type\}\(j\)\}then

9:Mark

jjfor early stopping

10:endif

11:schedule\(cluster, waiting, running\):

12:foreach

j∈j\\inrunning marked for early stoppingdo

13:Terminate

jj; release GPUs; save best checkpoint

14:endfor

15:return

𝒮\\mathcal\{S\}\.schedule\(cluster, waiting, running′\)

Early stopping vs\. deprioritization\.EvalSched deprioritizes post\-peak RLHF jobs, creating starvation: deprioritized jobs still occupy the system, receiving occasional time slices but never completing efficiently\. EvalStop*removes*the job from the system, freeing GPUs immediately\. The job’s best checkpoint is preserved; since the trigger requireskkconsecutive declines, it was savedkkeval steps ago\.

Information boundary\.EvalStop respects the non\-clairvoyant boundary\(Motwaniet al\.,[1994](https://arxiv.org/html/2606.04145#bib.bib21)\): it observes only job type, eval scores at scheduled checkpoints, and the count of consecutive declines\. It never accesses true job duration\.

Composability\.EvalStop wraps any scheduler implementation \(Algorithm[1](https://arxiv.org/html/2606.04145#alg1)\)\. It processes early\-stop decisions first \(freeing GPUs\), then delegates to the base scheduler, which sees the updated cluster state\. This clean separation means EvalStop can be added to existing production schedulers \(including RLHF\-specific frameworks like OpenRLHF\(Huet al\.,[2024](https://arxiv.org/html/2606.04145#bib.bib22)\)\) with minimal integration effort\.

Relationship to change\-point detection\.EvalStop’s mechanism \(detectingkkconsecutive declines in a time series\) can be viewed as a simplified change\-point detector\. We chose this over more sophisticated methods \(e\.g\., CUSUM, Bayesian change\-point detection\) for simplicity and interpretability: the thresholdkkhas a direct operational meaning \(“how many bad evals before we stop”\), which platform operators can reason about\. Section[4\.4](https://arxiv.org/html/2606.04145#S4.SS4)validates that this simple mechanism is robust across a range ofkkvalues\.

## 4Experiments

Simulator\.We built a discrete\-event simulator modelling a multi\-tenant fine\-tuning platform with heap\-based event dispatch, slot\-based GPU allocation with 2\-minute preemption overhead, and Poisson arrivals\. Training curves are parameterised per job type: LoRA \(monotonic exponential convergence\), DPO \(saturating gain with small plateau\), and RLHF\.Our RLHF workload is a mixture: 60% of RLHF jobs exhibit classical reward hacking \(eval peaks in\[0\.55,0\.75\]\[0\.55,0\.75\]progress and then declines, calibrated toGaoet al\.\([2023](https://arxiv.org/html/2606.04145#bib.bib10)\)\) while 40% are structurally “healthy” \(reward rises monotonically with noise through progress=1\.0\{=\}1\.0\)\. Training loss in both regimes decays exponentially to a plateau around 50–70% progress, matching real RLHF convergence behaviour\. This design forces any scheduler\-level detector to*discriminate*hacking from healthy runs; a detector that blindly kills every RLHF job will incur a large false\-positive rate on the healthy subset\. Ground\-truth hacking labels and peak\-progress are stored on the job but hidden from schedulers through a whitelist\-based proxy \(SchedulerJobView\)\.

Baselines\.We compare against scheduling\-only baselines and early\-stop detectors:*FIFO*,*SJF\-Est*\(non\-clairvoyant shortest\-job\-first\),*SRTF\-Est*\(preemptive shortest\-remaining\-time\-first\),*LossAware*\(SLAQ\-style\(Zhanget al\.,[2017](https://arxiv.org/html/2606.04145#bib.bib2)\)using training loss for priority\),*EvalSched*\(eval\-aware with deprioritisation only, no termination\), plus two strong early\-stop competitors:*StopAtpp\+SRTF*: a trivial oracle\-flavoured rule that terminates every RLHF job at fixed progressp∈\{0\.50,0\.65\}p\\in\\\{0\.50,0\.65\\\};*LossPlateau\+SRTF*: a principled loss\-only detector that early\-stops when relative loss improvement falls below 2% over a 3\-checkpoint window\. The first isolates “how much benefit comes merely from stopping long jobs early”; the second is the natural SLAQ\+\+ competitor that uses training\-loss plateau, not eval, to decide when to stop\.

Experiment matrix\.All experiments use 200 jobs, 5 tenants, and 5 seeds \(42, 123, 456, 789, 1024\)\. Statistical significance is assessed via Welch’s t\-test\. E1: mixed workload \(50/30/20 LoRA/DPO/RLHF, 32 GPUs\)\. E2: RLHF\-heavy \(80%, 64 GPUs\)\. E3: workload sensitivity \(3 mixes\)\. E4: composability \(EvalStop on 4 base schedulers\)\. E5: threshold sensitivity \(k∈\{1\.\.5\}k\\in\\\{1\.\.5\\\}\)\. E6: eval frequency sensitivity \(5–30% progress intervals\)\.

### 4\.1Main Result: Detection Quality and Systems Gains \(E2\)

Table 1:RLHF\-heavy workload \(80% RLHF, 64 GPUs, 200 jobs, 5 seeds\) with mixed hacking/healthy RLHF curves\. Early\-stop detectors are evaluated both on systems metrics \(JCT, TTFUC, wasted/saved compute\) and as classifiers of reward\-hacking runs \(precision, recall, FPR, computed against ground\-truth labels hidden from schedulers\)\.Boldmarks the best value;redmarks a destructive value\. Only EvalStop simultaneously achieves high precision and low FPR\.Table[1](https://arxiv.org/html/2606.04145#S4.T1)is the paper’s central result\. Two observations dominate:

\(i\) Systems metrics alone are deceptive\.The trivial StopAt\-0\.5 and StopAt\-0\.65 baselines \(which stop*every*RLHF job at a fixed progress\) achieve the best JCT and lowest wasted compute\. If we evaluated only JCT/TTFUC/Wasted we would conclude that eval\-awareness is unnecessary and a fixed\-progress heuristic suffices\. Their precision, however, is 57% and their false\-positive rate on healthy RLHF is 64\.5%: they destroy two thirds of the RLHF jobs that were converging normally, a quality cost that does not appear on any of the first four columns\.

\(ii\) Loss\-only detection is insufficient\.LossPlateau\+SRTF is a principled SLAQ\-style competitor that early\-stops on loss plateau alone\. It achieves recall of only 38\.3% \(it misses more than half of the truly\-hacking jobs\) while still firing on 24\.7% of healthy runs, because training loss plateaus at similar progress in*both*hacking and healthy RLHF\. Loss plateau is correlated with overoptimisation but does not discriminate from normal convergence\.

EvalStop\+SRTF, which uses the world\-feedback \(eval\) signal, is the only detector in Table[1](https://arxiv.org/html/2606.04145#S4.T1)that achieves high precision \(98\.3%\) and near\-zero FPR \(1\.5%\)*simultaneously with*a systems improvement over the no\-early\-stop baselines:\+9\.4%\+9\.4\\%JCT and−21\.8%\-21\.8\\%wasted compute over SRTF\-Est \(Welch’stt\-test,p=0\.03p\{=\}0\.03andp<0\.001p\{<\}0\.001respectively\)\.

Fairness\.EvalStop does not harm inter\-tenant fairness: Jain’s index is within noise of SRTF\-Est alone\.

### 4\.2Composability \(E4\)

Table 2:EvalStop as a composable overlay\. Improvement of EvalStop\+X over base X on RLHF\-heavy workload \(80% RLHF, 64 GPUs\)\. Precision and FPR measured against ground\-truth reward\-hacking labels\.Table[2](https://arxiv.org/html/2606.04145#S4.T2)shows that EvalStop improves*every*base scheduler with consistent gains:\+\+9–25% JCT,\+\+5–9% TTFUC,−\-20–22% wasted compute\. Precision stays above 97% and FPR below 3% in every combination\. The large JCT gain on LossAware reflects that LossAware alone has pathological priorities on this workload \(it over\-prioritises RLHF jobs whose loss is still dropping\); EvalStop partially compensates by terminating the overoptimising ones\. The consistency suggests that EvalStop addresses a source of waste \(post\-peak RLHF training driven by proxy/world divergence\) that is orthogonal to the base scheduling policy’s resource allocation strategy\.

### 4\.3Workload Sensitivity \(E3\)

Table 3:EvalStop\+SRTF vs\. SRTF\-Est across workload mixes \(64 GPUs\)\. Benefit scales with the hacking\-RLHF fraction\. Precision and FPR are measured against ground\-truth reward\-hacking labels\.Table[3](https://arxiv.org/html/2606.04145#S4.T3)confirms that EvalStop’s benefit scales with the hacking\-RLHF fraction\. This is expected: EvalStop fires only on jobs exhibiting eval\-score decline, which our workload only constructs for a subset of RLHF runs\. On LoRA\-heavy workloads \(10% RLHF\), EvalStop provides measurable wasted\-compute reduction with*perfect*precision and no false positives; it does not fire spuriously on LoRA, DPO, or healthy RLHF runs\. On a mixed workload \(E1: 50/30/20, 32 GPUs; see Appendix[B](https://arxiv.org/html/2606.04145#A2)\), EvalStop\+SRTF matches the trivial StopAt\-0\.65 on TTFUC while cutting the latter’s FPR from 9\.7% to 0\.0%\.

### 4\.4Threshold Sensitivity \(E5\)

![Refer to caption](https://arxiv.org/html/2606.04145v1/x3.png)Figure 3:E5: Effect of decline thresholdkkon EvalStop\+SRTF \(80% RLHF, 64 GPUs, 5 seeds\)\.k=2k\{=\}2\(green border\) balances early detection against false positives\.k=1k\{=\}1is too aggressive \(stops 160 jobs\);k≥4k\{\\geq\}4barely triggers\.Figure[3](https://arxiv.org/html/2606.04145#S4.F3)shows EvalStop’s sensitivity to the decline thresholdkk\. Atk=1k\{=\}1, EvalStop aggressively stops 127 jobs after a single eval decline \(JCT=\{=\}874, wasted=\{=\}19\.4%\), but risks false positives from noisy eval scores on healthy RLHF\. Atk=2k\{=\}2\(our default\), 93 jobs are stopped with JCT=\{=\}1018 and wasted=\{=\}26\.9%, a substantial improvement over the no\-early\-stop baseline \(SRTF JCT=\{=\}1124, wasted=\{=\}34\.5%\) while keeping FPR below 2%\. Atk≥4k\{\\geq\}4, EvalStop rarely triggers \(≤\{\\leq\}2 jobs stopped\), converging to baseline SRTF behaviour\.

Thek=2k\{=\}2choice reflects a precision/recall trade\-off: once world feedback shows*two consecutive*declines, the overoptimisation trend is likely real rather than noise, yielding the 98% precision / 99% recall numbers of Table[1](https://arxiv.org/html/2606.04145#S4.T1)\. This is a simple but effective change\-point heuristic\.

### 4\.5Eval Frequency Sensitivity \(E6\)

![Refer to caption](https://arxiv.org/html/2606.04145v1/x4.png)Figure 4:E6: Effect of eval frequency on EvalStop\+SRTF \(80% RLHF, 64 GPUs, 5 seeds\)\. More frequent evals \(5% intervals\) enable earlier detection and greater compute savings \(24%\), at the cost of more eval overhead\. Default 15% interval \(green border\) balances detection speed with evaluation cost\.Figure[4](https://arxiv.org/html/2606.04145#S4.F4)reveals a clear trade\-off between eval frequency and early\-stopping effectiveness\. With evals every 5% of training progress, EvalStop detects overoptimisation earlier and saves 24% of planned compute \(JCT=\{=\}867\)\. With evals every 30%, detection is delayed and less than 1% is saved \(JCT=\{=\}1123, essentially matching the no\-early\-stop baseline\)\.

This result has practical implications for platform operators: investing in more frequent evaluation \(even if each eval consumes GPU time\) can*more than pay for itself*through earlier overoptimisation detection\. The default 15% interval represents a reasonable balance for current RLHF workloads\.

### 4\.6Detector Robustness \(E7, E8\)

A natural concern with the Table[1](https://arxiv.org/html/2606.04145#S4.T1)numbers is that EvalStop’s 98% precision / 1\.5% FPR might be an artefact of a particular eval\-noise setting or of the specific 60% hacking\-fraction used to generate the workload\. To address this we ran two additional sweeps on RLHF\-heavy workloads; Figure[5](https://arxiv.org/html/2606.04145#S4.F5)summarises both\.

![Refer to caption](https://arxiv.org/html/2606.04145v1/x5.png)Figure 5:Detector robustness\.Left \(E7\):precision vs\. eval\-noise standard deviation\. EvalStop degrades gracefully \(100%→\\to81%\); the loss\-only and progress\-triggered baselines do not use eval and sit flat at∼52\{\\sim\}52–57%57\\%\. Green shading marks the realistic regime anchored to typical LLM benchmark standard errors\(Gaoet al\.,[2023](https://arxiv.org/html/2606.04145#bib.bib10)\)\.Right \(E8\):precision vs\. hacking base rate\. EvalStop stays above 89% across 20–80% base rates; StopAt\-0\.65’s precision follows the base\-rate identity line exactly, confirming it is a constant classifier rather than a detector\.E7: Eval\-noise sensitivity\.We vary the standard deviation of the per\-checkpoint eval\-score noise on RLHF jobs fromσ=0\.01\\sigma\{=\}0\.01\(near\-deterministic\) toσ=0\.12\\sigma\{=\}0\.12\(highly noisy, well above typical downstream\-benchmark variance\)\. Table[4](https://arxiv.org/html/2606.04145#S4.T4)reports precision, recall, and FPR for the three detectors\.

Table 4:E7: Eval\-noise sensitivity \(80% RLHF, 64 GPUs, 60% hacking\)\. EvalStop precision degrades smoothly; the progress\-triggered and loss\-plateau baselines are unaffected by eval noise but stay at low precision\.Precision degrades*gracefully*from 100% atσ=0\.01\\sigma\{=\}0\.01to 81% atσ=0\.12\\sigma\{=\}0\.12: there is no cliff at which EvalStop becomes indistinguishable from the simpler rules\. At every noise level EvalStop retains a precision gap of at least 24 pp over both competitors, because \(i\) StopAt\-0\.65 ignores the eval signal entirely and therefore does not degrade*or*improve with noise, and \(ii\) LossPlateau relies on the training\-loss signal, which we do not inject eval noise into; yet its precision stays at∼52%\{\\sim\}52\\%regardless, because loss plateaus at similar progress in both hacking and healthy regimes\. A stronger loss\-only detector \(e\.g\. using loss curvature, or a learned classifier on loss\+\+gradient features\) would likely narrow this gap, but cannot close it: training loss is monotonically non\-increasing in*both*regimes, so no purely loss\-based signal can perfectly separate hacking from healthy RLHF\. At realistic eval\-noise magnitudes \(σ≤0\.05\\sigma\{\\leq\}0\.05, matching typical LLM benchmark standard errors of 2–5%\(Gaoet al\.,[2023](https://arxiv.org/html/2606.04145#bib.bib10)\)\) EvalStop’s precision remains above 91%\.

E8: Hacking\-fraction sensitivity\.We next vary the fraction of RLHF jobs that actually exhibit reward hacking between 20% and 80%\. Precision is base\-rate\-sensitive by definition, so this is the right robustness axis for the detection story\.

Table 5:E8: Hacking\-fraction sensitivity \(80% RLHF, 64 GPUs, defaultσ\\sigma\)\. EvalStop’s precision stays above 89% even at 20% hacking base rate; StopAt\-0\.65 collapses to 20% precision / 77% FPR, confirming its “success” in Table[1](https://arxiv.org/html/2606.04145#S4.T1)was driven by the assumed base rate\.EvalStop’s precision*stays above 89%*across the entire 20–80% range\. By contrast, StopAt\-0\.65’s precision is*exactly equal*to the hacking fraction because it stops every RLHF job indiscriminately; it is a constant classifier whose apparent 57% precision at our default 60% setting is purely a base\-rate artefact\. EvalStop discriminates on the eval signal, so its precision is driven by detection quality rather than by the prevalence of the positive class, the classical statistical property one expects of a genuine detector\.

## 5Discussion

World feedback as a scheduling signal\.Our results show that downstream evaluation \(world feedback\) is a better signal for scheduling RLHF jobs than training loss \(proxy2\) or reward model score \(proxy\)\. This aligns with the growing recognition that proxy optimization in RLHF requires external grounding\(Gaoet al\.,[2023](https://arxiv.org/html/2606.04145#bib.bib10); Skalseet al\.,[2022](https://arxiv.org/html/2606.04145#bib.bib19); Moskovitzet al\.,[2024](https://arxiv.org/html/2606.04145#bib.bib18)\)\. EvalStop operationalizes this insight at the scheduler level: it transforms eval from a passive monitoring signal into an active control signal that shapes resource allocation\.

Complementarity with training\-level mitigations\.Prior work on reward hacking mitigation operates*within*the training loop: constrained RL\(Moskovitzet al\.,[2024](https://arxiv.org/html/2606.04145#bib.bib18)\), reward model ensembles, SFT regularization\. EvalStop operates*outside*the training loop, at the scheduler level\. These approaches are complementary: even with perfect reward hacking mitigation, EvalStop would correctly do nothing \(no eval decline⇒\\Rightarrowno early stopping\)\. With imperfect mitigation \(the current reality\), EvalStop provides a safety net\.

Quality preservation and false positives\.Early\-stopped jobs retain their best checkpoint; sincekkconsecutive declines are required, the best checkpoint was recordedkkeval steps ago\. In our simulator EvalStop’s false\-positive rate on healthy RLHF is 1\.5% \(Table[1](https://arxiv.org/html/2606.04145#S4.T1)\), meaning roughly one in seventy healthy RLHF jobs is terminated slightly early\. In contrast the progress\-triggered StopAt\-0\.65 baseline terminates∼\\sim65% of healthy RLHF jobs\. This difference \(detection vs\. unconditional termination\) is invisible in JCT/TTFUC alone and is the most important quality axis for any production deployment\.

Generality beyond RLHF\.The underlying principle \(using world feedback to detect when proxy optimisation has diverged\) applies to any training regime where the optimised objective can decouple from true quality\. This includes DPO with overtraining\(Rafailovet al\.,[2024](https://arxiv.org/html/2606.04145#bib.bib17)\), overfitting in small\-data fine\-tuning, and potentially mode collapse in generative models\. Empirical validation in those regimes is future work\.

Limitations\.\(1\)*Synthetic training curves\.*RLHF curves are parametric: hacking runs are calibrated to the overoptimisation dynamics ofGaoet al\.\([2023](https://arxiv.org/html/2606.04145#bib.bib10)\), healthy runs produce monotonically improving eval, and loss plateaus at𝒰​\(0\.5,0\.7\)\\mathcal\{U\}\(0\.5,0\.7\)progress in both\. Both regimes are idealisations\. Replaying the detector on publicly available RLHF/DPO training traces \(e\.g\. TRL, OpenRLHF, HuggingFace W&B runs\) is the primary missing validation\. \(2\)*High eval\-noise regime and noise model\.*E7 \(Table[4](https://arxiv.org/html/2606.04145#S4.T4)\) shows EvalStop degrades gracefully up toσ=0\.12\\sigma\{=\}0\.12, but does not test extreme regimes whereσ\\sigmais comparable to the peak\-to\-trough drop of the eval curve; atσ\>0\.15\\sigma\{\>\}0\.15thek=2k\{=\}2threshold is likely inadequate\. Our noise model is Gaussian and i\.i\.d\. across checkpoints; heavy\-tailed or auto\-correlated eval noise \(closer to bootstrap variance across benchmark items\) is likely to inflate FPR further\. Adaptive or confidence\-aware thresholds \(e\.g\. Bayesian change\-point detection\) would extend the operating range\. \(3\)*Evaluation cost modelling\.*We model eval as occupying GPUs for a fixed per\-model\-size duration but do not treat eval scheduling itself as a decision variable\. \(4\)*Homogeneous hardware\.*The simulator uses homogeneous GPUs without network I/O, gradient accumulation, or elastic scaling\.

## 6Related Work

ML cluster scheduling\.Tiresias\(Guet al\.,[2019](https://arxiv.org/html/2606.04145#bib.bib1)\)pioneered non\-clairvoyant scheduling\(Motwaniet al\.,[1994](https://arxiv.org/html/2606.04145#bib.bib21)\)for DL using multi\-level feedback queues\. Pollux\(Qiaoet al\.,[2021](https://arxiv.org/html/2606.04145#bib.bib3)\)co\-adapts batch sizes and resource allocation\. Gavel\(Narayananet al\.,[2020](https://arxiv.org/html/2606.04145#bib.bib4)\)and Sia\(Jayaramet al\.,[2023](https://arxiv.org/html/2606.04145#bib.bib5)\)address heterogeneous clusters\. Shockwave\(Zhenget al\.,[2023](https://arxiv.org/html/2606.04145#bib.bib6)\)handles dynamic adaptation with fair scheduling\. More recently, MAST\(Choudhuryet al\.,[2024](https://arxiv.org/html/2606.04145#bib.bib13)\)tackles geo\-distributed ML scheduling at hyperscale, and Parcae\(Duanet al\.,[2024](https://arxiv.org/html/2606.04145#bib.bib14)\)optimizes training on preemptible instances\. None use downstream eval quality as a scheduling signal\.

Quality\-aware scheduling\.SLAQ\(Zhanget al\.,[2017](https://arxiv.org/html/2606.04145#bib.bib2)\)uses training loss to predict quality improvement and allocate resources accordingly\. This works for convex\-loss ML models where loss correlates with quality, but fails for RLHF where loss is a proxy that decorrelates from quality post\-peak\. EvalStop extends the quality\-aware paradigm from proxy signals \(loss\) to world feedback \(eval\)\.

RLHF and reward hacking\.RLHF\(Ouyanget al\.,[2022](https://arxiv.org/html/2606.04145#bib.bib9); Stiennonet al\.,[2020](https://arxiv.org/html/2606.04145#bib.bib11)\)is now standard for LLM alignment, but reward overoptimization is a well\-documented failure mode\(Gaoet al\.,[2023](https://arxiv.org/html/2606.04145#bib.bib10); Skalseet al\.,[2022](https://arxiv.org/html/2606.04145#bib.bib19); Panet al\.,[2022](https://arxiv.org/html/2606.04145#bib.bib20)\)\.Rafailovet al\.\([2024](https://arxiv.org/html/2606.04145#bib.bib17)\)extended overoptimization scaling laws to DPO, andMoskovitzet al\.\([2024](https://arxiv.org/html/2606.04145#bib.bib18)\)proposed constrained RLHF\. These works focus on*preventing*reward hacking via training modifications\. EvalStop is complementary: it*detects*overoptimization via world feedback and*acts*on it at the scheduler level, without modifying the training algorithm\.

LLM fine\-tuning systems\.S\-LoRA\(Shenget al\.,[2024](https://arxiv.org/html/2606.04145#bib.bib15)\)and dLoRA\(Wuet al\.,[2024](https://arxiv.org/html/2606.04145#bib.bib16)\)address multi\-tenant LoRA serving with adapter orchestration\. MuxTune\(Xueet al\.,[2026](https://arxiv.org/html/2606.04145#bib.bib7)\)tackles multi\-task LoRA co\-scheduling via backbone multiplexing\.Konget al\.\([2025](https://arxiv.org/html/2606.04145#bib.bib8)\)study deadline\-aware scheduling for fine\-tuning with spot instances\. OpenRLHF\(Huet al\.,[2024](https://arxiv.org/html/2606.04145#bib.bib22)\)provides a scalable RLHF framework\. None address eval\-awareness or use world feedback for scheduling decisions\.

Early stopping\.Early stopping based on validation loss is a classic regularisation technique\(Prechelt,[1998](https://arxiv.org/html/2606.04145#bib.bib23)\)\. EvalStop differs in three ways: \(i\) it monitors*downstream eval quality*, which is the right signal for RLHF, where validation loss is monotonic even during reward hacking; \(ii\) it operates at the*scheduler level*, translating early\-stop decisions into cluster\-wide resource reallocation across multiple tenants; and \(iii\) it is evaluated as a*classifier*of reward hacking, against both a progress\-triggered rule that knows the synthetic peak location \(StopAt\-0\.65\) and a loss\-plateau detector, showing that the eval signal is needed to keep precision high and FPR low \(Table[1](https://arxiv.org/html/2606.04145#S4.T1)\)\.

## 7Conclusion

We introduced EvalStop, a composable scheduling primitive that uses world feedback \(downstream eval scores\) to detect and correct reward overoptimisation in RLHF training\. Our evaluation reframes scheduler\-level early stopping as a detection problem: on a workload containing both reward\-hacking and healthy RLHF runs, EvalStop achieves 98% precision and 1\.5% false\-positive rate while delivering\+\+9% JCT and−\-22% wasted compute on RLHF\-heavy mixes, and it composes with every base scheduler we tested\. Two strong simpler competitors \(a trivial fixed\-progress stop rule and a loss\-plateau detector\) either trade catastrophic FPR or half the recall for comparable systems metrics, showing that the world\-feedback signal is doing the detection work\.

As RLHF workloads grow on fine\-tuning platforms, the gap between proxy signals and world feedback will become an increasingly important consideration for resource management\. EvalStop shows that closing this gap at the scheduler level, with a simple detector, is both feasible and effective in simulation; validating on real training traces is the next step\.

## References

- A\. Choudhury, Y\. Wang, T\. Pelkonen, K\. Srinivasan, A\. Jain, S\. Lin, D\. David, S\. Soleimanifard, M\. Chen, A\. Yadav, R\. Tijoriwala, D\. Samoylov, and C\. Tang \(2024\)MAST: global scheduling of ML training across geo\-distributed datacenters at hyperscale\.In18th USENIX Symposium on Operating Systems Design and Implementation \(OSDI\),pp\. 563–580\.Cited by:[§6](https://arxiv.org/html/2606.04145#S6.p1.1)\.
- J\. Duan, Z\. Song, X\. Miao, X\. Xi, D\. Lin, H\. Xu, M\. Zhang, and Z\. Jia \(2024\)Parcae: proactive, liveput\-optimized DNN training on preemptible instances\.In21st USENIX Symposium on Networked Systems Design and Implementation \(NSDI\),pp\. 1121–1139\.Cited by:[§6](https://arxiv.org/html/2606.04145#S6.p1.1)\.
- L\. Gao, J\. Schulman, and J\. Hilton \(2023\)Scaling laws for reward model overoptimization\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 10835–10866\.Cited by:[Appendix A](https://arxiv.org/html/2606.04145#A1.p1.1),[§1](https://arxiv.org/html/2606.04145#S1.p1.1),[§1](https://arxiv.org/html/2606.04145#S1.p2.1),[§2\.3](https://arxiv.org/html/2606.04145#S2.SS3.p3.1),[Figure 5](https://arxiv.org/html/2606.04145#S4.F5),[Figure 5](https://arxiv.org/html/2606.04145#S4.F5.6.3),[§4\.6](https://arxiv.org/html/2606.04145#S4.SS6.p3.5),[§4](https://arxiv.org/html/2606.04145#S4.p1.2),[§5](https://arxiv.org/html/2606.04145#S5.p1.1),[§5](https://arxiv.org/html/2606.04145#S5.p5.5),[§6](https://arxiv.org/html/2606.04145#S6.p3.1)\.
- J\. Gu, M\. Chowdhury, K\. G\. Shin, Y\. Zhu, M\. Jeon, J\. Qian, H\. Liu, and C\. Zhuo \(2019\)Tiresias: a GPU cluster manager for distributed deep learning\.InProceedings of the 16th USENIX Symposium on Networked Systems Design and Implementation \(NSDI\),pp\. 485–500\.Cited by:[§1](https://arxiv.org/html/2606.04145#S1.p2.1),[§6](https://arxiv.org/html/2606.04145#S6.p1.1)\.
- J\. Hu, X\. Tao, W\. Peng,et al\.\(2024\)OpenRLHF: an easy\-to\-use, scalable and high\-performance RLHF framework\.arXiv preprint arXiv:2405\.11143\.Cited by:[§3](https://arxiv.org/html/2606.04145#S3.p4.1),[§6](https://arxiv.org/html/2606.04145#S6.p4.1)\.
- K\. R\. Jayaram, V\. Muthusamy, G\. Thomas, A\. Verma, and M\. Purcell \(2023\)Sia: heterogeneity\-aware, goodput\-optimized ML\-cluster scheduling\.InProceedings of the 29th ACM Symposium on Operating Systems Principles \(SOSP\),pp\. 642–657\.Cited by:[§1](https://arxiv.org/html/2606.04145#S1.p2.1),[§6](https://arxiv.org/html/2606.04145#S6.p1.1)\.
- L\. Kong, Y\. Xu, L\. Jiao, and C\. Xu \(2025\)Deadline\-aware online scheduling for LLM fine\-tuning with spot market predictions\.arXiv preprint arXiv:2512\.20967\.Cited by:[§6](https://arxiv.org/html/2606.04145#S6.p4.1)\.
- T\. Moskovitz, A\. K\. Singh, D\. Strouse, T\. Sandholm, R\. Salakhutdinov, A\. Dragan, and S\. McAleer \(2024\)Confronting reward model overoptimization with constrained RLHF\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2606.04145#S1.p2.1),[§2\.3](https://arxiv.org/html/2606.04145#S2.SS3.p3.1),[§5](https://arxiv.org/html/2606.04145#S5.p1.1),[§5](https://arxiv.org/html/2606.04145#S5.p2.1),[§6](https://arxiv.org/html/2606.04145#S6.p3.1)\.
- R\. Motwani, S\. Phillips, and E\. Torng \(1994\)Non\-clairvoyant scheduling\.InProceedings of the 4th Annual ACM\-SIAM Symposium on Discrete Algorithms \(SODA\),pp\. 422–431\.Cited by:[§1](https://arxiv.org/html/2606.04145#S1.p3.1),[§3](https://arxiv.org/html/2606.04145#S3.p3.1),[§6](https://arxiv.org/html/2606.04145#S6.p1.1)\.
- D\. Narayanan, K\. Santhanam, F\. Kazhamiaka, A\. Phanishayee, and M\. Zaharia \(2020\)Heterogeneity\-aware cluster scheduling policies for deep learning workloads\.InProceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation \(OSDI\),pp\. 481–498\.Cited by:[§1](https://arxiv.org/html/2606.04145#S1.p2.1),[§6](https://arxiv.org/html/2606.04145#S6.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. Lowe \(2022\)Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§1](https://arxiv.org/html/2606.04145#S1.p1.1),[§6](https://arxiv.org/html/2606.04145#S6.p3.1)\.
- A\. Pan, K\. Bhatia, and J\. Steinhardt \(2022\)The effects of reward misspecification: mapping and mitigating misaligned models\.InThe Tenth International Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2606.04145#S1.p1.1),[§6](https://arxiv.org/html/2606.04145#S6.p3.1)\.
- L\. Prechelt \(1998\)Early stopping — but when?\.InNeural Networks: Tricks of the Trade,G\. B\. Orr and K\. Müller \(Eds\.\),Lecture Notes in Computer Science, Vol\.1524,pp\. 55–69\.Cited by:[§1](https://arxiv.org/html/2606.04145#S1.p2.1),[§6](https://arxiv.org/html/2606.04145#S6.p5.1)\.
- A\. Qiao, S\. K\. Agrawal, S\. Gandham, A\. Carrell, A\. Phanishayee, M\. Zaharia, and I\. Stoica \(2021\)Pollux: co\-adaptive cluster scheduling for goodput\-optimized deep learning\.InProceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation \(OSDI\),pp\. 1–18\.Cited by:[§1](https://arxiv.org/html/2606.04145#S1.p2.1),[§6](https://arxiv.org/html/2606.04145#S6.p1.1)\.
- R\. Rafailov, Y\. Chittepu, R\. Park, H\. Sikchi, J\. Hejna, W\. B\. Knox, C\. Finn, and S\. Niekum \(2024\)Scaling laws for reward model overoptimization in direct alignment algorithms\.InAdvances in Neural Information Processing Systems 37 \(NeurIPS\),Cited by:[§2\.3](https://arxiv.org/html/2606.04145#S2.SS3.p3.1),[§5](https://arxiv.org/html/2606.04145#S5.p4.1),[§6](https://arxiv.org/html/2606.04145#S6.p3.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1](https://arxiv.org/html/2606.04145#S1.p1.1)\.
- Y\. Sheng, S\. Cao, D\. Li, C\. Hooper, N\. Lee, S\. Yang, C\. Chou, B\. Zhu, L\. Zheng, K\. Keutzer, J\. E\. Gonzalez, and I\. Stoica \(2024\)S\-LoRA: serving thousands of concurrent LoRA adapters\.InProceedings of Machine Learning and Systems 6 \(MLSys\),Cited by:[§6](https://arxiv.org/html/2606.04145#S6.p4.1)\.
- J\. Skalse, N\. H\. R\. Howe, D\. Krasheninnikov, and D\. Krueger \(2022\)Defining and characterizing reward hacking\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§1](https://arxiv.org/html/2606.04145#S1.p1.1),[§5](https://arxiv.org/html/2606.04145#S5.p1.1),[§6](https://arxiv.org/html/2606.04145#S6.p3.1)\.
- N\. Stiennon, L\. Ouyang, J\. Wu, D\. M\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. Christiano \(2020\)Learning to summarize from human feedback\.InAdvances in Neural Information Processing Systems,Vol\.33\.Cited by:[§1](https://arxiv.org/html/2606.04145#S1.p1.1),[§6](https://arxiv.org/html/2606.04145#S6.p3.1)\.
- B\. Wu, R\. Zhu, Z\. Zhang, P\. Sun, X\. Liu, and X\. Jin \(2024\)dLoRA: dynamically orchestrating requests and adapters for LoRA LLM serving\.In18th USENIX Symposium on Operating Systems Design and Implementation \(OSDI\),pp\. 911–927\.Cited by:[§6](https://arxiv.org/html/2606.04145#S6.p4.1)\.
- C\. Xue, Y\. Pan, W\. Cui, Q\. Chen, S\. Zhang, B\. He, and M\. Guo \(2026\)MuxTune: efficient multi\-task LLM fine\-tuning in multi\-tenant datacenters via spatial\-temporal backbone multiplexing\.arXiv preprint arXiv:2603\.02885\.Cited by:[§6](https://arxiv.org/html/2606.04145#S6.p4.1)\.
- H\. Zhang, L\. Stafman, A\. Or, and M\. J\. Freedman \(2017\)SLAQ: quality\-driven scheduling for distributed machine learning\.InProceedings of the ACM Symposium on Cloud Computing \(SoCC\),pp\. 390–404\.Cited by:[§1](https://arxiv.org/html/2606.04145#S1.p2.1),[1st item](https://arxiv.org/html/2606.04145#S2.I1.i1.p1.1),[§4](https://arxiv.org/html/2606.04145#S4.p2.2),[§6](https://arxiv.org/html/2606.04145#S6.p2.1)\.
- P\. Zheng, R\. Hu, Y\. Xu, A\. Jayarajan, A\. Li, J\. Ye, and W\. Chen \(2023\)Shockwave: fair and efficient cluster scheduling for dynamic adaptation in machine learning\.InProceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation \(NSDI\),pp\. 1–17\.Cited by:[§6](https://arxiv.org/html/2606.04145#S6.p1.1)\.

## Appendix ASimulator Configuration

Table[6](https://arxiv.org/html/2606.04145#A1.T6)lists the full simulator parameterisation\. Training curves are seeded per\-job for reproducibility\. RLHF curves come in two regimes: a*hacking*regime \(60% of RLHF jobs by default\), calibrated to the overoptimisation dynamics ofGaoet al\.\([2023](https://arxiv.org/html/2606.04145#bib.bib10)\), in which the eval peak occurs at 55–75% of training progress and then degrades; and a*healthy*regime \(40%\) in which the eval score rises monotonically with noise through progress=1\.0\{=\}1\.0\. Training loss in both regimes decays exponentially to a plateau at 50–70% progress, matching real RLHF convergence\. Ground\-truth hacking labels and peak progress are attached to each job but hidden from the scheduler via a whitelist\-based proxy \(SchedulerJobView\), so schedulers observe only the current loss, current eval score, and historical eval checkpoints\.

Table 6:Simulator hyperparameters and job\-type profiles\.
## Appendix BE1: Mixed Workload Full Results

Table[7](https://arxiv.org/html/2606.04145#A2.T7)presents the complete E1 results \(50% LoRA, 30% DPO, 20% RLHF, 32 GPUs\) omitted from the main text for space\. EvalStop\+SRTF achieves the best JCT and competitive TTFUC with no fairness penalty\.

Table 7:E1: Mixed workload \(50/30/20 LoRA/DPO/RLHF, 32 GPUs, 200 jobs, 5 seeds\)\. Precision / FPR columns only defined for stop\-enabled schedulers; “n/a” = no stops issued\.
## Appendix CE5–E6: Numerical Details

Tables[8](https://arxiv.org/html/2606.04145#A3.T8)and[9](https://arxiv.org/html/2606.04145#A3.T9)provide numerical values for the threshold and eval\-frequency sensitivity experiments plotted in Figures[3](https://arxiv.org/html/2606.04145#S4.F3)and[4](https://arxiv.org/html/2606.04145#S4.F4)\.

Table 8:E5: Threshold sensitivity \(EvalStop\+SRTF, 80% RLHF, 64 GPUs, 5 seeds\)\. Row withk=2k\{=\}2\(our default\) is highlighted\.Table 9:E6: Eval frequency sensitivity \(EvalStop\+SRTF, 80% RLHF, 64 GPUs, 5 seeds\)\.Δ\\DeltaJCT andΔ\\DeltaWasted are reported vs\. SRTF\-Est alone \(JCT=\{=\}1124, Wasted=\{=\}34\.5%\)\.