KARMA: Knowledge graph-based Automated Reasoning Materialization and Alignment

# KARMA: Knowledge graph-based Automated Reasoning Materialization and Alignment
Source: [https://arxiv.org/html/2607.03166](https://arxiv.org/html/2607.03166)
Jinkyeong Choi, Chaebin Jeong, Donghyeon Park Sejong University Seoul, South Korea \{jjinchoi, chaibin2000\}@sju\.ac\.kr, parkdh@sejong\.ac\.kr

###### Abstract

Template\-based contrastive synthesis is scalable, but its candidates often differ only in a few entity\-slots while sequence\-level optimization spreads supervision over mostly shared templates\. We formalize this as the Resolution Mismatch Problem and propose KARMA, which enumerates schema\-constrained paths over domain knowledge graphs and verbalizes them into slot\-aligned contrastive candidates\. Slot\-Parallel Alignment \(SPA\) then applies a decoupled slot\-level objective to route preference supervision to discriminative entity\-slots, with slot\-aware masked attention serving as an optional packed\-evaluation implementation\. Across biomedical, computer\-science, and chemistry benchmarks, KARMA outperforms base LLM and same\-data SFT baselines, and compares favorably with sequence\- and token\-level preference methods\.

*K*eywordsLarge Language Models \(LLMs\), Reasoning, Domain Adaptation, Preference Optimization, Knowledge Graph \(KG\)

## 1Introduction

Recent progress in LLM reasoning has increased the demand for reasoning supervision\[[35](https://arxiv.org/html/2607.03166#bib.bib8),[4](https://arxiv.org/html/2607.03166#bib.bib11),[9](https://arxiv.org/html/2607.03166#bib.bib12)\]\. However, scaling such supervision remains bottlenecked by trade\-offs among reliability, diversity, and cost\[[30](https://arxiv.org/html/2607.03166#bib.bib28),[14](https://arxiv.org/html/2607.03166#bib.bib24)\]\.

Prior approaches address this trade\-off in three ways\. \(1\) Human expert annotation yields reliable reasoning chains but is prohibitively expensive at the process level\[[30](https://arxiv.org/html/2607.03166#bib.bib28),[14](https://arxiv.org/html/2607.03166#bib.bib24),[32](https://arxiv.org/html/2607.03166#bib.bib31)\]\. \(2\) Synthetic data from teacher models improves scalability\[[33](https://arxiv.org/html/2607.03166#bib.bib15),[37](https://arxiv.org/html/2607.03166#bib.bib14)\], but inherits and propagates latent reasoning errors\[[7](https://arxiv.org/html/2607.03166#bib.bib16),[24](https://arxiv.org/html/2607.03166#bib.bib32)\]\. \(3\) Rule\- or template\-based generation reduces cost through programmatic construction, but narrow templates restrict factual coverage\[[23](https://arxiv.org/html/2607.03166#bib.bib18)\]\. Despite these differences in how reasoning data is generated, a structural limitation emerges once it is used for contrastive supervision: candidate differences are often concentrated in a few entity\-slots, while supervision is applied at the full\-sequence level\.

![Refer to caption](https://arxiv.org/html/2607.03166v1/x1.png)Figure 1:The Resolution Mismatch Problem in preference optimization\. \(a\) DPO compares a single chosen\-rejected pair at the sequence level\. \(b\) Multi\-DPO extends this to multiple rejected candidates, still at the sequence level\. \(c\) Our Slot\-Parallel Alignment \(SPA\) compares candidates sharing the same template and aligns preference at the resolution of discriminative entity\-slots\.We formalize this gap as theResolution Mismatch Problem\(Fig\.[1](https://arxiv.org/html/2607.03166#S1.F1)\): preference is often determined by a few discriminative entity\-slots, while optimization is applied to the full sequence\.The bottleneck is not how many candidates we compare, but at what resolution we compare them\.Standard DPO\[[21](https://arxiv.org/html/2607.03166#bib.bib22)\]\(Fig\.[1](https://arxiv.org/html/2607.03166#S1.F1)\-a\) and its multi\-candidate extensions \(Fig\.[1](https://arxiv.org/html/2607.03166#S1.F1)\-b\) widen the comparison set but retain sequence\-level granularity, which dilutes the signal on informative entity\-slots\[[20](https://arxiv.org/html/2607.03166#bib.bib43),[40](https://arxiv.org/html/2607.03166#bib.bib44),[41](https://arxiv.org/html/2607.03166#bib.bib46)\]\.

Knowledge graphs provide a natural way to address this mismatch: under fixed schemas, alternative paths yield candidates that share the same template, with differences localized to entity\-slots\[[28](https://arxiv.org/html/2607.03166#bib.bib38),[27](https://arxiv.org/html/2607.03166#bib.bib37),[18](https://arxiv.org/html/2607.03166#bib.bib36)\]\. We instantiate this idea inKARMA\(Knowledge graph\-basedAutomatedReasoningMaterialization andAlignment\), a unified framework whose synthesis pipeline produces such candidates and whose learning component,Slot\-Parallel Alignment \(SPA\), operates directly on them \(Fig\.[1](https://arxiv.org/html/2607.03166#S1.F1)\-c\)\.

The synthesis pipeline verbalizes KG paths under a shared template\[[31](https://arxiv.org/html/2607.03166#bib.bib39),[6](https://arxiv.org/html/2607.03166#bib.bib42),[1](https://arxiv.org/html/2607.03166#bib.bib40)\]and uses recurrence\-based support as a structural prior for preference assignment\. This exposes explicit discriminative entity\-slots while expanding factual coverage beyond hand\-written templates\.

On the learning side, SPA applies a decoupled slot\-level objective that supervises construction\-exposed entity slots while retaining the shared template as a language\-modeling signal\. For efficient packed evaluation, SPA can optionally use slot\-aware masked attention to approximate per\-candidate log\-likelihoods in a single forward pass without changing the slot\-level supervision signal\. Unlike prior token\-level methods\[[38](https://arxiv.org/html/2607.03166#bib.bib48),[3](https://arxiv.org/html/2607.03166#bib.bib50),[39](https://arxiv.org/html/2607.03166#bib.bib49),[15](https://arxiv.org/html/2607.03166#bib.bib45)\]that learn or estimate which tokens matter, SPA targets entity\-slots that are explicitly exposed by construction in KARMA\-synthesized data\.

When data and optimization share the same slot\-level resolution, preference learning can focus on discriminative evidence while preserving shared templates\. We evaluate KARMA across biomedical, chemistry, and computer\-science benchmarks, where it yields consistent gains over base LLM and SFT baselines\.

Our contributions are:

- •Aligned contrastive synthesis from KGs:We propose a KG\-grounded synthesis pipeline that constructs chosen\-rejected candidates sharing a verbalization template by design, localizing preference differences to entity\-slots and using KG path recurrence as a structural prior for preference assignment\.
- •Slot\-Parallel Alignment \(SPA\):We introduce a preference optimization method whose resolution matches the data: a decoupled slot\-level objective applies preference and anchor losses to construction\-exposed entity slots while preserving shared templates as a language\-modeling signal\. We further provide an optional slot\-aware masked attention implementation that reduces redundant candidate evaluation without estimating token importance\.

## 2Related Work

##### Trade\-offs in Reasoning Supervision

Reasoning supervision faces a cost\-coverage trade\-off: human process supervision yields reliable step\-level signals but is prohibitively expensive\[[30](https://arxiv.org/html/2607.03166#bib.bib28),[14](https://arxiv.org/html/2607.03166#bib.bib24),[32](https://arxiv.org/html/2607.03166#bib.bib31)\], teacher\-distilled data scales better but degrades under recursive use\[[33](https://arxiv.org/html/2607.03166#bib.bib15),[17](https://arxiv.org/html/2607.03166#bib.bib33),[7](https://arxiv.org/html/2607.03166#bib.bib16),[24](https://arxiv.org/html/2607.03166#bib.bib32)\], and template\-based generation lowers cost but restricts factual coverage and collapses contrastive differences onto a few entity\-slots\[[29](https://arxiv.org/html/2607.03166#bib.bib17),[23](https://arxiv.org/html/2607.03166#bib.bib18)\]\. KARMA turns this collapse into a resource: by grounding candidate construction in KG paths, it expands factual coverage while keeping discriminative variation localized to entity\-slots by construction\.

##### Fine\-grained Preference Alignment

Sequence\-level preference optimization\[[21](https://arxiv.org/html/2607.03166#bib.bib22),[2](https://arxiv.org/html/2607.03166#bib.bib23),[25](https://arxiv.org/html/2607.03166#bib.bib21),[16](https://arxiv.org/html/2607.03166#bib.bib20)\]dilutes supervision on the few discriminative positions when candidates share most tokens\[[40](https://arxiv.org/html/2607.03166#bib.bib44),[15](https://arxiv.org/html/2607.03166#bib.bib45)\]\. Token\-level methods address this by estimating which tokens matter via importance scoring, learned masks, or oracle selection\[[38](https://arxiv.org/html/2607.03166#bib.bib48),[3](https://arxiv.org/html/2607.03166#bib.bib50),[39](https://arxiv.org/html/2607.03166#bib.bib49),[42](https://arxiv.org/html/2607.03166#bib.bib47),[36](https://arxiv.org/html/2607.03166#bib.bib19)\], but require additional learning or external signals to identify them\. SPA instead exploits a structural prior from KARMA\-synthesized data: discriminative entity\-slots are known by construction, so supervision routes to them directly without estimation\.

##### Knowledge Graph Verbalization

Knowledge graphs ground language models by verbalizing structured facts into natural language\[[6](https://arxiv.org/html/2607.03166#bib.bib42),[1](https://arxiv.org/html/2607.03166#bib.bib40),[31](https://arxiv.org/html/2607.03166#bib.bib39),[5](https://arxiv.org/html/2607.03166#bib.bib41)\], with reasoning\-on\-graph approaches further leveraging KG paths as faithful intermediate traces\[[18](https://arxiv.org/html/2607.03166#bib.bib36),[27](https://arxiv.org/html/2607.03166#bib.bib37),[28](https://arxiv.org/html/2607.03166#bib.bib38)\]\. Both lines treat each verbalized path as a single positive instance, leaving the structural property that alternative paths share endpoints and schema unused\. KARMA exploits exactly this property to expose slot\-level discriminative structure for SPA\.

## 3Methodology

### 3\.1Overview

We proposeKARMA, a knowledge graph \(KG\)\-grounded contrastive synthesis pipeline, andSPA, a slot\-level preference optimization method co\-designed for KARMA\-synthesized data\. KARMA enumerates alternative KG paths between fixed source\-target entities under a fixed relational schema and verbalizes them into structurally aligned contrastive candidates that share a template scaffold while differing only at discriminative entity\-slots\. SPA then aligns preference optimization with this entity\-slot\-level structure through two core learning mechanisms: \(i\) slot\-parallel organization of candidate variants under shared context, and \(ii\) a decoupled hybrid objective that applies preference supervision to entity slots while preserving the shared template\. Slot\-aware masked attention is an optional packed\-evaluation mechanism that reduces redundant forward passes by approximating per\-candidate log\-likelihoods in a single forward pass\. Together, KARMA and SPA realize a single principle:align the granularity of supervision with the structural granularity of the data\.

![Refer to caption](https://arxiv.org/html/2607.03166v1/x2.png)Figure 2:Overview of KARMA and SPA\. KARMA synthesizes structurally aligned contrastive candidates from knowledge\-graph paths via path enumeration, support\-based top\-K selection, and template verbalization\. Because candidate differences are localized to sparse entity\-slots, sequence\-level preference optimization suffers from signal dilution\. SPA addresses this mismatch with a decoupled slot\-level objective that aligns learning with entity\-slot\-level supervision\. Slot\-aware masked attention, when enabled, serves as an optional packed\-evaluation implementation that reduces redundant candidate evaluation while leaving the slot\-level supervision signal unchanged\.
### 3\.2KARMA Data Synthesis: KG\-Grounded Contrastive Candidate Construction

KARMA constructs slot\-aligned contrastive candidates from a domain knowledge graph in three steps: it enumerates schema\-constrained paths between source\-target entity pairs, selects top\-KKalternatives per candidate pool by balancing support and structural diversity, and verbalizes the selected paths with a schema\-specific template shared within the pool\. The resulting candidates share the same source and target entities and relation schema, differing only at their intermediate entity slots\. This slot\-localized variation is the interface to SPA: it confines contrastive information to entity slots, exposing preference signal at the same granularity as the data variation\.

#### 3\.2\.1Path Enumeration

KARMA first constructs candidate pools by enumerating schema\-constrained multi\-hop paths from a domain knowledge graph \(KG\)𝒢=\(𝒩,ℰ\)\\mathcal\{G\}=\(\\mathcal\{N\},\\mathcal\{E\}\), where𝒩\\mathcal\{N\}is the set of entities andℰ⊆𝒩×ℛ×𝒩\\mathcal\{E\}\\subseteq\\mathcal\{N\}\\times\\mathcal\{R\}\\times\\mathcal\{N\}is the set of typed edges over relation typesℛ\\mathcal\{R\}\. A relation schema specifies an ordered pattern of typed edges that candidate paths must instantiate\. KARMA uses such schemas to collect alternative paths that share the same schema and source and target entities while differing only in their intermediate entities \(Fig\.[2](https://arxiv.org/html/2607.03166#S3.F2)A\)\.

For a given source entityese\_\{s\}, target entityete\_\{t\}, and relation schema, KARMA enumerates a schema\-constrained candidate pool𝒫\\mathcal\{P\}, a source\-to\-target subgraph of𝒢\\mathcal\{G\}, whose elements take the form

pm=\(es,e1\(m\),e2\(m\),…,en−1\(m\),et\),\{p^\{m\}=\\left\(e\_\{s\},e\_\{1\}^\{\(m\)\},e\_\{2\}^\{\(m\)\},\\ldots,e\_\{n\-1\}^\{\(m\)\},e\_\{t\}\\right\)\},\(1\)wherem=1,…,Mm=1,\\dots,Mindexes paths within the pool,nndenotes the number of hops in the relation schema,ej\(m\)e^\{\(m\)\}\_\{j\}denotes the entity at thejj\-th intermediate position of pathpmp^\{m\}forj=1,…,n−1j=1,\\dots,n\-1, andes,ete\_\{s\},e\_\{t\}denote the source and target entities\.

Each such pool corresponds to one source\-target pair and serves as the candidate set for one training instance\. Paths in the same pool share the same relation schema and source and target entities, but may differ in their intermediate entities\. Equivalently, each intermediate slotjjinduces a set of distinct slot entities𝒮j=\{ej,1,…,ej,dj\}\\mathcal\{S\}\_\{j\}=\\\{e\_\{j,1\},\\dots,e\_\{j,d\_\{j\}\}\\\}, from which each path selects one entity, soej\(m\)∈𝒮je\_\{j\}^\{\(m\)\}\\in\\mathcal\{S\}\_\{j\}\. This property allows KARMA to construct contrastive alternatives whose differences are localized to intermediate entity slots\.

#### 3\.2\.2Top\-KKCandidate Selection

Given a candidate pool𝒫\\mathcal\{P\}of sizeMM, written𝒫=\{p1,…,pM\}\\mathcal\{P\}=\\\{p^\{1\},\\dots,p^\{M\}\\\}, KARMA selectsKKpaths by balancing path support and structural diversity\. We measure the support of each path by the cumulative recurrence of its intermediate entities within the pool: each intermediate entity contributes its occurrence count across𝒫\\mathcal\{P\}, and the path’s support is the sum of these contributions\. Paths whose intermediate entities frequently recur thus receive higher support, reflecting structural connections that are commonly shared under the same source\-target pair\.

KARMA then greedily admits paths in descending order of support, subject to an overlap budgetδ\\deltathat limits the number of intermediate entities shared with previously admitted paths \(Fig\.[2](https://arxiv.org/html/2607.03166#S3.F2)B\)\. This budget enforces structural diversity by preventing near\-duplicate candidates\. The procedure stops whenKKpaths have been admitted; pools with fewer thanKKadmissible paths are discarded\.

We denote the admitted paths bypk⋆p^\{\\star\}\_\{k\}\(k=1,…,Kk=1,\\dots,K\), indexed in admission order, with intermediate entities written asej\(k\)e\_\{j\}^\{\(k\)\}for slotjjof thekk\-th selected path\. The ordered selected path sequence

𝒫⋆=\(p1⋆,…,pK⋆\)\\mathcal\{P\}^\{\\star\}=\(p^\{\\star\}\_\{1\},\\dots,p^\{\\star\}\_\{K\}\)\(2\)preserves this ordering through verbalization and provides the preference ordering used by SPA:p1⋆p^\{\\star\}\_\{1\}corresponds to the chosen candidate andp2:K⋆p^\{\\star\}\_\{2:K\}to the rejected candidates\. We empirically validate this chosen\-rejected partition with a GPT\-4 judge study in Appendix[E](https://arxiv.org/html/2607.03166#A5), and report end\-to\-end ablations against random path selection in Section[4\.3](https://arxiv.org/html/2607.03166#S4.SS3)\.

#### 3\.2\.3Template Verbalization

Each selected pathpk⋆∈𝒫⋆p^\{\\star\}\_\{k\}\\in\\mathcal\{P\}^\{\\star\}is verbalized into a natural\-language sequenceyky^\{k\}by populating a schema\-specific template with the entities along the path\. The template provides a fixed scaffold of textual fragments and entity slots determined by the relation schema, while the entities ofpk⋆p^\{\\star\}\_\{k\}are realized as multi\-token spans following a domain\-specific rule\. Within a candidate pool, all selected paths share the same source and target entities and the same relation schema, so they are verbalized under an identical template scaffold\. Consequently, the resulting ordered candidate sequence𝒴=\(y1,…,yK\)\\mathcal\{Y\}=\(y^\{1\},\\dots,y^\{K\}\)differs only at the intermediate entity slots, while all template fragments and endpoint entities remain shared \(Fig\.[2](https://arxiv.org/html/2607.03166#S3.F2)C\)\. This slot\-localized variation is precisely the structural property SPA exploits in the next section\.

Repeating this procedure over all retained source\-target pairs yields the training corpus

𝒟=\{\(xi,𝒴i\)\}i=1N,𝒴i=\(yi1,…,yiK\),\\mathcal\{D\}=\\\{\(x\_\{i\},\\mathcal\{Y\}\_\{i\}\)\\\}\_\{i=1\}^\{N\},\\qquad\\mathcal\{Y\}\_\{i\}=\(y\_\{i\}^\{1\},\\dots,y\_\{i\}^\{K\}\),\(3\)wherexix\_\{i\}is a prompt constructed from the source and target entities of theii\-th instance, and𝒴i\\mathcal\{Y\}\_\{i\}is the ordered verbalized candidate set inheriting the chosen/rejected ordering from the corresponding selected path sequence\.

### 3\.3Slot\-Parallel Alignment for Slot\-Level Preference Learning

SPA realigns preference optimization from sequence\-level to slot\-level resolution, matching the granularity at which KARMA candidates actually differ\. It concentrates supervision on discriminative entity slots through a decoupled hybrid objective: slot\-level preference and anchor losses supervise entity tokens, while a template LM loss preserves the shared scaffold\. Slot\-parallel packing evaluates variants under shared context, and slot\-aware masked attention optionally obtains per\-candidate log\-likelihoods in one forward pass instead ofK\+1K\{\+\}1, at a small accuracy cost\.

#### 3\.3\.1Slot\-Parallel Packing with Shared Context

For a single training instance, we drop the corpus indexiiand write𝒴=\(y1,…,yK\)\\mathcal\{Y\}=\(y^\{1\},\\dots,y^\{K\}\)\. Following the path notation in Section[3\.2\.2](https://arxiv.org/html/2607.03166#S3.SS2.SSS2), the candidates in𝒴\\mathcal\{Y\}differ only at the intermediate slotsj=1,…,n−1j=1,\\dots,n\-1, where slotjjholds the entityej\(k\)e\_\{j\}^\{\(k\)\}in candidateyky^\{k\}\. Lett0,…,tn−1t\_\{0\},\\dots,t\_\{n\-1\}be the shared template fragments, and letej\(1\)e\_\{j\}^\{\(1\)\}denote the chosen entity at slotjj, which we refer to as the*anchor*\. SPA constructs the packed sequence

𝐬=\[x;t0,e1\(1:K\),t1,…,en−1\(1:K\),tn−1\],\\mathbf\{s\}=\\bigl\[x;\\;t\_\{0\},\\;e\_\{1\}^\{\(1\{:\}K\)\},\\;t\_\{1\},\\;\\dots,\\;e\_\{n\-1\}^\{\(1\{:\}K\)\},\\;t\_\{n\-1\}\\bigr\],\(4\)whereej\(1:K\)e\_\{j\}^\{\(1\{:\}K\)\}denotes the ordered concatenation of slot\-jjentity variantsej\(1\),…,ej\(K\)e\_\{j\}^\{\(1\)\},\\dots,e\_\{j\}^\{\(K\)\}\. As illustrated in Fig\.[2](https://arxiv.org/html/2607.03166#S3.F2)E, SPA groups the variants of each intermediate slot into a local slot bundle while sharing the surrounding template fragments\. We shuffle the intra\-slot variant order during training to prevent positional shortcuts\.

![Refer to caption](https://arxiv.org/html/2607.03166v1/x3.png)Figure 3:Slot\-aware masked attention for packed candidate evaluation\. Green cells denote allowed attention, red cells denote blocked attention, and dark cells denote future tokens\. Compared with a plain causal mask, the SPA mask lets each variant attend to the shared template scaffold and its own slot state, while restricting previous slots to the anchored prefix so that each candidate’s log\-likelihood within the packed sequence approximates its standalone value underK\+1K\{\+\}1independent forward passes\.
#### 3\.3\.2Slot\-Aware Masked Attention

Since candidates in𝐬\\mathbf\{s\}share all template fragments and differ only at the intermediate entity slots,K\+1K\{\+\}1standalone forward passes would redundantly recompute identical template representations\. SPA replaces the causal attention mask with a slot\-aware mask so that all per\-candidate log\-likelihoods are obtained from a single forward pass while approximating their standalone values\.

We organize the mask by what each query token is allowed to attend to within the causal prefix\. Template tokens at fragmenttjt\_\{j\}attend only to the preceding template fragments and to the*anchor*entity of each past slot\. An entity tokenej\(k\)e\_\{j\}^\{\(k\)\}at slotjjin candidatekkattends to the preceding template fragments, to the past anchorse1\(1\),…,ej−1\(1\)e\_\{1\}^\{\(1\)\},\\dots,e\_\{j\-1\}^\{\(1\)\}, and to itself\. Consequently, each candidate’s log\-likelihood within the packed sequence is computed along its own anchored path, approximating the value it would have under an independent forward pass\.

Fig\.[3](https://arxiv.org/html/2607.03166#S3.F3)\(left\) illustrates the resulting mask for a packed sequence with two intermediate slots and three variants per slot\. With a plain causal mask \(top right\), later tokens can attend to multiple variants from previous slots, so a candidate’s log\-likelihood within the packed sequence diverges from its standalone value\. The SPA mask \(bottom right\) restricts each candidate to its own anchored path while keeping the template representation shared, so one forward pass over𝐬\\mathbf\{s\}approximatesK\+1K\{\+\}1independent passes while encoding the template scaffold only once\.

#### 3\.3\.3Decoupled Hybrid Objective

Given per\-candidate log\-likelihoods obtained from the packed sequence, applying a sequence\-level preference objective to𝐬\\mathbf\{s\}would still spread supervision over many template tokens that are identical across candidates\. Since KARMA candidates differ only at intermediate entity slots, SPA decouples the objective according to this structure: a slot\-level preference loss supervises entity choices, an anchor loss stabilizes the chosen entities, and a template language modeling loss preserves the shared scaffold \(Fig\.[2](https://arxiv.org/html/2607.03166#S3.F2)F\)\.

##### Slot\-level preference\.

For each slotjj, the anchor entityej\(1\)e\_\{j\}^\{\(1\)\}should be preferred to the rejected entitiesej\(k\)e\_\{j\}^\{\(k\)\}from candidatesk=2,…,Kk=2,\\dots,K\. Letπθ\\pi\_\{\\theta\}denote the policy model andπref\\pi\_\{\\mathrm\{ref\}\}a fixed reference model\. For an entity spanej\(k\)e\_\{j\}^\{\(k\)\}, we writeℓθ​\(ej\(k\)∣𝐬\)\\ell\_\{\\theta\}\(e\_\{j\}^\{\(k\)\}\\mid\\mathbf\{s\}\)andℓref​\(ej\(k\)∣𝐬\)\\ell\_\{\\mathrm\{ref\}\}\(e\_\{j\}^\{\(k\)\}\\mid\\mathbf\{s\}\)for its length\-normalized log\-likelihood underπθ\\pi\_\{\\theta\}andπref\\pi\_\{\\mathrm\{ref\}\}, computed under the slot\-aware mask\. We define the reference\-normalized score

rθ​\(ej\(k\)\)=ℓθ​\(ej\(k\)∣𝐬\)−ℓref​\(ej\(k\)∣𝐬\)\.r\_\{\\theta\}\(e\_\{j\}^\{\(k\)\}\)=\\ell\_\{\\theta\}\(e\_\{j\}^\{\(k\)\}\\mid\\mathbf\{s\}\)\-\\ell\_\{\\mathrm\{ref\}\}\(e\_\{j\}^\{\(k\)\}\\mid\\mathbf\{s\}\)\.\(5\)This score is the policy’s log\-likelihood gain over the reference on the entity spanej\(k\)e\_\{j\}^\{\(k\)\}\. SPA applies a squared margin loss between the anchor and each rejected entity at the same slot:

ℒpref=1\(n−1\)​\(K−1\)×∑j=1n−1∑k=2K\(rθ\(ej\(1\)\)−rθ\(ej\(k\)\)−12​β\)2,\\begin\{split\}\\mathcal\{L\}\_\{\\mathrm\{pref\}\}&=\\frac\{1\}\{\(n\-1\)\(K\-1\)\}\\\\ &\\quad\\times\\sum\_\{j=1\}^\{n\-1\}\\sum\_\{k=2\}^\{K\}\\left\(r\_\{\\theta\}\(e\_\{j\}^\{\(1\)\}\)\-r\_\{\\theta\}\(e\_\{j\}^\{\(k\)\}\)\-\\frac\{1\}\{2\\beta\}\\right\)^\{2\},\\end\{split\}\(6\)whereβ\\betacontrols the target margin\. Here,kkfollows the original top\-KKcandidate order:k=1k=1is the chosen candidate andk=2,…,Kk=2,\\dots,Kare rejected candidates\. This term provides the discriminative signal at the slots where candidates differ, rather than spreading preference supervision over the shared template\.

##### Anchor confidence\.

The preference term in Eq\.[6](https://arxiv.org/html/2607.03166#S3.E6)constrains relative scores, but it does not directly ensure that the chosen entity has high likelihood\. SPA therefore adds an anchor confidence term:

ℒanchor=−1n−1​∑j=1n−1ℓθ​\(ej\(1\)∣𝐬\)\.\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}=\-\\frac\{1\}\{n\-1\}\\sum\_\{j=1\}^\{n\-1\}\\ell\_\{\\theta\}\(e\_\{j\}^\{\(1\)\}\\mid\\mathbf\{s\}\)\.\(7\)This term stabilizes the likelihood of the selected path entities, complementing the relative comparison imposed byℒpref\\mathcal\{L\}\_\{\\mathrm\{pref\}\}\.

##### Template language modeling\.

The template fragments encode the relation scaffold shared by all candidates, but they are not discriminative for the slot\-level preference comparison\. Let𝒯\\mathcal\{T\}denote the set of shared template token positions in𝐬\\mathbf\{s\}, and letsus\_\{u\}denote theuu\-th token of𝐬\\mathbf\{s\}\. SPA preserves the scaffold with a standard causal language modeling loss:

ℒtemplate=−1\|𝒯\|​∑u∈𝒯log⁡πθ​\(su∣s<u\)\.\\mathcal\{L\}\_\{\\mathrm\{template\}\}=\-\\frac\{1\}\{\|\\mathcal\{T\}\|\}\\sum\_\{u\\in\\mathcal\{T\}\}\\log\\pi\_\{\\theta\}\(s\_\{u\}\\mid s\_\{<u\}\)\.\(8\)This term maintains the fluency and logical structure of the verbalization while leaving entity discrimination to the slot\-level losses\.

##### Full objective\.

The final SPA objective combines the slot\-level preference term with the two auxiliary losses:

ℒSPA=ℒpref\+λanchor​ℒanchor\+λtemplate​ℒtemplate,\\mathcal\{L\}\_\{\\mathrm\{SPA\}\}=\\mathcal\{L\}\_\{\\mathrm\{pref\}\}\+\\lambda\_\{\\mathrm\{anchor\}\}\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}\+\\lambda\_\{\\mathrm\{template\}\}\\mathcal\{L\}\_\{\\mathrm\{template\}\},\(9\)whereλanchor\\lambda\_\{\\mathrm\{anchor\}\}andλtemplate\\lambda\_\{\\mathrm\{template\}\}are scalar loss weights\. We averageℒSPA\\mathcal\{L\}\_\{\\mathrm\{SPA\}\}over training instances\.

DomainMetricBaselineSimple SFTOurs \(KARMA\)\(Base LLM\)\(1\) Open\-Data\(2\) KARMA\-DataBiomedicalMedQA \(Acc\)0\.3620\.4340\.2730\.541PubMedQA \(Acc\)0\.5110\.5120\.2290\.522MMLU\-Bio \(Acc\)0\.5050\.5710\.3810\.702MMLU\-Pro\-Bio \(Acc\)0\.2300\.3240\.2920\.532GPQA\-Bio \(Acc\)0\.7000\.5330\.3330\.667Computer ScienceMMLU\-CS \(Acc\)0\.6510\.6550\.4510\.701MMLU\-Pro\-CS \(Acc\)0\.1950\.2430\.1800\.256CSBench \(Acc\)0\.3450\.5060\.3720\.495ChemistryMMLU\-Chem \(Acc\)0\.5000\.4950\.3730\.630MMLU\-Pro\-Chem \(Acc\)0\.1480\.1780\.1260\.195GPQA\-Chem \(Acc\)0\.1610\.1720\.1720\.204Table 1:Main performance comparison between the baseline, simple SFT variants, and our proposed method \(KARMA\)\. Results are reported in accuracy \(Acc\)\. For KARMA, the optimal configuration is reported in Appendix\.[D](https://arxiv.org/html/2607.03166#A4)\. Thebestandsecond\-bestresults per row are highlighted\.

## 4Experiments

We evaluate KARMA across three knowledge\-intensive domains to answer three research questions: \(RQ1\) Does the co\-design of KARMA\-synthesized contrastive candidates and SPA’s slot\-level objective outperform both conventional SFT and sequence\-level preference optimization on the same data? \(RQ2\) How important is graph\-guided path selection for synthetic reasoning supervision? \(RQ3\) Does aligning the preference objective’s resolution with the entity\-slot structure of the data yield consistent gains over sequence\-level and token\-level alternatives?

### 4\.1Experimental Setup

We evaluate on three domains:Biomedical,CS, andChemistry, using 11 benchmarks that mix MMLU subsets\[[8](https://arxiv.org/html/2607.03166#bib.bib2)\], MMLU\-Pro subsets\[[34](https://arxiv.org/html/2607.03166#bib.bib3)\], and reasoning\-heavy benchmarks \(MedQA\[[11](https://arxiv.org/html/2607.03166#bib.bib4)\], PubMedQA\[[12](https://arxiv.org/html/2607.03166#bib.bib5)\], GPQA\[[22](https://arxiv.org/html/2607.03166#bib.bib6)\], CSBench\[[26](https://arxiv.org/html/2607.03166#bib.bib7)\]\)\. The base model is Qwen2\.5\-7B\-Instruct with LoRA fine\-tuning, and all evaluations use zero\-shot chain\-of\-thought prompting\. Full benchmark lists \([C\.1](https://arxiv.org/html/2607.03166#A3.SS1)\), dataset sources \([A](https://arxiv.org/html/2607.03166#A1)\), and implementation details \([B](https://arxiv.org/html/2607.03166#A2)\) are provided in Appendix\.

### 4\.2Main Results

As shown in Tab\.[1](https://arxiv.org/html/2607.03166#S3.T1), KARMA yields consistent gains over the baseline across the majority of benchmarks, with the largest improvements on reasoning\-heavy ones\. KARMA also outperforms Open\-Data SFT on the majority of benchmarks, indicating that the gains do not merely reflect additional in\-domain text but the structural properties of KARMA\-synthesized supervision\.

The contrast between KARMA\-Data SFT and KARMA further isolates the role of the training objective\. KARMA\-Data SFT uses identical KARMA\-synthesized chosen candidates but optimizes them with a flat sequence\-level SFT objective, and fails to consistently improve over the baseline\. This reflects an inherent incompatibility between KARMA’s data structure and sequence\-level supervision: chosen candidates from the same source\-target pair share almost all tokens except at the intermediate entity\-slots, so SFT on these candidates assigns equal weight to all positions and the model is dominantly supervised on the shared template tokens, which form a small set of patterns repeated across many examples\. This concentrates SFT updates on a low\-diversity distribution, causing the model to collapse toward the template phrasing while losing pretraining\-acquired knowledge in the long tail\[[19](https://arxiv.org/html/2607.03166#bib.bib51),[13](https://arxiv.org/html/2607.03166#bib.bib52)\]\. We view this as a data\-side analog of the Resolution Mismatch Problem, where the mismatch arises not between preference signal and optimization granularity, but between data structure and supervision granularity\. SPA resolves this by decoupling slot\-level preference supervision from template language modeling, converting the same data into the strongest signal in the comparison\. This confirms that the gains of KARMA arise from the co\-design of structurally aligned data and slot\-level supervision, not from data scale or content alone\. This answers RQ1: KG\-grounded contrastive synthesis improves domain reasoning beyond conventional SFT, but only when the training objective operates at the entity\-slot\-level resolution at which the candidates actually differ\.

### 4\.3Ablation Study

##### Ablation of Graph\-Guided Path Selection\.

MetricRandomKARMAΔ\\DeltaMedQA \(Acc\)0\.3810\.541\+0\.160PubMedQA \(Acc\)0\.4650\.522\+0\.057MMLU\-Bio \(Acc\)0\.5550\.702\+0\.147MMLU\-Pro\-Bio \(Acc\)0\.3380\.532\+0\.194GPQA\-Bio \(Acc\)0\.4000\.667\+0\.267Average0\.4280\.593\+0\.165Table 2:Ablation on path selection strategy in the biomedical domain\.Randomsamples paths from the KG;KARMAuses our support\-based selection\.Tab\.[1](https://arxiv.org/html/2607.03166#S3.T1)shows that the full KARMA framework outperforms baselines, but does not isolate which component drives the gain\. RQ2 asks whether KARMA’s support\-based path selection itself contributes, or whether any sampling from the same KG would suffice\. We replace support\-based selection with uniform random sampling, holding all other components fixed \(KG, source\-target pairs, schema, template, SPA objective, hyperparameters\)\. Tab\.[2](https://arxiv.org/html/2607.03166#S4.T2)shows that KARMA outperforms random sampling on every Biomedical benchmark, with an average gain of \+16\.5 points\.

The gain pattern is itself informative\. Improvements are largest on benchmarks that require chaining multiple facts, such as GPQA\-Bio \(\+26\.7\), MMLU\-Pro\-Bio \(\+19\.4\), and MedQA \(\+16\.0\), and remain substantial on knowledge\-heavy multiple\-choice reasoning \(MMLU\-Bio \+14\.7\), while PubMedQA, which often admits surface\-level factual lookup, shows only a marginal gain\. This selectivity is consistent withthe view of KG paths as a structural analogue of reasoning chains: support\-based selection acts as a simple evidence prior, where intermediate entities that recur across the pool reflect connections corroborated by multiple traversals in the graph, yielding chosen\-rejected contrasts that are informative for reasoning rather than mere surface distractors\. That such a lightweight, training\-free heuristic, relying only on entity recurrence statistics already implicit in the KG, suffices to produce these gains answers RQ2: KARMA’s advantage comes from*how*paths are selected, and the KG itself already carries much of the structure needed to make that selection effective\.

##### Ablation on Preference Objective Resolution\.

DomainBenchmarkDPO\[[21](https://arxiv.org/html/2607.03166#bib.bib22)\]MDPOTDPO\[[40](https://arxiv.org/html/2607.03166#bib.bib44)\]KARMAComputer ScienceMMLU\-CS0\.6510\.6830\.6890\.701MMLUPro\-CS0\.1970\.2490\.2560\.256CSBench0\.3560\.4460\.4750\.495ChemistryMMLU\-Chem0\.6070\.5700\.6160\.630MMLUPro\-Chem0\.1880\.1890\.1940\.195GPQA\-Chem0\.1720\.1720\.1820\.204Table 3:Ablation comparing preference optimization objectives at different resolutions\. SPA achieves the best or tied\-best performance across benchmarks\.Tab\.[3](https://arxiv.org/html/2607.03166#S4.T3)compares SPA with representative preference optimization baselines operating at different resolutions\. The goal is to isolate the effect of preference objective resolution: whether the preference signal is applied at the sequence, token, or slot level\. All methods are trained on the same KARMA\-synthesized candidates and share the same training pipeline and hyperparameters; only the preference objective differs\.

We consider three baselines\. DPO \(K=1K=1\) performs preference optimization using pairwise chosen\-rejected comparisons\. MDPO \(K=4K=4\) extends this formulation to multiple rejected candidates while still operating at the sequence level\. TDPO \(K=1K=1\) introduces token\-level KL regularization, providing a finer\-grained signal than sequence\-level objectives but without explicitly targeting the entity\-slots where KARMA \(K=4K=4\) candidates differ\. In contrast, SPA applies supervision directly at the slot level, aligning the optimization objective with the structural units of KARMA\-synthesized data\.

The results show that SPA achieves the best or tied\-best performance across all benchmarks, improving over the strongest baseline \(TDPO\) on reasoning\-heavy ones such as CSBench \(\+2\.0\) and GPQA\-Chem \(\+2\.2\)\. Sequence\-level objectives assign supervision to the entire candidate, token\-level regularization makes the signal more fine\-grained, and SPA further aligns it with the entity\-slot structure where the actual candidate differences arise\. This answers RQ3: aligning the preference objective’s resolution with the data’s entity\-slot structure yields consistent gains over both sequence\-level \(DPO, MDPO\) and token\-level \(TDPO\) alternatives, indicating that resolution alignment, beyond granularity alone, accounts for the improvement\.

##### Ablation of Slot\-Aware Masked Attention\.

DomainBenchmarkw/o Maskw/ MaskCSMMLU\-CS0\.7110\.701MMLUPro\-CS0\.2580\.256CSBench0\.4950\.495ChemistryMMLU\-Chem0\.6400\.630MMLUPro\-Chem0\.1980\.195GPQA\-Chem0\.1820\.204Efficiency \(per training step\)Step time \(s\)15\.5811\.32Peak memory \(GiB\)32\.8355\.39Reserved memory \(GiB\)61\.9659\.02Forward passes2​\(K\+1\)2\(K\{\+\}1\)2Table 4:Ablation of SPA’s slot\-aware masked attention on task performance and per\-step training efficiency\. Bold marks the w/ Mask column\.Tab\.[4](https://arxiv.org/html/2607.03166#S4.T4)characterizes the accuracy\-efficiency trade\-off of slot\-aware masked attention, which we use as the default packed\-evaluation implementation in SPA\. The default w/ Mask variant packs all candidates into one sequence and approximates their log\-likelihoods in a single forward pass, whereas w/o Mask evaluates candidates independently withK\+1K\{\+\}1forward passes per step\. Both variants share identical objectives, data, and hyperparameters; only the attention masking and packing differ\. Efficiency is measured on Qwen2\.5\-7B\-Instruct withK=4K\{=\}4rejected candidates\.

The two variants exhibit a clear trade\-off between accuracy and training cost\. On the accuracy side, disabling the mask matches or slightly improves performance on five of the six benchmarks, while the w/ Mask configuration remains within one accuracy point of the independent\-evaluation variant on those benchmarks and outperforms it on GPQA\-Chem\. On the efficiency side, the default packed evaluation under the slot\-aware mask reduces the number of forward passes per step from2​\(K\+1\)2\(K\{\+\}1\)to22, yielding a 1\.38×\\timeswall\-clock speedup\. The cost is marginally higher peak GPU memory, since the packed sequence concentrates all candidates into a single forward and materializes the slot\-aware mask, while reserved memory remains comparable\. Thus, slot\-aware masked attention is the default efficiency\-oriented implementation, not the source of SPA’s slot\-level supervision\. Users can trade wall\-clock speed for peak accuracy by enabling or disabling the mask, while the preference, anchor, and template losses remain unchanged\.

## 5Conclusion

We propose KARMA, a KG\-based pipeline that synthesizes structurally aligned contrastive candidates with explicit entity\-slot\-level differences\. By leveraging KG paths, KARMA provides broad factual coverage while preserving structural alignment across candidates\. We further introduce SPA, a slot\-level preference optimization method designed for KARMA\-style data\. SPA addresses the resolution mismatch of sequence and token\-level objectives by aligning supervision with the entity slots where candidates differ, enabling more precise preference attribution and improved reasoning performance\. Together, KARMA achieves substantial gains on most knowledge\-intensive reasoning benchmarks, while our analysis highlights the importance of KG path selection strategies\.

## 6Limitations

KARMA shares two limitations with prior KG\-grounded LLM research\[[28](https://arxiv.org/html/2607.03166#bib.bib38),[27](https://arxiv.org/html/2607.03166#bib.bib37),[18](https://arxiv.org/html/2607.03166#bib.bib36),[31](https://arxiv.org/html/2607.03166#bib.bib39)\]\. First, the synthesis pipeline relies on a domain knowledge graph with sufficient coverage and connectivity to support multi\-hop path sampling, so domains without such a KG would require automatic KG construction or hybrid retrieval\-based grounding\. Second, KARMA assumes that the target capability is expressible through an explicit schema and verbalization template, which restricts the framework to capabilities decomposable into discrete entity\-slot structures and leaves open\-ended generation outside its current scope\.

Beyond these, slot\-aware masked attention is an efficiency\-oriented implementation choice rather than the source of SPA’s supervision signal: the default masked setting reduces per\-step forward passes from2​\(K\+1\)2\(K\{\+\}1\)to22and yields a 1\.38×\\timeswall\-clock speedup, while disabling it recovers independent candidate evaluation at higher training cost\. Users can choose either setting based on whether wall\-clock time or peak accuracy is the binding constraint\. Finally, all experiments use Qwen2\.5\-7B\-Instruct with LoRA fine\-tuning, and generality across other model families and scales remains to be verified\.

## References

- \[1\]O\. Agarwal, H\. Ge, S\. Shakeri, and R\. Al\-Rfou\(2021\-06\)Knowledge graph based synthetic corpus generation for knowledge\-enhanced language model pre\-training\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,K\. Toutanova, A\. Rumshisky, L\. Zettlemoyer, D\. Hakkani\-Tur, I\. Beltagy, S\. Bethard, R\. Cotterell, T\. Chakraborty, and Y\. Zhou \(Eds\.\),Online,pp\. 3554–3565\.External Links:[Link](https://aclanthology.org/2021.naacl-main.278/),[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.278)Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p5.1),[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px3.p1.1)\.
- \[2\]J\. H\. Cho, J\. Oh, M\. Kim, and B\. Lee\(2025\-11\)Rethinking DPO: the role of rejected responses in preference misalignment\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 8159–8176\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.433/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.433),ISBN 979\-8\-89176\-335\-7Cited by:[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px2.p1.1)\.
- \[3\]F\. Christopoulou, R\. Cardenas, G\. Lampouras, H\. Bou Ammar, and J\. Wang\(2025\-11\)SparsePO: controlling preference alignment of LLMs via sparse token masks\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 25477–25503\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.1389/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1389),ISBN 979\-8\-89176\-335\-7Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p6.1),[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px2.p1.1)\.
- \[4\]DeepSeek\-AI\(2025\)DeepSeek\-R1: incentivizing reasoning capability in LLMs via reinforcement learning\.Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p1.1)\.
- \[5\]S\. Dernbach, K\. Agarwal, A\. Zuniga, M\. Henry, and S\. Choudhury\(2024\)GLaM: fine\-tuning large language models for domain knowledge graph alignment via neighborhood partitioning and generative subgraph encoding\.InProceedings of the AAAI Symposium Series,Cited by:[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px3.p1.1)\.
- \[6\]C\. Gardent, A\. Shimorina, S\. Narayan, and L\. Perez\-Beltrachini\(2017\-09\)The WebNLG challenge: generating text from RDF data\.InProceedings of the 10th International Conference on Natural Language Generation,J\. M\. Alonso, A\. Bugarín, and E\. Reiter \(Eds\.\),Santiago de Compostela, Spain,pp\. 124–133\.External Links:[Link](https://aclanthology.org/W17-3518/),[Document](https://dx.doi.org/10.18653/v1/W17-3518)Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p5.1),[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px3.p1.1)\.
- \[7\]A\. Gudibande, E\. Wallace, C\. Snell, X\. Geng, H\. Liu, P\. Abbeel, S\. Levine, and D\. Song\(2023\)The false promise of imitating proprietary llms\.External Links:2305\.15717,[Link](https://arxiv.org/abs/2305.15717)Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p2.1),[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px1.p1.1)\.
- \[8\]D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt\(2021\)Measuring massive multitask language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[§4\.1](https://arxiv.org/html/2607.03166#S4.SS1.p1.1)\.
- \[9\]A\. Jaech, A\. Kalai, A\. Lerer, A\. Richardson, A\. El\-Kishky, A\. Low, A\. Helyar, A\. Madry, A\. Beutel, A\. Carney,et al\.\(2024\)Openai o1 system card\.Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p1.1)\.
- \[10\]B\. Jin, C\. Xie, J\. Zhang, K\. K\. Roy, Y\. Zhang, Z\. Li, R\. Li, X\. Tang, S\. Wang, Y\. Meng, and J\. Han\(2024\-08\)Graph chain\-of\-thought: augmenting large language models by reasoning on graphs\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 163–184\.External Links:[Link](https://aclanthology.org/2024.findings-acl.11/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.11)Cited by:[§A\.1](https://arxiv.org/html/2607.03166#A1.SS1.p1.1)\.
- \[11\]D\. Jin, E\. Pan, N\. Oufattole, W\. Weng, H\. Fang, and P\. Szolovits\(2021\)What disease does this patient have? a large\-scale open domain question answering dataset from medical exams\.Applied SciencesarXiv preprint arXiv:2501\.12948arXiv preprint arXiv:2412\.16720arXiv preprint arXiv:2305\.10403arXiv preprint arXiv:2408\.03314arXiv preprint arXiv:2211\.14275arXiv preprint arXiv:2406\.06592arXiv preprint arXiv:2410\.08146NaturearXiv preprint arXiv:2308\.09583IEEE Transactions on Knowledge and Data EngineeringIEEE Transactions on Knowledge and Data EngineeringarXiv preprint arXiv:2404\.12358IEEE Transactions on Audio, Speech and Language Processing11\(14\)\.External Links:[Link](https://www.mdpi.com/2076-3417/11/14/6421),ISSN 2076\-3417Cited by:[§4\.1](https://arxiv.org/html/2607.03166#S4.SS1.p1.1)\.
- \[12\]Q\. Jin, B\. Dhingra, Z\. Liu, W\. Cohen, and X\. Lu\(2019\-11\)PubMedQA: a dataset for biomedical research question answering\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 2567–2577\.External Links:[Link](https://aclanthology.org/D19-1259/),[Document](https://dx.doi.org/10.18653/v1/D19-1259)Cited by:[§4\.1](https://arxiv.org/html/2607.03166#S4.SS1.p1.1)\.
- \[13\]P\. Kung and N\. Peng\(2023\-07\)Do models really learn to follow instructions? an empirical study of instruction tuning\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 1317–1328\.External Links:[Link](https://aclanthology.org/2023.acl-short.113/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-short.113)Cited by:[§4\.2](https://arxiv.org/html/2607.03166#S4.SS2.p2.1)\.
- \[14\]H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe\(2023\)Let’s verify step by step\.InThe twelfth international conference on learning representations,Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p1.1),[§1](https://arxiv.org/html/2607.03166#S1.p2.1),[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px1.p1.1)\.
- \[15\]A\. Liu, H\. Bai, Z\. Lu, Y\. Sun, X\. Kong, X\. S\. Wang, J\. Shan, A\. M\. Jose, X\. Liu, L\. Wen, P\. S\. Yu, and M\. Cao\(2025\)TIS\-DPO: token\-level importance sampling for direct preference optimization with estimated weights\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p6.1),[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px2.p1.1)\.
- \[16\]T\. Liu, Z\. Qin, J\. Wu, J\. Shen, M\. Khalman, R\. Joshi, Y\. Zhao, M\. Saleh, S\. Baumgartner, J\. Liu, P\. J\. Liu, and X\. Wang\(2025\-04\)LiPO: listwise preference optimization through learning\-to\-rank\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 2404–2420\.External Links:[Link](https://aclanthology.org/2025.naacl-long.121/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.121),ISBN 979\-8\-89176\-189\-6Cited by:[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px2.p1.1)\.
- \[17\]H\. Luo, Q\. Sun, C\. Xu, P\. Zhao, J\. Lou, C\. Tao, X\. Geng, Q\. Lin, S\. Chen, and D\. Zhang\(2023\)WizardMath: empowering mathematical reasoning for large language models via reinforced evol\-instruct\.Cited by:[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px1.p1.1)\.
- \[18\]L\. Luo, Y\. Li, G\. Haffari, and S\. Pan\(2024\)Reasoning on graphs: faithful and interpretable large language model reasoning\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p4.1),[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2607.03166#S6.p1.1)\.
- \[19\]Y\. Luo, Z\. Yang, F\. Meng, Y\. Li, J\. Zhou, and Y\. Zhang\(2025\)An empirical study of catastrophic forgetting in large language models during continual fine\-tuning\.Cited by:[§4\.2](https://arxiv.org/html/2607.03166#S4.SS2.p2.1)\.
- \[20\]R\. Rafailov, J\. Hejna, R\. Park, and C\. Finn\(2024\)FromrrtoQ∗Q^\{\*\}: your language model is secretly a Q\-function\.Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p3.1)\.
- \[21\]R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn\(2023\)Direct preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p3.1),[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px2.p1.1),[Table 3](https://arxiv.org/html/2607.03166#S4.T3.1.1.3.2.1.2.1)\.
- \[22\]D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman\(2024\)GPQA: a graduate\-level google\-proof q&a benchmark\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=Ti67584b98)Cited by:[§4\.1](https://arxiv.org/html/2607.03166#S4.SS1.p1.1)\.
- \[23\]H\. Riaz, S\. S\. Bhabesh, V\. Arannil, M\. Ballesteros, and G\. Horwood\(2025\-07\)MetaSynth: meta\-prompting\-driven agentic scaffolds for diverse synthetic data generation\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 18770–18803\.External Links:[Link](https://aclanthology.org/2025.findings-acl.962/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.962),ISBN 979\-8\-89176\-256\-5Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p2.1),[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px1.p1.1)\.
- \[24\]I\. Shumailov, Z\. Shumaylov, Y\. Zhao, N\. Papernot, R\. Anderson, and Y\. Gal\(2024\)AI models collapse when trained on recursively generated data\.631\(8022\),pp\. 755–759\.Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p2.1),[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px1.p1.1)\.
- \[25\]F\. Song, B\. Yu, M\. Li, H\. Yu, F\. Huang, Y\. Li, and H\. Wang\(2024\)Preference ranking optimization for human alignment\.InProceedings of the Thirty\-Eighth AAAI Conference on Artificial Intelligence and Thirty\-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence,AAAI’24/IAAI’24/EAAI’24\.External Links:ISBN 978\-1\-57735\-887\-9,[Link](https://doi.org/10.1609/aaai.v38i17.29865),[Document](https://dx.doi.org/10.1609/aaai.v38i17.29865)Cited by:[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px2.p1.1)\.
- \[26\]X\. Song, M\. Diao, G\. Dong, Z\. Wang, Y\. Fu, R\. Qiao, Z\. Wang, D\. Fu, H\. Wu, B\. Liang, W\. Zeng, Y\. Wang, Z\. GongQue, J\. Yu, Q\. Tan, and W\. Xu\(2025\)CS\-bench: a comprehensive benchmark for large language models towards computer science mastery\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=fjEZ2LPceZ)Cited by:[§4\.1](https://arxiv.org/html/2607.03166#S4.SS1.p1.1)\.
- \[27\]J\. Sun, C\. Xu, L\. Tang, S\. Wang, C\. Lin, Y\. Gong, L\. M\. Ni, H\. Shum, and J\. Guo\(2024\)Think\-on\-graph: deep and responsible reasoning of large language model on knowledge graph\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p4.1),[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2607.03166#S6.p1.1)\.
- \[28\]X\. Tan, X\. Wang, Q\. Liu, X\. Xu, X\. Yuan, and W\. Zhang\(2025\)Paths\-over\-graph: knowledge graph empowered large language model reasoning\.InProceedings of the ACM on Web Conference 2025,WWW ’25,New York, NY, USA,pp\. 3505–3522\.External Links:ISBN 9798400712746,[Link](https://doi.org/10.1145/3696410.3714892),[Document](https://dx.doi.org/10.1145/3696410.3714892)Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p4.1),[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2607.03166#S6.p1.1)\.
- \[29\]M\. N\. Uddin, E\. R\. George, E\. Blanco, and S\. Corman\(2024\-06\)Generating uncontextualized and contextualized questions for document\-level event argument extraction\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 5612–5627\.External Links:[Link](https://aclanthology.org/2024.naacl-long.312/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.312)Cited by:[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px1.p1.1)\.
- \[30\]J\. Uesato, N\. Kushman, R\. Kumar, F\. Song, N\. Siegel, L\. Wang, A\. Creswell, G\. Irving, and I\. Higgins\(2022\)Solving math word problems with process\- and outcome\-based feedback\.Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p1.1),[§1](https://arxiv.org/html/2607.03166#S1.p2.1),[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px1.p1.1)\.
- \[31\]J\. Wang, M\. Chen, B\. Hu, D\. Yang, Z\. Liu, Y\. Shen, P\. Wei, Z\. Zhang, J\. Gu, J\. Zhou, J\. Z\. Pan, W\. Zhang, and H\. Chen\(2024\-11\)Learning to plan for retrieval\-augmented large language models from knowledge graphs\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 7813–7835\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.459/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.459)Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p5.1),[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2607.03166#S6.p1.1)\.
- \[32\]P\. Wang, L\. Li, Z\. Shao, R\. Xu, D\. Dai, Y\. Li, D\. Chen, Y\. Wu, and Z\. Sui\(2024\)Math\-shepherd: verify and reinforce LLMs step\-by\-step without human annotations\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9426–9439\.Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p2.1),[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px1.p1.1)\.
- \[33\]Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. Hajishirzi\(2023\-07\)Self\-instruct: aligning language models with self\-generated instructions\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 13484–13508\.External Links:[Link](https://aclanthology.org/2023.acl-long.754/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.754)Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p2.1),[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px1.p1.1)\.
- \[34\]Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. Chen\(2024\)MMLU\-pro: a more robust and challenging multi\-task language understanding benchmark\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 95266–95290\.External Links:[Document](https://dx.doi.org/10.52202/079017-3018),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/ad236edc564f3e3156e1b2feafb99a24-Paper-Datasets_and_Benchmarks_Track.pdf)Cited by:[§4\.1](https://arxiv.org/html/2607.03166#S4.SS1.p1.1)\.
- \[35\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, brian ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou\(2022\)Chain of thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,A\. H\. Oh, A\. Agarwal, D\. Belgrave, and K\. Cho \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=_VjQlMeSB_J)Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p1.1)\.
- \[36\]J\. Wu, X\. Li, R\. Wang, Y\. Xia, Y\. Xiong, J\. Wang, T\. Yu, X\. Chen, B\. Kveton, L\. Yao, J\. Shang, and J\. McAuley\(2025\)OCEAN: offline chain\-of\-thought evaluation and alignment in large language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=rlgplAuN2p)Cited by:[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px2.p1.1)\.
- \[37\]C\. Xu, Q\. Sun, K\. Zheng, X\. Geng, P\. Zhao, J\. Feng, C\. Tao, Q\. Lin, and D\. Jiang\(2024\)WizardLM: empowering large pre\-trained language models to follow complex instructions\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=CfXh93NDgH)Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p2.1)\.
- \[38\]K\. Yang, Z\. Liu, Q\. Xie, J\. Huang, E\. Min, and S\. Ananiadou\(2025\-11\)Selective preference optimization via token\-level reward function estimation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 7032–7056\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.359/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.359),ISBN 979\-8\-89176\-332\-6Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p6.1),[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px2.p1.1)\.
- \[39\]N\. Yang, H\. Lin, Y\. Liu, B\. Tian, G\. Liu, and H\. Zhang\(2026\)Token\-importance guided direct preference optimization\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=cMEnMVvMw9)Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p6.1),[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px2.p1.1)\.
- \[40\]Y\. Zeng, G\. Liu, W\. Ma, N\. Yang, H\. Zhang, and J\. Wang\(2024\)Token\-level direct preference optimization\.InProceedings of the 41st International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p3.1),[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px2.p1.1),[Table 3](https://arxiv.org/html/2607.03166#S4.T3.1.1.5.2.1.2.1)\.
- \[41\]W\. Zhou, S\. Zhang, L\. Zhao, and T\. Meng\(2025\)T\-REG: preference optimization with token\-level reward regularization\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Cited by:[§1](https://arxiv.org/html/2607.03166#S1.p3.1)\.
- \[42\]M\. Zhu, X\. Chen, Z\. Wang, B\. Yu, H\. Zhao, and J\. Jia\(2025\)TGDPO: harnessing token\-level reward guidance for enhancing direct preference optimization\.InProceedings of the 42nd International Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2607.03166#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix

## Appendix ADataset Construction Details

### A\.1Knowledge Graph Sources

KARMA constructs training data from domain\-specific knowledge graphs from GRBench\[[10](https://arxiv.org/html/2607.03166#bib.bib1)\], summarized in Table[5](https://arxiv.org/html/2607.03166#A1.T5)\. Each graph is represented as a typed node\-edge structure, where nodes contain textual attributes and outgoing typed neighbors\. The biomedical graph uses heterogeneous node types such as diseases, symptoms, compounds, genes, anatomy, side effects, pharmacologic classes, biological processes, molecular functions, cellular components, and pathways\. The academic graph uses paper nodes with citation edges, where each paper contains title, abstract, and keyword fields\.

DomainGraph nodesEdge typesNode attributesBiomedicalTyped biomedical entitiesTyped biomedical relationsNameComputer SciencePapersreference,cited\_byTitle, abstract, keywordsChemistryPapersreference,cited\_byTitle, abstract, keywords

Table 5:Summary of domain knowledge graphs used for KARMA synthesis\.
### A\.2Relation Schemas

A relation schema defines the edge\-type pattern used to enumerate paths from a source entity to a target entity\. In the academic graph, schemas are metapaths over citation edges\. We use four\-hop schemas by default, generated from all combinations ofreferenceandcited\_by\. In the biomedical graph, schemas are typed metapaths over biomedical node and relation types\.

### A\.3Path Enumeration and Filtering

For each schema, KARMA enumerates paths that follow the prescribed edge\-type sequence, as described in Algorithm[1](https://arxiv.org/html/2607.03166#alg1)\. Each path connects a source entityese\_\{s\}to a target entityete\_\{t\}through a sequence of intermediate entities\. For a fixed source\-target pair, all paths satisfying the same schema form a candidate pool\. Candidate pools with fewer thanKKvalid paths are discarded\. We remove loops in intermediate positions and apply intermediate caps during traversal to avoid path explosion\.

Algorithm 1Schema\-Constrained Path Pool Construction1:Knowledge graph

GG, schema set

𝒮\\mathcal\{S\}, candidate number

KK
2:Candidate pools

𝒞\\mathcal\{C\}
3:

𝒞←∅\\mathcal\{C\}\\leftarrow\\emptyset
4:foreach schema

s∈𝒮s\\in\\mathcal\{S\}do

5:foreach source entity

ese\_\{s\}do

6:Traverse

GGaccording to schema

ss
7:Collect reachable target entities

ete\_\{t\}
8:foreach target entity

ete\_\{t\}do

9:Enumerate all middle paths

ℳ​\(es,et\)\\mathcal\{M\}\(e\_\{s\},e\_\{t\}\)
10:if

\|ℳ​\(es,et\)\|≥K\|\\mathcal\{M\}\(e\_\{s\},e\_\{t\}\)\|\\geq Kthen

11:Add

\(es,et,s,ℳ​\(es,et\)\)\(e\_\{s\},e\_\{t\},s,\\mathcal\{M\}\(e\_\{s\},e\_\{t\}\)\)to

𝒞\\mathcal\{C\}
12:endif

13:endfor

14:endfor

15:endfor

16:return

𝒞\\mathcal\{C\}

### A\.4Support\-based Top\-K Selection

For each source\-target candidate pool, we selectKKpaths using an entity\-support score and an overlap constraint, as detailed in Algorithm[2](https://arxiv.org/html/2607.03166#alg2)\. The support score of a middle pathmmis the sum of the occurrence counts of its intermediate entities within the same source\-target pool:

score​\(m\)=∑e∈mfreq​\(e\)\.\\mathrm\{score\}\(m\)=\\sum\_\{e\\in m\}\\mathrm\{freq\}\(e\)\.\(10\)We then greedily select high\-score paths while allowing at mostδ\\deltaoverlapping intermediate entities with the already selected paths\. In the academic\-domain generator, we useK=4K=4andδ=1\\delta=1\.

Algorithm 2Support\-based Top\-KKPath Selection1:Middle paths

ℳ​\(es,et\)\\mathcal\{M\}\(e\_\{s\},e\_\{t\}\), candidate number

KK, overlap budget

δ\\delta
2:Selected paths

ℳ⋆\\mathcal\{M\}^\{\\star\}
3:if

\|ℳ​\(es,et\)\|<K\|\\mathcal\{M\}\(e\_\{s\},e\_\{t\}\)\|<Kthen

4:return

∅\\emptyset
5:endif

6:Count entity frequencies

c​\(e\)c\(e\)over all intermediate entities in

ℳ​\(es,et\)\\mathcal\{M\}\(e\_\{s\},e\_\{t\}\)
7:Score each path by

s​\(m\)=∑e∈mc​\(e\)s\(m\)=\\sum\_\{e\\in m\}c\(e\)
8:Sort paths in descending order of

s​\(m\)s\(m\)
9:

ℳ⋆←∅,U←∅\\mathcal\{M\}^\{\\star\}\\leftarrow\\emptyset,\\quad U\\leftarrow\\emptyset
10:foreach path

mmin sorted orderdo

11:

Em←\{e:e∈m\}E\_\{m\}\\leftarrow\\\{e:e\\in m\\\}
12:if

\|Em∩U\|≤δ\|E\_\{m\}\\cap U\|\\leq\\deltathen

13:Add

mmto

ℳ⋆\\mathcal\{M\}^\{\\star\}
14:

U←U∪EmU\\leftarrow U\\cup E\_\{m\}
15:endif

16:if

\|ℳ⋆\|=K\|\\mathcal\{M\}^\{\\star\}\|=Kthen

17:break

18:endif

19:endfor

20:if

\|ℳ⋆\|<K\|\\mathcal\{M\}^\{\\star\}\|<Kthen

21:return

∅\\emptyset
22:endif

23:return

ℳ⋆\\mathcal\{M\}^\{\\star\}

### A\.5Template Verbalization

Each selected path is verbalized into a natural\-language candidate using a schema\-specific template\. Entity mentions are marked with bold delimiters, e\.g\.,\*\*entity\*\*\. These marked spans define the entity slots used by SPA\. The selected paths for the same source\-target pair share the same template scaffold and differ only in their intermediate entity slots\.

### A\.6Dataset Statistics

Table[6](https://arxiv.org/html/2607.03166#A1.T6)reports dataset statistics\. For each domain, we report the number of nodes and edges in the source knowledge graph, the number of relation schemas used for path enumeration, the number of generated training samples, and the number of candidates per pool sizeKK\.

Domain\#Nodes\#Edges\#Schemas\#SamplesKKBiomedical47K4\.23M5221M4Computer Science5\.26M65M161M4Chemistry1\.85M41M161M4Table 6:Dataset statistics for KARMA\-synthesized training data\.

## Appendix BTraining Details

### B\.1Model and LoRA Configuration

All experiments use Qwen2\.5\-7B\-Instruct as the base model\. We fine\-tune the model with LoRA adapters\. The LoRA rank is 32 and the LoRA alpha is 32\. Adapters are inserted into the attention and feed\-forward projection modules\. The full configuration is reported in Table[7](https://arxiv.org/html/2607.03166#A2.T7)\. The reference modelπref\\pi\_\{\\mathrm\{ref\}\}used by SPA and all DPO\-family baselines is obtained by disabling the LoRA adapter on the policy modelπθ\\pi\_\{\\theta\}; that is,πθ\\pi\_\{\\theta\}andπref\\pi\_\{\\mathrm\{ref\}\}share identical base parameters and differ only in whether the trained adapters are active\.

ConfigurationValueBase modelQwen2\.5\-7B\-InstructFine\-tuning methodLoRALoRA rank32LoRA alpha32Target modulesq\_proj,k\_proj,v\_proj,o\_projgate\_proj,up\_proj,down\_projPrecisionbfloat16Max sequence length4096Gradient checkpointingEnabledAttention implementationSDPAReference modelπref\\pi\_\{\\mathrm\{ref\}\}Policy model with LoRA adapter disabledTable 7:Model and LoRA configuration\.HyperparameterValueLearning rate5×10−55\\times 10^\{\-5\}Epochs3Per\-device batch size8Gradient accumulation1Max gradient norm1\.0λanchor\\lambda\_\{\\mathrm\{anchor\}\}0\.5λtemplate\\lambda\_\{\\mathrm\{template\}\}0\.1Entity margin parameterβ\\beta0\.1Variant shufflingEnabledTable 8:SPA training hyperparameters\. The preference\-loss weightλpref\\lambda\_\{\\mathrm\{pref\}\}is fixed to1\.01\.0in all experiments based on the sensitivity analysis in Table[14](https://arxiv.org/html/2607.03166#A4.T14), and is therefore absorbed intoℒpref\\mathcal\{L\}\_\{\\mathrm\{pref\}\}in Eq\.[9](https://arxiv.org/html/2607.03166#S3.E9)\.
### B\.2Training Protocol

All training runs share the same general\-purpose hyperparameters \(learning rate, optimizer, batch size, epochs, gradient clipping, LoRA configuration\) across baselines and KARMA, so that performance differences reflect the training objective rather than tuning\. Method\-specific hyperparameters, such asλanchor\\lambda\_\{\\mathrm\{anchor\}\},λtemplate\\lambda\_\{\\mathrm\{template\}\}, andβ\\betafor SPA, are set as in Table[8](https://arxiv.org/html/2607.03166#A2.T8)for KARMA, and the corresponding original defaults are used for DPO, MDPO, and TDPO\. We do not perform validation\-based checkpoint selection; the final epoch checkpoint is used for evaluation\. Each domain dataset is used for training only, and evaluation is performed on the held\-out benchmarks listed in Table[11](https://arxiv.org/html/2607.03166#A3.T11)\.

All results reported in this paper, including the main results in Table[1](https://arxiv.org/html/2607.03166#S3.T1)and all ablations, are computed as the mean of 5 independent runs with different random seeds\.

### B\.3Compute Infrastructure

All experiments are run on NVIDIA A100 GPU\. A single SPA training run for one domain \(1M samples, 3 epochs\) takes approximately40GPU\-hours on1GPU\. The total compute budget across 5 random seeds, three domains, all baseline methods, and ablation studies is on the order of a few thousand A100 GPU\-hours\.

### B\.4Baseline Implementations

We compare KARMA with the base model, open\-data SFT, KARMA\-data SFT, and preference\-optimization baselines, as listed in Table[9](https://arxiv.org/html/2607.03166#A2.T9)\. All fine\-tuned baselines share the same base model, LoRA configuration, learning rate, batch size, epochs, and optimizer as KARMA; they differ only in the training objective and, where applicable, in the structure of the input data\. Each baseline retains its original method\-specific hyperparameters \(e\.g\., the KL coefficient in DPO, the token\-level regularization weight in TDPO\) at the values recommended in their respective papers\.

BaselineDescriptionBase LLMQwen2\.5\-7B\-Instruct without task\-specific fine\-tuningOpen\-Data SFTSFT on open\-domain or domain instruction dataKARMA\-Data SFTSFT on KARMA\-generated chosen responsesDPOPairwise sequence\-level preference optimizationMDPOMulti\-rejected sequence\-level preference optimizationTDPOToken\-level preference optimization baselineKARMA / SPASlot\-level preference optimization on KARMA candidatesTable 9:Baseline implementation summary\. All baselines share the same base model, LoRA configuration, and general\-purpose training hyperparameters\.
### B\.5Slot\-aware Mask Implementation

SPA packs candidate variants into a single sequence by replacing each entity slot with a local bundle of chosen and rejected variants\. The attention mask starts from a causal mask and then blocks cross\-candidate contamination according to Table[10](https://arxiv.org/html/2607.03166#A2.T10)\. PAD separators inserted between variants are blocked as both queries and keys\.

Query tokenAllowed keysBlocked keysTemplate tokenPrevious templates \+ previous anchorsRejected variantsAnchor entity tokenPrevious templates \+ previous anchors \+ itselfOther variants in same slotRejected entity tokenPrevious templates \+ previous anchors \+ itselfOther variants in same slotPAD separatorNoneAll tokensFuture tokenNoneCausal future positionsTable 10:Slot\-aware attention mask rules used in SPA\.
### B\.6SPA Loss Computation

The full SPA loss computation procedure is summarized in Algorithm[3](https://arxiv.org/html/2607.03166#alg3)\. Given a packed sequence with slot\-aware masking, the policy and reference models compute log\-probabilities for each variant\. Slot\-level preference and anchor confidence losses are accumulated across slots, and a template language modeling loss is computed on non\-entity tokens\.

Algorithm 3SPA Loss Computation1:Packed sequence

ss, slot variants

\{ej\(k\)\}\\\{e\_\{j\}^\{\(k\)\}\\\}, policy model

πθ\\pi\_\{\\theta\}, reference model

πref\\pi\_\{\\mathrm\{ref\}\}
2:SPA loss

ℒSPA\\mathcal\{L\}\_\{\\mathrm\{SPA\}\}
3:Compute policy log\-probabilities under the slot\-aware mask

4:Compute reference log\-probabilities with the frozen reference model

5:foreach slot

jjdo

6:Compute reference\-normalized score

rθ​\(ej\(k\)\)r\_\{\\theta\}\(e\_\{j\}^\{\(k\)\}\)for each variant

7:Compare anchor

ej\(1\)e\_\{j\}^\{\(1\)\}against rejected variants

ej\(k\),k\>1e\_\{j\}^\{\(k\)\},k\>1
8:Accumulate slot\-level preference loss

ℒpref\\mathcal\{L\}\_\{\\mathrm\{pref\}\}
9:Accumulate anchor confidence loss

ℒanchor\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}
10:endfor

11:Compute template language modeling loss

ℒtemplate\\mathcal\{L\}\_\{\\mathrm\{template\}\}on non\-entity tokens

12:

ℒSPA←ℒpref\+λanchor​ℒanchor\+λtemplate​ℒtemplate\\mathcal\{L\}\_\{\\mathrm\{SPA\}\}\\leftarrow\\mathcal\{L\}\_\{\\mathrm\{pref\}\}\+\\lambda\_\{\\mathrm\{anchor\}\}\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}\+\\lambda\_\{\\mathrm\{template\}\}\\mathcal\{L\}\_\{\\mathrm\{template\}\}
13:return

ℒSPA\\mathcal\{L\}\_\{\\mathrm\{SPA\}\}

## Appendix CEvaluation Details

### C\.1Benchmark Sources

We evaluate on biomedical, chemistry, and computer\-science benchmarks\. The benchmark groups are summarized in Table[11](https://arxiv.org/html/2607.03166#A3.T11)\.

DomainBenchmarkSubset / categoryBiomedicalMMLU\-BioAnatomy, biology, medicine, genetics, nutrition, virologyMMLU\-Pro\-BioBiologyMedQAFull benchmarkPubMedQAFull benchmarkGPQA\-BioBiologyChemistryMMLU\-ChemCollege chemistry, high school chemistryMMLU\-Pro\-ChemChemistryGPQA\-ChemChemistryComputer ScienceMMLU\-CSCS, computer security, high school CS, machine learningMMLU\-Pro\-CSComputer ScienceCSBenchReasoning task typeTable 11:Benchmark groups used for domain evaluation\.
### C\.2Prompt Templates

All benchmarks are evaluated using the same model wrapper and benchmark\-specific prompt templates, summarized by category in Table[12](https://arxiv.org/html/2607.03166#A3.T12)\.

Benchmark typePrompt formatMultiple\-choice QAQuestion \+ answer choices \+ instruction to select one optionYes/no/maybe QAQuestion \+ instruction to output the normalized labelReasoning QAQuestion \+ optional chain\-of\-thought style instructionTable 12:Prompt template categories used in evaluation\.
### C\.3Answer Extraction

Generated responses are normalized to benchmark\-specific answer formats before scoring\. For multiple\-choice tasks, the predicted option is extracted from the generated text and compared against the gold option\. For label\-based QA tasks such as yes/no/maybe classification, the generated text is mapped to the corresponding normalized label\.

### C\.4Inference Configuration

The inference configuration used across all benchmarks is reported in Table[13](https://arxiv.org/html/2607.03166#A3.T13)\.

ConfigurationValueBase modelQwen2\.5\-7B\-InstructAdapter handlingLoRA adapter loading with optional mergeBatch size8 by defaultMax new tokens512Number of shots0 by defaultDistributed evaluationSupported withtorchrunOutput formatJSON result fileTable 13:Inference configuration used for evaluation\.

## Appendix DHyperparameter Sensitivity

Table[14](https://arxiv.org/html/2607.03166#A4.T14)reports sensitivity to the loss weightsλpref\\lambda\_\{\\mathrm\{pref\}\},λanchor\\lambda\_\{\\mathrm\{anchor\}\}, andλtemplate\\lambda\_\{\\mathrm\{template\}\}on the Biomedical domain\. The analysis examines whether SPA is robust to the relative weighting of slot\-level preference learning, anchor confidence, and template language modeling\. Based on this analysis, we fixλpref=1\.0\\lambda\_\{\\mathrm\{pref\}\}=1\.0in all main experiments and absorb it intoℒpref\\mathcal\{L\}\_\{\\mathrm\{pref\}\}in the final objective \(Eq\.[9](https://arxiv.org/html/2607.03166#S3.E9)\), tuning onlyλanchor\\lambda\_\{\\mathrm\{anchor\}\}andλtemplate\\lambda\_\{\\mathrm\{template\}\}\. The selected configuration is\(λpref,λanchor,λtemplate\)=\(1\.0,0\.5,0\.1\)\(\\lambda\_\{\\mathrm\{pref\}\},\\lambda\_\{\\mathrm\{anchor\}\},\\lambda\_\{\\mathrm\{template\}\}\)=\(1\.0,0\.5,0\.1\), which achieves the best average accuracy across MedQA, PubMedQA, and MMLU\-Pro\-Bio\.

λpref\\lambda\_\{\\mathrm\{pref\}\}λanchor\\lambda\_\{\\mathrm\{anchor\}\}λtemplate\\lambda\_\{\\mathrm\{template\}\}MedQAPubMedQAMMLU\-Pro\-BioAvg1\.01\.01\.00\.4940\.2350\.1690\.2991\.01\.00\.50\.5410\.2870\.2420\.3571\.01\.00\.10\.5410\.5220\.3620\.4751\.00\.51\.00\.4470\.3130\.3620\.3741\.00\.50\.50\.5880\.2610\.1930\.3471\.00\.50\.10\.5410\.5220\.5320\.5321\.00\.11\.00\.5170\.4440\.1690\.3771\.00\.10\.50\.5650\.2610\.5560\.4601\.00\.10\.10\.5650\.2350\.5070\.4360\.51\.01\.00\.4940\.3650\.1930\.3510\.51\.00\.50\.3060\.3390\.2170\.2880\.51\.00\.10\.4700\.3910\.2660\.3760\.50\.51\.00\.4940\.4440\.3380\.4250\.50\.50\.50\.6120\.3390\.2420\.3970\.50\.50\.10\.6590\.3650\.4110\.4780\.50\.11\.00\.5410\.3130\.6040\.4860\.50\.10\.50\.5880\.1830\.5800\.4500\.50\.10\.10\.5880\.4960\.5070\.5300\.11\.01\.00\.5650\.2090\.2900\.3540\.11\.00\.50\.4470\.2870\.3140\.3490\.11\.00\.10\.3290\.3390\.1210\.2630\.10\.51\.00\.4940\.1040\.1450\.2480\.10\.50\.50\.4700\.2610\.0480\.2600\.10\.50\.10\.4940\.1830\.2660\.3140\.10\.11\.00\.4000\.1830\.3140\.2990\.10\.10\.50\.4230\.2350\.5070\.3890\.10\.10\.10\.5170\.1830\.3140\.338Table 14:Sensitivity of SPA to the loss weightsλpref\\lambda\_\{\\mathrm\{pref\}\},λanchor\\lambda\_\{\\mathrm\{anchor\}\}, andλtemplate\\lambda\_\{\\mathrm\{template\}\}on the Biomedical domain\. The selected configuration\(λpref,λanchor,λtemplate\)=\(1\.0,0\.5,0\.1\)\(\\lambda\_\{\\mathrm\{pref\}\},\\lambda\_\{\\mathrm\{anchor\}\},\\lambda\_\{\\mathrm\{template\}\}\)=\(1\.0,0\.5,0\.1\)\(bold\) achieves the highest average accuracy across MedQA, PubMedQA, and MMLU\-Pro\-Bio\. Based on this analysis,λpref\\lambda\_\{\\mathrm\{pref\}\}is fixed to1\.01\.0in the final objective\.SettingValueJudge modelGPT\-4Sampled candidate pools10,00010\{,\}000Candidates per pool \(KK\)4Chance baseline \(uniform overKK\)25\.0%25\.0\\%Agreement with KARMA chosen path74\.8%74\.8\\%Tie handlingCounted as disagreementTable 15:GPT\-4 validation of KARMA chosen\-path selection\. For each sampled pool, GPT\-4 ranks all candidates without preference labels; agreement denotes the fraction of pools in which GPT\-4’s top\-ranked candidate matches the chosen path selected by support\-based Top\-KK\.
## Appendix EChosen Path Validation

To verify that the support\-based Top\-KKselection \(Algorithm[2](https://arxiv.org/html/2607.03166#alg2)\) yields a chosen path that is genuinely the most plausible among the candidates in a pool, we conduct an LLM\-judge validation study using GPT\-4\. For each candidate pool, GPT\-4 is provided with the source\-target pair, the shared schema scaffold, and allKKverbalized candidates without preference labels, and is asked to identify the candidate whose intermediate entities form the most factually and semantically plausible reasoning chain between the source and target\. We then measure the agreement rate between GPT\-4’s top\-ranked candidate and the chosen path selected by our support\-based procedure\. Ties are conservatively counted as disagreement\.

As reported in Table[15](https://arxiv.org/html/2607.03166#A4.T15), GPT\-4 agrees with our chosen selection on74\.8%74\.8\\%of sampled pools, well above the25%25\\%chance baseline forK=4K=4candidates\. This indicates that entity\-support scoring is a reliable proxy for path plausibility, and that the resulting chosen\-rejected partition reflects a meaningful preference signal rather than an artifact of the heuristic\. The remaining disagreement reflects cases in which multiple candidates within a pool are comparably plausible, which is consistent with the resolution mismatch motivating SPA: at the entity\-slot resolution, several variants may be defensible, and the learning objective must be capable of expressing fine\-grained preferences among them\.

## Appendix FQualitative Analysis

Table[16](https://arxiv.org/html/2607.03166#A6.T16)shows a representative example of a KARMA\-generated candidate pool from the biomedical domain, and Table[17](https://arxiv.org/html/2607.03166#A6.T17)provides the corresponding path\-only view\.

FieldExampleInstructionUsing the following biomedical data, explain how the biological cascade initiating from Anatomy ’internal carotid artery’ reaches Biological Process\.Chosen candidateThis path represents a key hypothesis explaining the biological causality between \*\*’internal carotid artery’\*\* and \*\*’phospholipase C\-activating G\-protein coupled receptor signaling pathway’\*\*\. \*\*\[Detailed Mechanism\]\*\*: From a clinical perspective, \*\*’internal carotid artery’\*\* forms a biological interaction network with \*\*migraine\*\*, \*\*’migraine’\*\* represents a condition where \*\*Ergotamine\*\* is administered for symptomatic relief or cure, \*\*’Ergotamine’\*\* forms a biological interaction network with \*\*HTR2A\*\*, \*\*’HTR2A’\*\* is involved in \*\*phospholipase C\-activating G\-protein coupled receptor signaling pathway\*\*, regulating physiological homeostasis\. Furthermore ultimately leading to this conclusion\.Rejected candidate 1This path represents a key hypothesis explaining the biological causality between \*\*’internal carotid artery’\*\* and \*\*’phospholipase C\-activating G\-protein coupled receptor signaling pathway’\*\*\. \*\*\[Detailed Mechanism\]\*\*: From a clinical perspective, \*\*’internal carotid artery’\*\* forms a biological interaction network with \*\*migraine\*\*, \*\*’migraine’\*\* represents a condition where \*\*Cyproheptadine\*\* is administered for symptomatic relief or cure, \*\*’Cyproheptadine’\*\* forms a biological interaction network with \*\*HTR2A\*\*, \*\*’HTR2A’\*\* is involved in \*\*phospholipase C\-activating G\-protein coupled receptor signaling pathway\*\*, regulating physiological homeostasis\. Furthermore ultimately leading to this conclusion\.Rejected candidate 2This path represents a key hypothesis explaining the biological causality between \*\*’internal carotid artery’\*\* and \*\*’phospholipase C\-activating G\-protein coupled receptor signaling pathway’\*\*\. \*\*\[Detailed Mechanism\]\*\*: From a clinical perspective, \*\*’internal carotid artery’\*\* forms a biological interaction network with \*\*migraine\*\*, \*\*’migraine’\*\* represents a condition where \*\*Methylergometrine\*\* is administered for symptomatic relief or cure, \*\*’Methylergometrine’\*\* forms a biological interaction network with \*\*HTR2A\*\*, \*\*’HTR2A’\*\* is involved in \*\*phospholipase C\-activating G\-protein coupled receptor signaling pathway\*\*, regulating physiological homeostasis\. Furthermore ultimately leading to this conclusion\.Rejected candidate 3This path represents a key hypothesis explaining the biological causality between \*\*’internal carotid artery’\*\* and \*\*’phospholipase C\-activating G\-protein coupled receptor signaling pathway’\*\*\. \*\*\[Detailed Mechanism\]\*\*: From a clinical perspective, \*\*’internal carotid artery’\*\* forms a biological interaction network with \*\*migraine\*\*, \*\*’migraine’\*\* represents a condition where \*\*Cyclobenzaprine\*\* is administered for symptomatic relief or cure, \*\*’Cyclobenzaprine’\*\* forms a biological interaction network with \*\*HTR2A\*\*, \*\*’HTR2A’\*\* is involved in \*\*phospholipase C\-activating G\-protein coupled receptor signaling pathway\*\*, regulating physiological homeostasis\. Furthermore ultimately leading to this conclusion\.Table 16:Example of a generated candidate pool\.Preference labelCandidate pathChoseninternal carotid artery→\\rightarrowmigraine→\\rightarrowErgotamine→\\rightarrowHTR2A→\\rightarrowphospholipase C\-activating GPCR signaling pathwayRejected 1internal carotid artery→\\rightarrowmigraine→\\rightarrowCyproheptadine→\\rightarrowHTR2A→\\rightarrowphospholipase C\-activating GPCR signaling pathwayRejected 2internal carotid artery→\\rightarrowmigraine→\\rightarrowMethylergometrine→\\rightarrowHTR2A→\\rightarrowphospholipase C\-activating GPCR signaling pathwayRejected 3internal carotid artery→\\rightarrowmigraine→\\rightarrowCyclobenzaprine→\\rightarrowHTR2A→\\rightarrowphospholipase C\-activating GPCR signaling pathwayTable 17:Path\-only view of the candidate pool in Table[16](https://arxiv.org/html/2607.03166#A6.T16)\. All four candidates share the same Anatomy, Disease, Gene, and Biological Process nodes, while differing only in the Compound node \(bold\), illustrating the entity\-slot resolution at which preference learning must operate\.