Pareto-Optimal RTL Code Generation via Multi-Objective Reinforcement Learning with Large Language Models

# Pareto-Optimal RTL Code Generation via Multi-Objective Reinforcement Learning with Large Language Models
Source: [https://arxiv.org/html/2610.26904](https://arxiv.org/html/2610.26904)
Kenji UebayashiHikaru TomonariMasaya TsunokakeAffiliation:Koichiro YawataTatsuya SasakiYusuke OhtsuboKota DohiYasuhiro SogawaAffiliation:Research & Development Group, Hitachi, Ltd\.Affiliation:Correspondence:koki\.takeshita\.cx@hitachi\.com

###### Abstract

Recent studies have proposed reinforcement learning with verifiable rewards \(RLVR\) methods that enable LLMs to generate high\-quality RTL circuits from natural\-language specifications\. Existing approaches optimize circuit PPA \(Power, Performance, Area\) by aggregating multiple objectives into a single scalar reward, making it difficult to simultaneously generate diverse Pareto\-optimal solutions across different trade\-offs such as low power, low latency, and small area\. However, in practical design, the relative importance of each objective varies depending on product requirements and design constraints; thus, obtaining a diverse and high\-quality candidate set is beneficial in that designers can choose a preferred trade\-off a posteriori\. In this work, we are the first to propose a learning method for LLM\-based circuit generation that optimizes, not the quality of a single circuit, but the quality and diversity of the entire circuit set obtained through multiple generations in the PPA objective space\. The proposed method consists of \(1\) a generation\-based training pipeline that reformulates coverage improvement of the entire Pareto solution set into a sequential circuit generation problem, and \(2\) a reward function based on Hypervolume Contribution \(HVC\) that measures how much a generated circuit contributes to the Pareto front\. Evaluation on the RTLLM v2 benchmark shows that the proposed method achieves \+13\.3 percentage points in relative Hypervolume over a baseline without PPA optimization and \+14\.4 percentage points over a single\-objective reinforcement learning method, while also demonstrating superiority in circuit diversity\.

## 1Introduction

Recently, reinforcement learning with verifiable rewards \(RLVR\), such as GRPO, has emerged\([Shao et al\., 2024](https://arxiv.org/html/2610.26904#bib.bib1)\), and the performance of LLMs on problems requiring reasoning capabilities has improved dramatically\. Led by OpenAI’s o1 and o3\([OpenAI, 2024](https://arxiv.org/html/2610.26904#bib.bib3);[OpenAI, 2025](https://arxiv.org/html/2610.26904#bib.bib4)\)and DeepSeek\-R1\([Guo et al\., 2025](https://arxiv.org/html/2610.26904#bib.bib2)\), subsequent models have been continuously released\([Yang et al\., 2025](https://arxiv.org/html/2610.26904#bib.bib5);[Ettinger et al\., 2025](https://arxiv.org/html/2610.26904#bib.bib6)\)\. RLVR enables the acquisition of advanced logical reasoning capabilities in tasks where the quality of answers can be automatically verified, such as mathematics, programming, and question answering\.

This paradigm has also been applied to electronic design automation \(EDA\), particularly to the NL\-Code task of generating hardware description languages \(HDLs\) such as Verilog\([Flake et al\., 2020](https://arxiv.org/html/2610.26904#bib.bib21)\)from natural\-language specifications\.[Zhu et al\. \(2026\)](https://arxiv.org/html/2610.26904#bib.bib12);[Wang et al\. \(2026\)](https://arxiv.org/html/2610.26904#bib.bib7)automatically verify whether a generated circuit conforms to its natural\-language specification using behavioral simulators such as Icarus Verilog\([Williams and Baxter, 2002](https://arxiv.org/html/2610.26904#bib.bib13)\), and improve specification compliance by learning with binary verification results as rewards\.[Chen et al\. \(2025\)](https://arxiv.org/html/2610.26904#bib.bib8);[Hsiao et al\. \(2025\)](https://arxiv.org/html/2610.26904#bib.bib9)additionally estimate the PPA \(Power, Performance, Area\) of generated circuits using downstream circuit evaluation tools such as OpenROAD\([Ajayi et al\., 2019](https://arxiv.org/html/2610.26904#bib.bib14)\), aggregate PPA into a single scalar reward via multiplicative or weighted\-sum scalarization, and perform reinforcement learning\. This enables generating not only specification\-compliant but also higher\-performance circuits\.

However, these existing studies optimize the generation quality of a single circuit under the scalarization function used during training\. That is, trained models do not explicitly learn to obtain a solution set spanning different trade\-offs such as low power, low latency, and small area\. On the other hand, in circuit design, the relative importance of objectives such as power, delay, and area varies depending on product requirements and design constraints, and trade\-offs frequently arise among these objectives\. Therefore, obtaining a diverse set of Pareto\-optimal candidates is beneficial because it allows designers to choose a preferred candidate a posteriori according to the desired trade\-off\([Dobeš et al\., 2013](https://arxiv.org/html/2610.26904#bib.bib16);[Vişan et al\., 2022](https://arxiv.org/html/2610.26904#bib.bib17);[Liu and Carloni, 2013](https://arxiv.org/html/2610.26904#bib.bib18)\)\. Furthermore, diverse and high\-quality circuit candidates may be useful for generating promising parent individuals in search methods that combine LLM\-based generation with evolutionary algorithms, such as AlphaEvolve\([Novikov et al\., 2025](https://arxiv.org/html/2610.26904#bib.bib20)\)\. \(Details on the benefits of diverse and high\-quality RTL generation are discussed in Section[2\.2](https://arxiv.org/html/2610.26904#S2.SS2)\.\)

In this work, for LLM\-based circuit generation, we propose a learning method that aims to improve the quality and diversity in the PPA objective space of the entire circuit set obtained through multiple generations, rather than maximizing a single scalar reward for a single circuit generation\. To this end, \(1\) we introduce a generation\-based training pipeline that reformulates the set\-level optimization problem of improving coverage of the entire Pareto set into a sequential decision\-making problem of generating the next candidate to add, enabling learning via GRPO\-based methods\. \(2\) We introduce a reward function based on Hypervolume Contribution \(HVC\) that encourages the generation of solution groups close to the ideal Pareto front\. For the novel problem setting of Pareto solution exploration with LLMs in circuit generation, we define an evaluation metric called relative Hypervolume \(rHV\) and evaluate on RTLLM v2, a standard benchmark for NL\-Code tasks\. The proposed method achieves \+13\.3 percentage points in rHV over the baseline and \+14\.4 percentage points over the single\-objective learning method\.

## 2Background

### 2\.1Abstraction Levels and Analysis Methods of Hardware Description Languages

In digital hardware design, circuits are described using hardware description languages \(HDLs\), which serve a role analogous to programming languages in software\. HDLs are broadly categorized by their level of abstraction, ranging from high\-level behavioral descriptions that focus on what the circuit should do, to low\-level structural descriptions that specify connections between logic gates or standard cells\. Among these, one of the central representations used in LLM\-based circuit generation is Register Transfer Level \(RTL\), which describes data transfers between registers and clock\-cycle behavior\. RTL is widely used in industry and is amenable to behavioral verification and performance evaluation through simulation\([Liu et al\., 2025](https://arxiv.org/html/2610.26904#bib.bib15)\)\. Representative languages include Verilog\([Flake et al\., 2020](https://arxiv.org/html/2610.26904#bib.bib21)\)and VHDL\([Shahdad, 1986](https://arxiv.org/html/2610.26904#bib.bib22)\)\.

Test code for verifying whether an RTL\-designed circuit correctly satisfies its specification is also written in HDL\. This is called a testbench\. Specification verification is performed by executing the testbench on a dedicated behavioral simulator\. For Verilog, a representative behavioral simulator is Icarus Verilog\([Williams and Baxter, 2002](https://arxiv.org/html/2610.26904#bib.bib13)\)\.

To estimate the PPA \(Power, Performance, Area\) of an RTL\-designed circuit, it must first be converted to a lower\-level gate\-level representation\. This conversion is called logic synthesis\. A representative logic synthesis tool is Yosys\([Wolf and Glaser, 2013](https://arxiv.org/html/2610.26904#bib.bib23)\)\. After logic synthesis, physical design tools such as OpenROAD\([Ajayi et al\., 2019](https://arxiv.org/html/2610.26904#bib.bib14)\)perform placement and routing to estimate PPA\.

### 2\.2Benefits of Diverse and High\-Quality RTL Generation

In multi\-objective circuit design, the relative importance of objectives such as power, delay, and area varies depending on product requirements and design constraints\. Therefore, obtaining a diverse set of Pareto\-optimal candidates is beneficial because it allows designers to choose a preferred candidate a posteriori according to the desired trade\-off\([Dobeš et al\., 2013](https://arxiv.org/html/2610.26904#bib.bib16);[Vişan et al\., 2022](https://arxiv.org/html/2610.26904#bib.bib17);[Liu and Carloni, 2013](https://arxiv.org/html/2610.26904#bib.bib18)\)\. While similar trade\-off exploration can also be performed using downstream logic synthesis and physical design tools, such evaluation loops are computationally expensive, requiring many synthesis runs, with each candidate taking minutes to hours; thus, iterative downstream evaluation is costly\([Bai et al\., 2023](https://arxiv.org/html/2610.26904#bib.bib19)\)\.[Novikov et al\. \(2025\)](https://arxiv.org/html/2610.26904#bib.bib20)also argues that directly manipulating RTL to improve circuits has the advantage of performing optimization early in the design flow\. For this reason, generating diverse and high\-quality circuits at the upstream RTL stage is practically useful in that promising alternatives can be identified before invoking costly downstream evaluation\.

Furthermore, the diversity of RTL candidates may also be useful in search methods that combine code generation and improvement by trained LLMs with evolutionary algorithms, such as AlphaEvolve\([Novikov et al\., 2025](https://arxiv.org/html/2610.26904#bib.bib20)\)\. AlphaEvolve maintains a database of previously generated programs and their evaluation results, reusing them as parent candidates and sources of inspiration while generating next candidates\. Experiments have verified its effectiveness across diverse domains, and in particular, for a task targeting TPU arithmetic circuits, it demonstrated that directly modifying RTL can present optimization candidates early in the design flow\. As an important implication,[Novikov et al\. \(2025\)](https://arxiv.org/html/2610.26904#bib.bib20)states that retaining candidates that are good under different criteria, rather than keeping only candidates that are good under a single criterion, can stimulate the generation of more diverse candidate solutions\. This insight suggests that diverse and high\-quality RTL candidates may be valuable not only for human designers but also as a promising parent candidate pool for subsequent evolutionary search\.

## 3Related Work

### 3\.1RLVR in Circuit Generation

ChipSeek\-R1\([Chen et al\., 2025](https://arxiv.org/html/2610.26904#bib.bib8)\)proposed a reinforcement learning framework for RTL generation based on a hierarchical reward design combining syntactic correctness, functional correctness, synthesizability, and PPA\. For PPA rewards, it aggregates PPA values estimated using OpenROAD\([Ajayi et al\., 2019](https://arxiv.org/html/2610.26904#bib.bib14)\)into a single reward via a multiplication\-based scalarization function, achieving RTL code generation with PPA performance exceeding human designs\. BUFFALO\([Hsiao et al\., 2025](https://arxiv.org/html/2610.26904#bib.bib9)\)applies GRPO to the buffer tree generation task, updating model parameters based on circuit quality metrics obtained from downstream physical design tool evaluation\. Specifically, it normalizes the timing improvement and area increase and aggregates them into a scalar reward via a weighted sum, which is then maximized during training\.

All of these methods optimize single circuit generation under a single scalarization function, and do not explicitly learn to generate diverse and high\-quality solution sets spanning different PPA trade\-offs\. In contrast, this work aims to improve the quality and diversity of the entire circuit set obtained through multiple generations for a given specification in the PPA objective space\.

### 3\.2Evaluation of Multi\-Objective Reinforcement Learning

[Teoh et al\. \(2025\)](https://arxiv.org/html/2610.26904#bib.bib11)proposed the Normalized Hypervolume Generalization Ratio \(NHGR\) for evaluating cross\-environment generalization in multi\-objective reinforcement learning\. NHGR is defined by normalizing each objective based on the element\-wise minimum and maximum of the optimal Pareto front for each evaluation context, then dividing the normalized hypervolume of the approximate front obtained by the generalist by the normalized hypervolume of the optimal front\. This normalization is introduced to prevent the difficulty of comparison that arises when ordinary Hypervolume depends on the scale of objective values and the range of achievable Pareto fronts for each problem\. Moreover, by defining it as a ratio, it offers the advantage of intuitively interpreting how good the quality and diversity of the solution set are relative to the ideal Pareto front\. When the true Pareto front is unavailable, non\-dominated value vectors from specialist agents trained independently for each context are aggregated as an approximation of the Pareto front\.

Our setting shares the motivation with[Teoh et al\. \(2025\)](https://arxiv.org/html/2610.26904#bib.bib11)but differs in the problem unit\. In[Teoh et al\. \(2025\)](https://arxiv.org/html/2610.26904#bib.bib11), the evaluation target is the “environment context,” and specialists can be trained for each context\. In our work, we target LLM\-based circuit generation, with evaluation units being individual samples in the benchmark\. Thus, training a specialist LLM separately for each sample to construct a reference front is impractical\. Instead, we integrate solutions obtained from all models and all trials for each problem and use the resulting non\-dominated set as a pseudo Pareto front\.

## 4Proposed Method

Figure[1](https://arxiv.org/html/2610.26904#S4.F1)shows an overview of the proposed method\. The method is a generation\-based training pipeline that alternates between “reinforcement learning with HVC reward” and “dataset reconstruction\.” By sequentially learning to generate the next circuit to add given the circuit history obtained so far, the method improves the quality and diversity of the overall circuit set obtained in the end\. “Reinforcement learning with HVC reward” provides the LLM, conditioned on circuit history, with Hypervolume Contribution \(HVC\) as a reward indicating how much the generated circuit contributes to the Pareto front relative to the existing circuit set, and trains it using reinforcement learning\. “Dataset reconstruction” aggregates circuits and their evaluation results generated during training, merges them as new circuit history into the current training dataset, and constructs the next\-generation training dataset\. This enables efficient learning of Pareto solution exploration capabilities while progressively extending circuit history\.

### 4\.1Reinforcement Learning with HVC Reward

In this phase, the model takes as input a specification written in natural language and previously generated circuits along with their PPAs during training, and learns to output the next circuit to generate via reinforcement learning\. The dataset used consists of an initial dataset containing specifications written in natural language, corresponding testbenches, and ground\-truth circuit examples, merged with circuit history extracted through “dataset reconstruction\.” As a reward, we use Hypervolume Contribution \(HVC\), which indicates how much the Hypervolume increases in the objective function space when the circuit is added to the existing circuit set\. This enables the model to learn to generate circuits with large contributions to the Pareto front relative to the existing circuit set at each generation\. At test time, the model recursively adds generated circuits and their evaluation values to the circuit history while generating circuits\. Since the model is trained to generate circuits with the largest Hypervolume Contribution at each generation iteration, the quality and diversity of the overall circuit set obtained in the end are expected to improve\.

#### Reinforcement Learning Algorithm

We use GRPO \(Group Relative Policy Optimization\)\([Shao et al\., 2024](https://arxiv.org/html/2610.26904#bib.bib1)\), which generates multiple responses\{yi\}i=1G\\\{y\_\{i\}\\\}\_\{i=1\}^\{G\}from the old policyπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}for each inputqq, computes advantages from relative rewards within the group, and updates the policy with clipping and KL regularization without a separate value function\.

𝒥GRPO​\(θ\)=𝔼q,\{yi\}i=1G∼πθold\[1G∑i=1G1\|yi\|∑t=1\|yi\|\(min\(ri,tA^i,t,clip\(ri,t,1±ϵ\)A^i,t\)−βDKL\[πθ∥πref\]\)\]\\displaystyle\\begin\{aligned\} \\mathcal\{J\}\_\{\\mathrm\{GRPO\}\}\(\\theta\)&=\\mathbb\{E\}\_\{q,\\,\\\{y\_\{i\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\}\\Biggl\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{\|y\_\{i\}\|\}\\sum\_\{t=1\}^\{\|y\_\{i\}\|\}\\Bigl\(\\min\\\!\\bigl\(r\_\{i,t\}\\,\\hat\{A\}\_\{i,t\},\\;\\operatorname\{clip\}\(r\_\{i,t\},\\,1\{\\pm\}\\epsilon\)\\,\\hat\{A\}\_\{i,t\}\\bigr\)\\\\ &\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\-\\beta\\,D\_\{\\mathrm\{KL\}\}\\\!\\bigl\[\\pi\_\{\\theta\}\\,\\\|\\,\\pi\_\{\\mathrm\{ref\}\}\\bigr\]\\Bigr\)\\Biggr\]\\end\{aligned\}\(1a\)ri,t=πθ​\(yi,t∣q,yi,<t\)πθold​\(yi,t∣q,yi,<t\),A^i,t=A^i=Ri−mean⁡\(\{Rj\}j=1G\)std⁡\(\{Rj\}j=1G\)\\displaystyle r\_\{i,t\}=\\frac\{\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid q,\\,y\_\{i,<t\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(y\_\{i,t\}\\mid q,\\,y\_\{i,<t\}\)\},\\qquad\\hat\{A\}\_\{i,t\}=\\hat\{A\}\_\{i\}=\\frac\{R\_\{i\}\-\\mathrm\{mean\}\(\\\{R\_\{j\}\\\}\_\{j=1\}^\{G\}\)\}\{\\mathrm\{std\}\(\\\{R\_\{j\}\\\}\_\{j=1\}^\{G\}\)\}\(1b\)
![Refer to caption](https://arxiv.org/html/2610.26904v1/overview.v2.png)Figure 1:Overview of the proposed method\. By alternating between reinforcement learning with HVC reward \(top\) and dataset reconstruction \(bottom\), the model learns Pareto solution exploration capabilities while progressively extending circuit history\.
#### Input Prompt

The input prompt consists of circuit specificationssand circuit historyℋ\\mathcal\{H\}\. The circuit history lists previously generated circuits and their PPA values\. This allows the model to obtain information for exploring new Pareto solutions different from existing solutions\. The actual prompt template is shown in Appendix[C](https://arxiv.org/html/2610.26904#A3)\.

#### Hypervolume Contribution \(HVC\)

To compute HVC, testbench verification is first performed on Icarus Verilog\([Williams and Baxter, 2002](https://arxiv.org/html/2610.26904#bib.bib13)\)for each candidate generated through group sampling\. Only candidates that pass are subjected to PPA estimation using Yosys\([Wolf and Glaser, 2013](https://arxiv.org/html/2610.26904#bib.bib23)\)and OpenROAD\([Ajayi et al\., 2019](https://arxiv.org/html/2610.26904#bib.bib14)\), following a similar approach to[Chen et al\. \(2025\)](https://arxiv.org/html/2610.26904#bib.bib8)\. Candidates failing the testbench or with invalid PPA receiveHVC=0\\mathrm\{HVC\}=0\. For candidates with valid PPA, normalization based on the ground\-truth circuit’s PPA is performed to define objective function vectors\. The objective function vector for candidateiiis:

𝐨i=\(oP,i,oD,i,oA,i\),oP,i=PgoldenPgen,i\+ϵ,oD,i=DgoldenDgen,i\+ϵ,oA,i=AgoldenAgen,i\+ϵ\\mathbf\{o\}\_\{i\}=\(o\_\{P,i\},o\_\{D,i\},o\_\{A,i\}\),\\quad o\_\{P,i\}=\\frac\{P\_\{\\text\{golden\}\}\}\{P\_\{\\text\{gen\},i\}\+\\epsilon\},\\;o\_\{D,i\}=\\frac\{D\_\{\\text\{golden\}\}\}\{D\_\{\\text\{gen\},i\}\+\\epsilon\},\\;o\_\{A,i\}=\\frac\{A\_\{\\text\{golden\}\}\}\{A\_\{\\text\{gen\},i\}\+\\epsilon\}\(2\)wherePgolden,Dgolden,AgoldenP\_\{\\text\{golden\}\},D\_\{\\text\{golden\}\},A\_\{\\text\{golden\}\}are the ground\-truth circuit’s PPA,Pgen,i,Dgen,i,Agen,iP\_\{\\text\{gen\},i\},D\_\{\\text\{gen\},i\},A\_\{\\text\{gen\},i\}are candidateii’s PPA, andϵ\\epsilonis a small constant to avoid division by zero\. This normalization makes each component 1\.0 for performance equal to the ground\-truth circuit and greater than 1\.0 when exceeding it\. Note thatPgen,i,Dgen,i,Agen,iP\_\{\\text\{gen\},i\},D\_\{\\text\{gen\},i\},A\_\{\\text\{gen\},i\}are all better when smaller\.

Figure[2](https://arxiv.org/html/2610.26904#S4.F2)shows an example of multiple candidates and their HVCs in the objective function space\. HVC is defined as the increment in Hypervolume with origin as the reference point when adding candidateii:

HVC⁡\(𝐨i\)=HV⁡\(\{𝐨j∣j≤i\}\)−HV⁡\(\{𝐨j∣j<i\}\)\\mathrm\{HVC\}\(\\mathbf\{o\}\_\{i\}\)=\\mathrm\{HV\}\(\\\{\\mathbf\{o\}\_\{j\}\\mid j\\leq i\\\}\)\-\\mathrm\{HV\}\(\\\{\\mathbf\{o\}\_\{j\}\\mid j<i\\\}\)\(3\)whereHV\\mathrm\{HV\}is a function that computes the Hypervolume in the objective function space\. We use the implementation of[Fortin et al\. \(2012\)](https://arxiv.org/html/2610.26904#bib.bib24)for computing Hypervolume\. Since the objective function𝐨i\\mathbf\{o\}\_\{i\}is a maximization metric, a larger HVC means that the candidate contributes more to the Pareto front\. Due to high reward variance, reward values are clipped to the range\[0,3\.0\]\[0,\\,3\.0\]for training stabilization\.

The method uses rewardRiR\_\{i\}as a combination of testbench verification reward and HVC\. Candidates passing the testbench receive a pass reward of1\.01\.0plus HVC, while failing candidates receive00\. In summary, the variables in Equation[1](https://arxiv.org/html/2610.26904#S4.E1)are determined as follows:

q=\(s,ℋ\),yi∼πθold\(⋅∣q\),Ri=\{1\.0\+HVC⁡\(𝐨i\)if testbench passed0otherwiseq=\(s,\\;\\mathcal\{H\}\),\\quad y\_\{i\}\\sim\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(\\cdot\\mid q\),\\quad R\_\{i\}=\\begin\{cases\}1\.0\+\\mathrm\{HVC\}\(\\mathbf\{o\}\_\{i\}\)&\\text\{if testbench passed\}\\\\ 0&\\text\{otherwise\}\\end\{cases\}\(4\)
Figure 2:Example of Hypervolume Contribution \(HVC\)\. In the objective function space, the HVC is the Hypervolume increment with origin as the reference point when adding candidateii\. A larger HVC means the candidate contributes more to the Pareto front\.

### 4\.2Generation\-based Dataset Reconstruction

After each generation of training, circuits generated during training are aggregated to construct the next\-generation training dataset\. This process progressively extends circuit history, enabling the model to learn Pareto solution exploration with longer histories as training progresses\.

Specifically, for each candidate generated during training, the generated circuit, its PPA value, HVC, and the input circuit history are recorded\. Among circuits generated for the same original sample \(i\.e\., sharing the same specification and circuit history\), the candidate with the highest HVC is selected\. The selected circuit is then concatenated with the previous circuit history and added to the next\-generation dataset\. To prevent excessive input length growth, samples with a history length of 5 or more are removed\. Additionally, to control dataset size, an upper limit of 1,000 samples per history length is imposed, with the oldest samples removed when this limit is exceeded\. Finally, the original dataset without circuit history is added to complete the next\-generation training dataset\.

Through this dataset reconstruction, starting from history length 0, samples with history lengths 1, 2, …are added as generations progress\. By training on circuit histories of various lengths, the model acquires sequential Pareto solution exploration capabilities\.

## 5Experiments

### 5\.1Baselines

We evaluate all methods on two publicly available RTL\-generation models as base models:VeriReason\-3B\([Wang et al\., 2026](https://arxiv.org/html/2610.26904#bib.bib7)\)andCodeV\-R1\-Distil\-7B\([Zhu et al\., 2026](https://arxiv.org/html/2610.26904#bib.bib12)\)\. Using two base models with different training recipes allows us to verify that the effectiveness of the proposed method is not specific to a particular base model\.VeriReason\-3Bis a Qwen2\.5\-Coder\-3B\-based model trained through supervised fine\-tuning on RTL generation data and GRPO reinforcement learning based on testbench feedback, but it does not directly optimize PPA objectives as rewards\.CodeV\-R1\-Distil\-7Bis a Qwen2\.5\-Coder\-7B\-Instruct\-based reasoning\-enhanced Verilog generation model that similarly emphasizes functional correctness without directly optimizing PPA objectives\. Both are thus positioned as existing strong RTL generation models that emphasize functional correctness\.

On top of each base model, we compare the following five methods, designed from three perspectives to comprehensively verify the effectiveness of the proposed method: \(1\) the unmodified base model without PPA optimization \(Base\) to evaluate the effect of PPA\-aware training; \(2\) single\-scalar\-reward reinforcement learning \(Single\-Objective PPA\) to verify the superiority of HVC reward as multi\-objective optimization; \(3\) models using the same generation\-based training pipeline as the proposed method but with different reward functions \(Sequential TB, Sequential TB\+PPA\) to verify that generation\-based training alone is insufficient and that combining it with HVC reward is essential\.

Baseis the unmodified base model \(VeriReason\-3B or CodeV\-R1\-Distil\-7B\) without any additional PPA\-aware training, and is positioned as an existing strong RTL generation model that emphasizes functional correctness\.Single\-Objective PPAis a model additionally trained with GRPO on the base model using testbench pass reward plus scalarized PPA reward similar to[Chen et al\. \(2025\)](https://arxiv.org/html/2610.26904#bib.bib8)\(the formula is shown in Appendix[B](https://arxiv.org/html/2610.26904#A2)\), and corresponds to the single\-objective reinforcement learning baseline in this work\.Sequential TBis a model trained using the same generation\-based training pipeline as the proposed method on the base model, with only the reward function replaced by testbench pass reward\. It performs sequential generation conditioned on circuit history during evaluation, but PPA\-based optimization or contributions to the Pareto front are not included in the reward\.Sequential TB\+PPAis a model trained using the same generation\-based training pipeline as the proposed method on the base model, with only the reward function replaced by the sum of testbench pass reward and PPA scalar reward\. That is, training is performed in a sequential generation setting, but the optimization target remains a single scalar reward without directly optimizing the quality or diversity of the entire solution set\.Sequential HVC \(Ours\)is a model trained using the proposed generation\-based training pipeline on the base model with reward based on testbench pass reward and Hypervolume Contribution \(HVC\)\. By learning to sequentially generate candidates with large contributions to the Pareto front relative to existing circuit history, the method aims to improve the quality and diversity of the entire solution set\.

### 5\.2Dataset

Training data were constructed through two\-stage filtering based on VeriReason’s\([Wang et al\., 2026](https://arxiv.org/html/2610.26904#bib.bib7)\)hard level \(1,149 problems\)\. First, 889 problems where the ground\-truth circuit passed testbench verification and PPA estimation were extracted and used as training data for non\-generation\-based single\-objective training \(Single\-Objective PPA\)\. Next, for these 889 problems, multiple circuit candidates were generated using CodeV\-R1\-Distil\-7B\([Zhu et al\., 2026](https://arxiv.org/html/2610.26904#bib.bib12)\), and 78 problems where negative correlation \(r<0r<0\) was observed between area and delay were extracted\. This is because problems where the Pareto front degenerates to a single point provide weak learning signals for multi\-objective optimization\. These 78 problems were used for generation\-based training \(Sequential TB/Sequential TB\+PPA/Sequential HVC\)\. For evaluation, 45 problems out of 50 in RTLLM v2\([Liu et al\., 2024](https://arxiv.org/html/2610.26904#bib.bib10)\)where testbench verification and PPA estimation succeeded were used\.

### 5\.3Evaluation Method

For each RTLLM v2 problem, circuits were generated by recursively adding generated circuits and their evaluation values to the circuit history, and evaluated using the following three metrics\. Evaluation was conducted with high\-temperature sampling \(T=0\.5T=0\.5\) with multiple trials per problem to reduce variance\.

#### Relative Hypervolume \(rHV\)

This metric is inspired by the NHGR of[Teoh et al\. \(2025\)](https://arxiv.org/html/2610.26904#bib.bib11)described in Section 3\.2\. In this work, for each problemii, we construct areference archiveΠi=ND⁡\(Ui\)\\Pi\_\{i\}=\\mathrm\{ND\}\(U\_\{i\}\)as the non\-dominated set of the unionUiU\_\{i\}of solutions obtained from all models and all trials, and define rHV as follows:

rHVm,i,j=HV⁡\(S~m,i,j,𝐫\)HV⁡\(Π~i,𝐫\)\\mathrm\{rHV\}\_\{m,i,j\}=\\frac\{\\mathrm\{HV\}\(\\tilde\{S\}\_\{m,i,j\};\\mathbf\{r\}\)\}\{\\mathrm\{HV\}\(\\tilde\{\\Pi\}\_\{i\};\\mathbf\{r\}\)\}\(5\)whereSm,i,jS\_\{m,i,j\}is the set of circuits generated by methodmmon problemiiin trialjj,S~m,i,j\\tilde\{S\}\_\{m,i,j\}andΠ~i\\tilde\{\\Pi\}\_\{i\}are these solution sets mapped to the objective space normalized to\[0,1\]\[0,\\,1\]using the minimum and maximum PPA values inUiU\_\{i\}, and𝐫\\mathbf\{r\}is the reference point\. rHV takes values in\[0,1\]\[0,\\,1\], with values closer to 1\.0 indicating achievement of a Pareto front closer to the reference archive\. For aggregate evaluation, we report the mean across all problems and trials:rHV¯m\\bar\{\\mathrm\{rHV\}\}\_\{m\}\. See Appendix[A](https://arxiv.org/html/2610.26904#A1)for detailed mathematical definitions of rHV\.

#### Win Rate

For each problemiiand trialjj, a win is assigned to the model achieving the highest Hypervolume among all compared models\. When multiple models achieve the same maximum Hypervolume, the win is split equally among them\. Formally, the Win Rate for modelmmon problemiiand trialjjis defined as:

WRm,i,j=𝟏\[HVm,i,j=maxm′HVm′,i,j\]\|\{m′:HVm′,i,j=maxm′′⁡HVm′′,i,j\}\|\\mathrm\{WR\}\_\{m,i,j\}=\\frac\{\\mathbf\{1\}\[\\mathrm\{HV\}\_\{m,i,j\}=\\max\_\{m^\{\\prime\}\}\\mathrm\{HV\}\_\{m^\{\\prime\},i,j\}\]\}\{\|\\\{m^\{\\prime\}:\\mathrm\{HV\}\_\{m^\{\\prime\},i,j\}=\\max\_\{m^\{\\prime\\prime\}\}\\mathrm\{HV\}\_\{m^\{\\prime\\prime\},i,j\}\\\}\|\}\(6\)where𝟏​\[⋅\]\\mathbf\{1\}\[\\cdot\]is the indicator function\. For aggregate evaluation, we report the mean across all problems and trials:WR¯m=1nd​nt​∑i,jWRm,i,j\\bar\{\\mathrm\{WR\}\}\_\{m\}=\\frac\{1\}\{n\_\{d\}n\_\{t\}\}\\sum\_\{i,j\}\\mathrm\{WR\}\_\{m,i,j\}\. Win Rate takes values in\[0,1\]\[0,\\,1\], with larger values indicating greater superiority over other models\. Since Win Rate considers only relative wins within each problem, it can be directly aggregated without normalization even when Hypervolume scales vary significantly across problems\.

#### Circuit Diversity

The number of circuits with unique PPA values generated per problem and trial\. Higher values indicate that more diverse circuits can be generated in a single trial\.

## 6Results

### 6\.1Relative Hypervolume Comparison

Table 1:Relative Hypervolume \(rHV\) for each model\. Each column shows the rHV measured after a given number of sequential generation iterations \(iter\), where the model generates a new circuit conditioned on the circuit history accumulated so far\.Table[1](https://arxiv.org/html/2610.26904#S6.T1)shows the relative Hypervolume \(rHV\) for each model\. The proposed method \(Sequential HVC\) achieved the highest rHV at iter=5 and iter=10\. With VeriReason\-3B as the base model, at iter=10, it showed 32\.4% compared to Base’s 19\.1% \(\+13\.3 percentage points\) and Single\-Objective PPA’s 18\.0% \(\+14\.4 percentage points\)\. With CodeV\-R1\-Distil\-7B as the base model, at iter=10, it likewise showed 43\.2% compared to Base’s 38\.4% \(\+4\.8 percentage points\) and Single\-Objective PPA’s 37\.8% \(\+5\.4 percentage points\), confirming that the effectiveness of the proposed method does not depend on the choice of base model\.

### 6\.2Win Rate Comparison

Figure[3](https://arxiv.org/html/2610.26904#S6.F3)shows the Win Rate progression per generation count \(iteration\)\. The proposed method \(Sequential HVC\) achieved the highest Win Rate at almost all iterations in both base models; the only exception is iter=1 for CodeV\-R1\-Distil\-7B, where Sequential TB is marginally higher\. As the generation count increases, its Win Rate rises steeply and its lead over the other methods widens\. This indicates that the proposed method not only finds good circuits initially but also continues exploration using circuit history to acquire more diverse Pareto\-optimal solutions\.

Notably, the Win Rate of the proposed method is maintained at a high level even beyond iter=5, the maximum circuit\-history length seen during training, rather than degrading for longer exploration\. This confirms that the proposed method generalizes to exploration horizons longer than those used in training\.

Conversely, the Win Rate of Sequential TB and Sequential TB\+PPA declines as the number of iterations increases, leaving them with the lowest Win Rate at later iterations in both base models\. This suggests that with testbench reward or scalar PPA reward alone, reward hacking occurs where the model simply regenerates previously generated circuits, so that additional iterations fail to explore new Pareto solutions\.

### 6\.3Circuit Diversity Comparison

![Refer to caption](https://arxiv.org/html/2610.26904v1/win_rate.verireason.png)\(a\)VeriReason\-3B
![Refer to caption](https://arxiv.org/html/2610.26904v1/win_rate.codev-r1.png)\(b\)CodeV\-R1\-Distil\-7B

Figure 3:Win Rate progression per generation count\.![Refer to caption](https://arxiv.org/html/2610.26904v1/circuit_diversity.verireason.png)\(a\)VeriReason\-3B
\(b\)CodeV\-R1\-Distil\-7B
Figure 4:Circuit diversity comparison\.Figure[4](https://arxiv.org/html/2610.26904#S6.F4)shows the circuit diversity comparison\. The proposed method \(Sequential HVC\) was confirmed to generate the most diverse circuits in both base models\. Base and Single\-Objective PPA show intermediate diversity, with their relative ordering depending on the base model\.

On the other hand, Sequential TB and Sequential TB\+PPA exhibit the lowest circuit diversity in both base models, frequently producing only a single unique circuit per trial\. This directly evidences the reward hacking observed in the Win Rate results in Section[6\.2](https://arxiv.org/html/2610.26904#S6.SS2): with testbench reward or scalar PPA reward alone, the model tends to regenerate previously generated circuits, suppressing exploration of new circuits\. This underscores that explicitly incorporating contributions to the Pareto front into the reward, as in HVC, is essential in the proposed generation\-based pipeline\. To further confirm that the gains of the proposed method do not merely stem from avoiding duplicate generations, we conduct an ablation that adds an explicit repetition penalty to the sequential baseline in Appendix[E\.4](https://arxiv.org/html/2610.26904#A5.SS4)\.

## 7Conclusion

The experimental results of this work provide important insights for handling multi\-objective optimization in reinforcement learning with verifiable rewards for LLMs\.

First, multi\-objective reinforcement learning with HVC reward \(Sequential HVC\) was shown to be effective for Pareto\-optimal solution exploration compared to single\-objective learning \(Single\-Objective PPA\)\. At iter=10, the proposed method outperformed Single\-Objective PPA by 14\.4 percentage points in rHV and also showed superiority in circuit diversity\. This result suggests that HVC reward can effectively learn trade\-off structures that cannot be captured by conventional approaches that aggregate multiple objective functions into a single scalar value\.

Second, it became clear that reward design is extremely important in the proposed generation\-based training pipeline\. Sequential TB and Sequential TB\+PPA yielded the lowest circuit diversity and, at later iterations, the lowest Win Rate across both base models, indicating that with testbench reward or scalar PPA reward alone, reward hacking occurred where the model simply regenerates previously generated circuits\. This indicates that to promote exploration of new solutions in generation\-based training, contributions to the Pareto front must be explicitly incorporated into the reward\.

Third, as generation count increases, the superiority of the proposed method expands, and performance improvement continues even for longer exploration \(up to iter=10\) than training \(up to iter=5\)\. This generalization capability suggests that the proposed method acquires the essential ability of sequential Pareto solution exploration using circuit history, rather than simply overfitting to specific exploration lengths\.

#### Limitations

This work has several limitations\. First, this work targets only circuit design, and application to other engineering fields such as control engineering is unverified\. Second, this work verified only the setting where the objective function space dimension is 3 \(PPA\)\. Since the proposed method requires computing Hypervolume for each rollout during training, addressing increased computational complexity when the objective function space dimension increases remains a challenge\. Third, due to LLM context length limitations, there is an upper limit on the length of circuit history provided as input\. Therefore, for problems with very many solutions on the Pareto front, it is not possible to provide all history as input, making it difficult to enumerate all solutions\. Fourth, limitations remain regarding data scale: the evaluation benchmark is limited to a single benchmark, RTLLM v2, and generalization to other benchmarks or more diverse circuit specifications remains unverified\. Additionally, the training data used for generation\-based training \(Sequential TB/Sequential TB\+PPA/Sequential HVC\) consists of only 78 problems, and verifying effectiveness on larger\-scale datasets remains future work\.

## 8Acknowledgements

We sincerely thank Naoaki Okazaki and Youmi Ma of Institute of Science Tokyo for their valuable guidance, insightful discussions, and constructive feedback throughout this research\. This work was supported by ABCI 3\.0, which is provided by AIST and AIST Solutions\.

## References

- Ajayiet al\.\(2019\)T\. Ajayi, D\. Blaauw, T\.\-B\. Chan, C\.\-K\. Cheng, V\. A\. Chhabria, D\. K\. Choo, M\. Coltella, S\. Dobre, R\. Dreslinski, M\. Fogaça,et al\.OpenROAD: Toward a Self\-Driving, Open\-Source Digital Layout Implementation Tool Chain\.InProceedings of Government Microcircuit Applications and Critical Technology Conference,pp\. 1105–1110\.Cited by:[§1](https://arxiv.org/html/2610.26904#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.26904#S2.SS1.p3.1),[§3\.1](https://arxiv.org/html/2610.26904#S3.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.26904#S4.SS1.SSS0.Px3.p1.1)\.
- Baiet al\.\(2023\)Y\. Bai, A\. Sohrabizadeh, Z\. Qin, Z\. Hu, Y\. Sun, and J\. CongTowards a Comprehensive Benchmark for High\-Level Synthesis Targeted to FPGAs\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 45288–45299\.Cited by:[§2\.2](https://arxiv.org/html/2610.26904#S2.SS2.p1.1)\.
- Chenet al\.\(2025\)Z\. Chen, K\. Chang, Z\. Li, X\. He, C\. Chen, C\. Li, M\. Wang, H\. Xu, Y\. Han, and Y\. WangChipSeek\-R1: Generating Human\-Surpassing RTL with LLM via Hierarchical Reward\-Driven Reinforcement Learning\.arXiv preprint arXiv:2507\.04736\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2507.04736)Cited by:[§1](https://arxiv.org/html/2610.26904#S1.p2.1),[§3\.1](https://arxiv.org/html/2610.26904#S3.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.26904#S4.SS1.SSS0.Px3.p1.1),[§5\.1](https://arxiv.org/html/2610.26904#S5.SS1.p3.1)\.
- Dobešet al\.\(2013\)J\. Dobeš, J\. Michal, and V\. BiolkováMultiobjective Optimization for Electronic Circuit Design in Time and Frequency Domains\.Radioengineering22\(1\),pp\. 136–152\.Cited by:[§1](https://arxiv.org/html/2610.26904#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.26904#S2.SS2.p1.1)\.
- Ettingeret al\.\(2025\)A\. Ettinger, A\. Bertsch, B\. Kuehl, D\. W\. Graham, D\. Heineman, D\. Groeneveld, F\. Brahman, F\. Timbers, H\. Ivison,et al\.OLMo 3\.arXiv preprint arXiv:2512\.13961\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2512.13961),[Link](https://arxiv.org/abs/2512.13961)Cited by:[§1](https://arxiv.org/html/2610.26904#S1.p1.1)\.
- Flakeet al\.\(2020\)P\. Flake, P\. Moorby, S\. Golson, A\. Salz, and S\. DavidmannVerilog HDL and its ancestors and descendants\.Proceedings of the ACM on Programming Languages4\(HOPL\)\.External Links:[Document](https://dx.doi.org/10.1145/3386337)Cited by:[§1](https://arxiv.org/html/2610.26904#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.26904#S2.SS1.p1.1)\.
- Fortinet al\.\(2012\)F\. Fortin, F\. D\. Rainville, M\. Gardner, M\. Parizeau, and C\. GagnéDEAP: evolutionary algorithms made easy\.Journal of Machine Learning Research13,pp\. 2171–2175\.Cited by:[§4\.1](https://arxiv.org/html/2610.26904#S4.SS1.SSS0.Px3.p2.2)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang,et al\.DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[§1](https://arxiv.org/html/2610.26904#S1.p1.1)\.
- Hsiaoet al\.\(2025\)H\. Hsiao, Y\. Lu, S\. K\. Lim, and H\. RenBUFFALO: PPA\-Configurable, LLM\-based Buffer Tree Generation via Group Relative Policy Optimization\.In2025 IEEE/ACM International Conference On Computer Aided Design \(ICCAD\),Vol\.,pp\. 1–9\.External Links:[Document](https://dx.doi.org/10.1109/ICCAD66269.2025.11240744)Cited by:[§1](https://arxiv.org/html/2610.26904#S1.p2.1),[§3\.1](https://arxiv.org/html/2610.26904#S3.SS1.p1.1)\.
- Liu and Carloni \(2013\)H\. Liu and L\. P\. CarloniOn learning\-based methods for design\-space exploration with High\-Level Synthesis\.In2013 50th ACM/EDAC/IEEE Design Automation Conference \(DAC\),Vol\.,pp\. 1–7\.External Links:[Document](https://dx.doi.org/)Cited by:[§1](https://arxiv.org/html/2610.26904#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.26904#S2.SS2.p1.1)\.
- Liuet al\.\(2024\)S\. Liu, Y\. Lu, W\. Fang, M\. Li, and Z\. XieOpenLLM\-RTL: Open Dataset and Benchmark for LLM\-Aided Design RTL Generation \(Invited\)\.InProceedings of the 43rd IEEE/ACM International Conference on Computer\-Aided Design \(ICCAD\),pp\. 1–9\.External Links:[Document](https://dx.doi.org/10.1145/3676536.3697118)Cited by:[§5\.2](https://arxiv.org/html/2610.26904#S5.SS2.p1.1)\.
- Liuet al\.\(2025\)Y\. Liu, H\. Zhang, Y\. Zhou, Z\. Shi, C\. Xu, and Q\. XuDeepRTL2: A Versatile Model for RTL\-Related Tasks\.InFindings of the Association for Computational Linguistics: ACL 2025,Vienna, Austria,pp\. 6485–6500\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.336)Cited by:[§2\.1](https://arxiv.org/html/2610.26904#S2.SS1.p1.1)\.
- Novikovet al\.\(2025\)A\. Novikov, N\. Vũ, M\. Eisenberger, E\. Dupont, P\. Huang, A\. Z\. Wagner, S\. Shirobokov, B\. Kozlovskii, F\. J\. R\. Ruiz, A\. Mehrabian,et al\.AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery\.arXiv preprint arXiv:2506\.13131\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2506.13131)Cited by:[§1](https://arxiv.org/html/2610.26904#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.26904#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2610.26904#S2.SS2.p2.1)\.
- OpenAI \(2024\)OpenAIOpenAI o1 System Card\.Technical reportOpenAI\.External Links:[Link](https://cdn.openai.com/o1-system-card-20241205.pdf)Cited by:[§1](https://arxiv.org/html/2610.26904#S1.p1.1)\.
- OpenAI \(2025\)OpenAIOpenAI o3 and o4\-mini System Card\.Technical reportOpenAI\.External Links:[Link](https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf)Cited by:[§1](https://arxiv.org/html/2610.26904#S1.p1.1)\.
- Shahdad \(1986\)M\. ShahdadAn Overview of VHDL Language and Technology\.In23rd ACM/IEEE Design Automation Conference,Vol\.,pp\. 320–326\.External Links:[Document](https://dx.doi.org/10.1109/DAC.1986.1586107)Cited by:[§2\.1](https://arxiv.org/html/2610.26904#S2.SS1.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\.arXiv preprint arXiv:2402\.03300\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2402.03300)Cited by:[§1](https://arxiv.org/html/2610.26904#S1.p1.1),[§4\.1](https://arxiv.org/html/2610.26904#S4.SS1.SSS0.Px1.p1.1)\.
- Teohet al\.\(2025\)J\. Teoh, P\. Varakantham, and P\. VamplewOn Generalization Across Environments In Multi\-Objective Reinforcement Learning\.InThe Thirteenth International Conference on Learning Representations,Note:ICLR 2025 PosterCited by:[§3\.2](https://arxiv.org/html/2610.26904#S3.SS2.p1.1),[§3\.2](https://arxiv.org/html/2610.26904#S3.SS2.p2.1),[§5\.3](https://arxiv.org/html/2610.26904#S5.SS3.SSS0.Px1.p1.1)\.
- Vişanet al\.\(2022\)C\. Vişan, O\. Pascu, M\. Stănescu, E\. Şandru, C\. Diaconu, A\. Buzo, G\. Pelz, and H\. CucuAutomated Circuit Sizing with Multi\-objective Optimization based on Differential Evolution and Bayesian Inference\.Knowledge\-Based Systems258,pp\. 109987\.External Links:[Document](https://dx.doi.org/10.1016/j.knosys.2022.109987)Cited by:[§1](https://arxiv.org/html/2610.26904#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.26904#S2.SS2.p1.1)\.
- Wanget al\.\(2026\)Y\. Wang, G\. Sun, W\. Ye, G\. Qu, and A\. LiVeriReason: Reinforcement Learning with Testbench Feedback for Reasoning\-Enhanced Verilog Generation\.InGLSVLSI,Cited by:[§1](https://arxiv.org/html/2610.26904#S1.p2.1),[§5\.1](https://arxiv.org/html/2610.26904#S5.SS1.p1.1),[§5\.2](https://arxiv.org/html/2610.26904#S5.SS2.p1.1)\.
- Williams and Baxter \(2002\)S\. Williams and M\. BaxterIcarus Verilog: Open\-Source Verilog More Than a Year Later\.Linux Journal2002\(99\),pp\. 3\.Cited by:[§1](https://arxiv.org/html/2610.26904#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.26904#S2.SS1.p2.1),[§4\.1](https://arxiv.org/html/2610.26904#S4.SS1.SSS0.Px3.p1.1)\.
- Wolf and Glaser \(2013\)C\. Wolf and J\. GlaserYosys \- A Free Verilog Synthesis Suite\.InProceedings of Austrochip 2013,Cited by:[§2\.1](https://arxiv.org/html/2610.26904#S2.SS1.p3.1),[§4\.1](https://arxiv.org/html/2610.26904#S4.SS1.SSS0.Px3.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 Technical Report\.arXiv preprint arXiv:2505\.09388\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.09388),[Link](https://arxiv.org/abs/2505.09388)Cited by:[§1](https://arxiv.org/html/2610.26904#S1.p1.1)\.
- Yueet al\.\(2025\)Y\. Yue, Z\. Chen, R\. Lu, A\. Zhao, Z\. Wang, S\. Song, and G\. HuangDoes reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?\.InNeurIPS,Cited by:[§E\.5](https://arxiv.org/html/2610.26904#A5.SS5.p2.1)\.
- Zhuet al\.\(2026\)Y\. Zhu, D\. Huang, H\. Lyu, X\. Zhang, C\. Li, W\. Shi, Y\. Wu, J\. Mu, J\. Wang, Y\. Zhao,et al\.QiMeng\-CodeV\-R1: Reasoning\-Enhanced Verilog Generation\.InNeurIPS,Cited by:[§1](https://arxiv.org/html/2610.26904#S1.p2.1),[§5\.1](https://arxiv.org/html/2610.26904#S5.SS1.p1.1),[§5\.2](https://arxiv.org/html/2610.26904#S5.SS2.p1.1)\.

## Appendix ADetailed Definition of rHV

This section provides the detailed mathematical definition of relative Hypervolume \(rHV\) and describes the properties of rHV when an objective function degenerates\.

### A\.1Mathematical Definition

Throughout this section, a*solution*xxdenotes a single generated circuit\. Each solutionxxis associated with the three\-dimensional objective vector𝐨⁡\(x\)=\(oP​\(x\),oD​\(x\),oA​\(x\)\)\\mathbf\{o\}\(x\)=\(o\_\{P\}\(x\),o\_\{D\}\(x\),o\_\{A\}\(x\)\)defined in Equation[2](https://arxiv.org/html/2610.26904#S4.E2), whose components are the golden\-normalized power, delay, and area objectives\. We writefi,k​\(x\)f\_\{i,k\}\(x\)for thekk\-th component of this vector evaluated on problemii\(k∈\{1,2,3\}k\\in\\\{1,2,3\\\}\); all three objectives are*maximized*, i\.e\., larger is better\.

Letx1\(m,i,j,l\),x2\(m,i,j,l\),…x^\{\(m,i,j,l\)\}\_\{1\},x^\{\(m,i,j,l\)\}\_\{2\},\\dotsdenote the circuits generated by methodmmon problemiiin trialjjat iterationll\. The cumulative solution setSm,i,j\(ℓ\)S\_\{m,i,j\}^\{\(\\ell\)\}collects all circuits generated by methodmmon problemiiin trialjjfrom the first iteration through iterationℓ\\ell:

Sm,i,j\(ℓ\)=⋃l=1ℓ\{x1\(m,i,j,l\),x2\(m,i,j,l\),…\}S\_\{m,i,j\}^\{\(\\ell\)\}=\\bigcup\_\{l=1\}^\{\\ell\}\\left\\\{x^\{\(m,i,j,l\)\}\_\{1\},x^\{\(m,i,j,l\)\}\_\{2\},\\dots\\right\\\}\(7\)This is the per\-trial solution set whose Hypervolume the numerator of rHV measures\.

To obtain a common reference for problemii, we take the unionUiU\_\{i\}of the cumulative solution sets over all methodsm∈ℳm\\in\\mathcal\{M\}and allntn\_\{t\}trials at the final iterationnitern\_\{\\mathrm\{iter\}\}:

Ui=⋃m∈ℳ⋃j=1ntSm,i,j\(niter\)U\_\{i\}=\\bigcup\_\{m\\in\\mathcal\{M\}\}\\bigcup\_\{j=1\}^\{n\_\{t\}\}S\_\{m,i,j\}^\{\(n\_\{\\mathrm\{iter\}\}\)\}\(8\)
The reference archiveΠi\\Pi\_\{i\}is the non\-dominated set of this union, i\.e\., the pseudo Pareto front used as the denominator of rHV:

Πi=ND⁡\(Ui\)\\Pi\_\{i\}=\\mathrm\{ND\}\(U\_\{i\}\)\(9\)
Because the raw PPA values differ in scale across objectives and problems, each objectivek∈\{1,2,3\}k\\in\\\{1,2,3\\\}is min–max normalized using the extreme values observed inUiU\_\{i\}:

f~i,k​\(x\)=fi,k​\(x\)−qi,klowqi,khigh−qi,klow\+ε\\tilde\{f\}\_\{i,k\}\(x\)=\\frac\{f\_\{i,k\}\(x\)\-q^\{\\mathrm\{low\}\}\_\{i,k\}\}\{q^\{\\mathrm\{high\}\}\_\{i,k\}\-q^\{\\mathrm\{low\}\}\_\{i,k\}\+\\varepsilon\}\(10\)whereqi,klow=minx∈Ui⁡fi,k​\(x\)q^\{\\mathrm\{low\}\}\_\{i,k\}=\\min\_\{x\\in U\_\{i\}\}f\_\{i,k\}\(x\)andqi,khigh=maxx∈Ui⁡fi,k​\(x\)q^\{\\mathrm\{high\}\}\_\{i,k\}=\\max\_\{x\\in U\_\{i\}\}f\_\{i,k\}\(x\)are the per\-objective minimum and maximum over the union, andε=10−20\\varepsilon=10^\{\-20\}avoids division by zero\. For any solution setZZ, we writeZ~\\tilde\{Z\}for the set obtained by mapping every solutionx∈Zx\\in Zto its normalized objective vector\(f~i,1​\(x\),f~i,2​\(x\),f~i,3​\(x\)\)\(\\tilde\{f\}\_\{i,1\}\(x\),\\tilde\{f\}\_\{i,2\}\(x\),\\tilde\{f\}\_\{i,3\}\(x\)\)\. In particular,S~m,i,j\(ℓ\)\\tilde\{S\}\_\{m,i,j\}^\{\(\\ell\)\}andΠ~i\\tilde\{\\Pi\}\_\{i\}are the normalized per\-trial solution set and the normalized reference archive, respectively\.

rHV is then the Hypervolume of the normalized per\-trial solution set divided by that of the normalized reference archive:

rHVm,i,j\(ℓ\)=HV⁡\(S~m,i,j\(ℓ\),𝐫\)HV⁡\(Π~i,𝐫\)\\mathrm\{rHV\}\_\{m,i,j\}^\{\(\\ell\)\}=\\frac\{\\mathrm\{HV\}\\left\(\\tilde\{S\}\_\{m,i,j\}^\{\(\\ell\)\};\\mathbf\{r\}\\right\)\}\{\\mathrm\{HV\}\\left\(\\tilde\{\\Pi\}\_\{i\};\\mathbf\{r\}\\right\)\}\(11\)where the reference point is𝐫=\(−δ,−δ,−δ\)\\mathbf\{r\}=\(\-\\delta,\-\\delta,\-\\delta\)withδ=10−12\\delta=10^\{\-12\}\. The reference point is placed slightly below the origin, rather than at the origin itself, so that rHV remains well\-defined and reduces to a 2\-dimensional relative Hypervolume when an objective degenerates \(Section[A\.2](https://arxiv.org/html/2610.26904#A1.SS2)\)\.

### A\.2Property of rHV When an Objective Degenerates

###### Proposition 1\.

Consider a 3\-objective maximization problem with the normalized reference point

𝐫=\(−δ,−δ,−δ\),δ\>0\.\\mathbf\{r\}=\(\-\\delta,\-\\delta,\-\\delta\),\\qquad\\delta\>0\.Assume that the third objective is constant across all solutions, and after normalization:

f~i,3​\(x\)=0\.\\tilde\{f\}\_\{i,3\}\(x\)=0\.Then, for any point set

P⊂\[0,1\]2×\{0\},P\\subset\[0,1\]^\{2\}\\times\\\{0\\\},the following holds:

HV⁡\(P,\(−δ,−δ,−δ\)\)=δ​HV​\(P12,\(−δ,−δ\)\),\\mathrm\{HV\}\\\!\\left\(P;\(\-\\delta,\-\\delta,\-\\delta\)\\right\)=\\delta\\,\\mathrm\{HV\}\\\!\\left\(P\_\{12\};\(\-\\delta,\-\\delta\)\\right\),where

P12=\{\(u,v\)∣\(u,v,0\)∈P\}P\_\{12\}=\\\{\(u,v\)\\mid\(u,v,0\)\\in P\\\}is the projection onto the first and second objectives\.

###### Proof\.

By the definition of Hypervolume for maximization problems:

HV⁡\(P,𝐫\)=λ3​\(⋃p∈P\[r1,p1\]×\[r2,p2\]×\[r3,p3\]\)\.\\mathrm\{HV\}\(P;\\mathbf\{r\}\)=\\lambda\_\{3\}\\\!\\left\(\\bigcup\_\{p\\in P\}\[r\_\{1\},p\_\{1\}\]\\times\[r\_\{2\},p\_\{2\}\]\\times\[r\_\{3\},p\_\{3\}\]\\right\)\.
Since every point has the form\(u,v,0\)∈P\(u,v,0\)\\in P, each point spans a hyperrectangle:

\[−δ,u\]×\[−δ,v\]×\[−δ,0\]\.\[\-\\delta,u\]\\times\[\-\\delta,v\]\\times\[\-\\delta,0\]\.Therefore:

⋃\(u,v,0\)∈P\[−δ,u\]×\[−δ,v\]×\[−δ,0\]=\(⋃\(u,v\)∈P12\[−δ,u\]×\[−δ,v\]\)×\[−δ,0\]\.\\bigcup\_\{\(u,v,0\)\\in P\}\[\-\\delta,u\]\\times\[\-\\delta,v\]\\times\[\-\\delta,0\]=\\left\(\\bigcup\_\{\(u,v\)\\in P\_\{12\}\}\[\-\\delta,u\]\\times\[\-\\delta,v\]\\right\)\\times\[\-\\delta,0\]\.By the product property of volume:

HV⁡\(P,\(−δ,−δ,−δ\)\)\\displaystyle\\mathrm\{HV\}\\\!\\left\(P;\(\-\\delta,\-\\delta,\-\\delta\)\\right\)=λ2​\(⋃\(u,v\)∈P12\[−δ,u\]×\[−δ,v\]\)⋅δ\\displaystyle=\\lambda\_\{2\}\\\!\\left\(\\bigcup\_\{\(u,v\)\\in P\_\{12\}\}\[\-\\delta,u\]\\times\[\-\\delta,v\]\\right\)\\cdot\\delta\(12\)=δ​HV​\(P12,\(−δ,−δ\)\)\.\\displaystyle=\\delta\\,\\mathrm\{HV\}\\\!\\left\(P\_\{12\};\(\-\\delta,\-\\delta\)\\right\)\.\(13\)∎

Applying this result to both the numerator and denominator:

HV⁡\(S~m,i,j\(ℓ\),𝐫\)=δ​HV​\(\(S~m,i,j\(ℓ\)\)12,\(−δ,−δ\)\),\\mathrm\{HV\}\\\!\\left\(\\tilde\{S\}\_\{m,i,j\}^\{\(\\ell\)\};\\mathbf\{r\}\\right\)=\\delta\\,\\mathrm\{HV\}\\\!\\left\(\(\\tilde\{S\}\_\{m,i,j\}^\{\(\\ell\)\}\)\_\{12\};\(\-\\delta,\-\\delta\)\\right\),HV⁡\(Π~i,𝐫\)=δ​HV​\(\(Π~i\)12,\(−δ,−δ\)\)\.\\mathrm\{HV\}\\\!\\left\(\\tilde\{\\Pi\}\_\{i\};\\mathbf\{r\}\\right\)=\\delta\\,\\mathrm\{HV\}\\\!\\left\(\(\\tilde\{\\Pi\}\_\{i\}\)\_\{12\};\(\-\\delta,\-\\delta\)\\right\)\.Therefore:

rHVm,i,j\(ℓ\)=HV⁡\(\(S~m,i,j\(ℓ\)\)12,\(−δ,−δ\)\)HV⁡\(\(Π~i\)12,\(−δ,−δ\)\)\.\\mathrm\{rHV\}\_\{m,i,j\}^\{\(\\ell\)\}=\\frac\{\\mathrm\{HV\}\\\!\\left\(\(\\tilde\{S\}\_\{m,i,j\}^\{\(\\ell\)\}\)\_\{12\};\(\-\\delta,\-\\delta\)\\right\)\}\{\\mathrm\{HV\}\\\!\\left\(\(\\tilde\{\\Pi\}\_\{i\}\)\_\{12\};\(\-\\delta,\-\\delta\)\\right\)\}\.Therefore, when the third objective takes the same value across all solutions, the third objective appears only as a common factor in both numerator and denominator, and rHV effectively reduces to a 2\-dimensional relative Hypervolume based on the first and second objectives\.

## Appendix BPPA Reward

rPPA=PgoldenPgen×AgoldenAgen×DgoldenDgen\+ϵr\_\{\\text\{PPA\}\}=\\frac\{P\_\{\\text\{golden\}\}\}\{P\_\{\\text\{gen\}\}\}\\times\\frac\{A\_\{\\text\{golden\}\}\}\{A\_\{\\text\{gen\}\}\}\\times\\frac\{D\_\{\\text\{golden\}\}\}\{D\_\{\\text\{gen\}\}\+\\epsilon\}\(14\)wherePP,AA, andDDdenote power, area, and delay, respectively\. The subscript “golden” refers to the ground\-truth circuit and “gen” refers to the generated circuit\.ϵ\\epsilonis a small constant to avoid division by zero\.

## Appendix CPrompt Templates

We show the prompt templates used for multi\-objective reinforcement learning below\.\{spec\}represents the circuit specification,\{circuit\}represents the generated Verilog circuit, and\{ppa\}represents PPA values as placeholders\.

### C\.1System Prompt

thereasoningprocessinthemindandthenprovidestheuser

withtheanswer\.Thereasoningprocessandanswerareenclosed

within<think\></think\>and<answer\></answer\>tags,

respectively\.

Nowtheuserasksyoutowriteverilogcodeasshownin<Spec\>\.

Afterthinking,enclosethefinalverilogcodein“‘verilog“‘

within<answer\></answer\>tags\.

Notethatcircuitssatisfyingthesamespecificationhave

alreadybeenobtained,andyouneedtoderiveaPareto\-optimal

solutionwithdifferentPPAvaluesfromthosecircuits\.The

circuitsobtainedsofarandtheirPPAvaluesarelistedbelow

in<\#Circuithistory\>\.

### C\.2User Prompt \(iteration = 1\)

\{spec\}

\#\#Circuithistory

Nopreviouscircuit\.

### C\.3User Prompt \(iteration≥\\geq2\)

\{spec\}

\#\#Circuithistory

\#\#\#previouscircuit1

“‘verilog

\{circuit\_1\}

“‘

\-PPA:\{ppa\_1\}

\#\#\#previouscircuit2

“‘verilog

\{circuit\_2\}

“‘

\-PPA:\{ppa\_2\}

…

\#\#\#previouscircuiti

“‘verilog

\{circuit\_i\}

“‘

\-PPA:\{ppa\_i\}

## Appendix DHyperparameters

Table[2](https://arxiv.org/html/2610.26904#A4.T2)reports the hyperparameters used for training\.

Table 2:Training hyperparameters\. Common GRPO settings are shared across all methods on both base models; Max Steps differs by base model\.†Applies only to generation\-based training methods \(Sequential TB / Sequential TB\+PPA / Sequential HVC\); not used for Single\-Objective PPA\.

## Appendix EAdditional Analysis

### E\.1Bootstrap Confidence Intervals

To assess statistical reliability, we computed 95% confidence intervals for the rHV differences between Sequential HVC \(Ours\) and each baseline at iter=10 using a nested paired bootstrap\. Specifically, we resampled the 45 problems with replacement and, within each sampled problem, resampled the 30 trials with replacement while preserving paired trial comparisons, and repeated this procedure 10,000 times\. Table[3](https://arxiv.org/html/2610.26904#A5.T3)reports bootstrap confidence intervals \(95%\) of the rHV difference between Sequential HVC \(Ours\) and each baseline at iter=10\.

Table 3:Bootstrap confidence intervals \(95%\) of the rHV difference between Sequential HVC \(Ours\) and each baseline at iter=10\.For both VeriReason\-3B and CodeV\-R1\-Distil\-7B, the 95% confidence intervals of the improvements over all baselines do not cross zero, indicating that the gains are stable even when accounting for both problem\-level and trial\-level variation\.

### E\.2Relative Hypervolume with Respect to the Baseline\-Only Archive

To quantify the effect of including our own outputs in the reference archive, we conduct an out\-of\-self reference analysis: we remove all outputs of Sequential HVC \(Ours\) across all trials and rebuild the reference archive using only the four baseline methods \(Base, Single\-Objective PPA, Sequential TB, Sequential TB\+PPA\), and evaluate rHV with respect to this baseline\-only archive\. However, for this baseline\-only reference archive, the hypervolume of the reference archive becomes extremely small for some problems, so the mean of this ratio\-based rHV is dominated by a few outliers\. We therefore compare methods using the median rHV instead of the mean for this additional analysis\. Table[4](https://arxiv.org/html/2610.26904#A5.T4)reports the median relative Hypervolume \(rHV\) computed with respect to this baseline\-only archive\.

Table 4:Median relative Hypervolume \(rHV\) computed with respect to the baseline\-only archive\. Each column shows the median rHV after a given number of sequential generation iterations \(iter\)\.Sequential HVC \(Ours\) still achieves the highest median rHV with respect to the baseline\-only archive for both the VeriReason\-3B and CodeV\-R1\-Distil\-7B bases, confirming that the original conclusion is not merely an artifact of including Sequential HVC in the reference archive\.

### E\.3Objective\-Space Coverage

To gain a more intuitive understanding of how each method covers the Pareto front, we visualize the solutions generated by each model at test time as two\-dimensional projections onto pairs of objective axes\. Figure[5](https://arxiv.org/html/2610.26904#A5.F5)shows the resulting scatter plots for each method\. Because all three objectives are normalized so that larger values are better \(Equation[2](https://arxiv.org/html/2610.26904#S4.E2)\), solutions lying toward the upper\-right region of each projection are more desirable; a method is therefore preferable when its solutions are both widely spread and pushed toward this region\. The visualization confirms that the proposed method \(Sequential HVC\) distributes diverse solutions across the desirable upper\-right region, complementing the aggregated rHV results reported in the main text\.

![Refer to caption](https://arxiv.org/html/2610.26904v1/imgs/scatter_plot/scatter_plot.verireason.png)\(a\)VeriReason\-3B
![Refer to caption](https://arxiv.org/html/2610.26904v1/imgs/scatter_plot/scatter_plot.codev-r1.png)\(b\)CodeV\-R1\-Distil\-7B

Figure 5:Scatter plots of the generated solutions in the objective space for each base model\. Visualizing the solutions in the objective space clarifies how each method covers different trade\-off regions of the Pareto front\. Sequential HVC \(Ours\) attains a broader coverage of the objective space than the baselines\.
### E\.4Repetition Penalty Ablation

The Sequential HVC reward encourages the model to generate circuits that contribute to the Pareto front, which also has the side effect of discouraging duplicate generations\. It is therefore not obvious whether the gains of Sequential HVC come from directly optimizing the Pareto\-front contribution or simply from learning to avoid repeated generations and produce unique circuits\. To disentangle these two effects, we run an ablation that adds an explicit repetition penalty to the sequential baseline and test whether avoiding duplicates alone yields a comparable improvement\. Figure[6](https://arxiv.org/html/2610.26904#A5.F6)reports the Win Rate and circuit diversity of the sequential baseline with and without the repetition penalty\. While the repetition penalty substantially suppresses the generation of duplicate circuits, it still underperforms Sequential HVC in terms of both rHV and Win Rate\. This indicates that avoiding repetition alone is not sufficient for effective Pareto\-front exploration, and that directly optimizing the Pareto\-front contribution through the HVC reward is crucial\.

![Refer to caption](https://arxiv.org/html/2610.26904v1/imgs/rep_penalty/rep_penalty.rhv.png)\(a\)Relative Hypervolume \(rHV\)
![Refer to caption](https://arxiv.org/html/2610.26904v1/rep_penalty.win_rate.png)\(b\)Win Rate
![Refer to caption](https://arxiv.org/html/2610.26904v1/rep_penalty.circuit_diversity.png)\(c\)Circuit Diversity

Figure 6:Effect of adding an explicit repetition penalty to the sequential baseline, evaluated in terms of relative Hypervolume \(top\), Win Rate \(bottom left\), and circuit diversity \(bottom right\)\. Although the repetition penalty reduces duplicate generations, it does not match the improvement of Sequential HVC, indicating that the gains of our method arise from directly optimizing the Pareto\-front contribution rather than merely avoiding duplicates\.
### E\.5Relative Hypervolume of the Union across Trials

The main results evaluate rHV on a per\-trial basis\. To assess the aggregate exploration capability of each method, we additionally evaluate rHV after taking the union of the solutions generated across multiple trials for each method\. Figure[7](https://arxiv.org/html/2610.26904#A5.F7)reports the resulting rHV, providing a view of how much of the Pareto front each method can cover when its independent trials are combined\.

Figure[7](https://arxiv.org/html/2610.26904#A5.F7)arranges the results in a grid of bar charts, with rows corresponding to the number of iterations \(1, 5, and 10\) and columns corresponding to the number of unioned trialsNtrials∈\{1,5,10,15,20,25,30\}N\_\{\\mathrm\{trials\}\}\\in\\\{1,5,10,15,20,25,30\\\}; each subplot compares the rHV of the five methods as bars\. For both VeriReason\-3B and CodeV\-R1\-Distil\-7B, Sequential HVC shows an advantage whenNtrialsN\_\{\\mathrm\{trials\}\}is small\. For the CodeV\-R1\-Distil\-7B experiments, even atNtrials=30N\_\{\\mathrm\{trials\}\}=30Sequential HVC achieves higher rHV than all methods except the Base model, to which it remains comparable, consistent with the phenomenon reported by[Yue et al\. \(2025\)](https://arxiv.org/html/2610.26904#bib.bib25), where RL\-trained models improve small\-kkpass@kkperformance while the base model catches up askkgrows\. Overall, the main contribution of our method is not to dominate the Base model under an extremely large archive budget, but to obtain diverse and high\-quality circuits efficiently with a smaller inference budget\. This matters for practical use cases where designers want to quickly obtain diverse candidate circuits, and for downstream inference\-time search or evolutionary algorithms that benefit from efficient generation of promising initial candidates\.

![Refer to caption](https://arxiv.org/html/2610.26904v1/imgs/union/rhv_union_grid_uncapped.verireason.png)\(a\)VeriReason\-3BFigure 7:Relative Hypervolume \(rHV\) evaluated on the union of solutions generated across multiple trials for each method\. Aggregating the solutions from independent trials measures the overall Pareto\-front coverage attainable by each method\. Sequential HVC \(Ours\) retains its advantage over the baselines for smaller numbers of unioned trials\. The results for CodeV\-R1\-Distil\-7B are shown on the following page\.![Refer to caption](https://arxiv.org/html/2610.26904v1/imgs/union/rhv_union_grid_uncapped.codev-r1.png)\(a\)CodeV\-R1\-Distil\-7BFigure 8:\(continued\) Relative Hypervolume \(rHV\) evaluated on the union of solutions generated across multiple trials for each method\.