ParBench: A Benchmark for Reliable Evaluation of LLM Parallel Code Translation
Summary
ParBench is a benchmark framework for evaluating LLM-based parallel API translation across CUDA, OpenMP, OpenCL, and OpenMP target offload, focusing on executable, reproducible conditions and robustness testing.
View Cached Full Text
Cached at: 07/28/26, 06:25 AM
# ParBench: A Benchmark for Reliable Evaluation of LLM Parallel Code Translation Source: [https://arxiv.org/abs/2607.22588](https://arxiv.org/abs/2607.22588) [View PDF](https://arxiv.org/pdf/2607.22588) > Abstract:Modern compute\-intensive software must migrate across a changing ecosystem of accelerators, programming APIs, compiler stacks, and portability layers, including CUDA, OpenMP, OpenCL, and OpenMP target offload\. Large language models and autonomous coding agents are increasingly proposed for such migration, but the field lacks reliable ways to measure whether they preserve the low\-level parallel semantics that make translations behaviorally valid, including thread indexing, synchronization, memory management, host\-device coordination, and API\-specific execution structure\. We present ParBench, a kernel\-centric benchmark framework for evaluating LLM\-based parallel API translation under executable, reproducible conditions\. ParBench fixes the surrounding build, run, and verification infrastructure through declarative benchmark specifications and asks models to translate only the computational kernels\. It draws on multiple open\-source HPC suites and covers representative cross\-API translation directions among CUDA, OpenMP, OpenCL, and OpenMP target offload\. To test whether success reflects robust translation rather than surface\-form memorization, ParBench includes AST\-driven, intended behavior\-preserving, baseline\-validated source augmentation\. Evaluations on state\-of\-the\-art open and proprietary LLMs show persistent barriers to reliable parallel code translation, including direction asymmetry, multi\-file coordination, incomplete API adaptation, and uneven robustness to source\-level perturbations\. Code is available at[this https URL](https://github.com/Scientific-Computing-Lab/ParBench)\. ## Submission history From: Gal Oren \[[view email](https://arxiv.org/show-email/fdaccbc7/2607.22588)\] **\[v1\]**Tue, 9 Jun 2026 19:00:09 UTC \(841 KB\)
Similar Articles
KernelBench-X: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels
KernelBench-X is a new benchmark for evaluating LLM-generated GPU kernels, revealing that task structure impacts correctness more than method design and that correctness does not guarantee hardware efficiency.
RepBench: Compiling Benchmarks into Capability Representations for Large Language Models
RepBench is a benchmark-grounded data layer for representation probing in LLMs, built from 13,427 benchmark papers and 353 public datasets, yielding 46,149 probe texts across 94 capabilities. It evaluates capability vector transfer across models and readout methods.
Benchmarking LLMs
A study or report on benchmarking large language models, likely comparing performance across various tasks.
Homebench – Benchmark local LLMs for speed, memory, and quality
Homebench is a zero-config terminal tool that benchmarks locally-run LLMs for speed, memory, and quality, presenting a live leaderboard. It supports Ollama, LM Studio, llama.cpp, vLLM, and OpenAI-compatible servers.
ProgramBench (5 minute read)
ProgramBench is a new benchmark that evaluates AI agents' ability to reconstruct complete software projects from compiled binaries and documentation without access to source code or decompilation tools.