Natural Language Code Retrieval for 1C:Enterprise: An Open Benchmark and Efficient Bi-Encoder

# Natural Language Code Retrieval for 1C:Enterprise: An Open Benchmark and Efficient Bi-Encoder
Source: [https://arxiv.org/html/2608.19957](https://arxiv.org/html/2608.19957)
Konstantin Chesnokov\*OrcID:[0009\-0007\-0162\-9344](https://orcid.org/0009-0007-0162-9344)Affiliation:Independent Researcher, Moscow, Russia \*Corresponding authorE\-mail[konstphx@gmail\.com](mailto:[email protected])

###### Abstract

Natural language code retrieval is a rapidly evolving task in computer science\. However, the 1C:Enterprise ecosystem combines Russian syntax with highly domain\-specific terminology, for which open datasets and specialized models have been virtually non\-existent\. We present a comprehensive pipeline for 1C code retrieval: an open benchmark of 3,413 real\-world, PII\-scrubbed query–code pairs, a reproducible evaluation harness, and a specialized bi\-encoder\. To overcome scarce labeled data, we fine\-tune on 784,057 synthetic triplets generated by[google/gemma\-4\-26B\-A4B\-it](https://huggingface.co/google/gemma-4-26B-A4B-it)from public code repositories, using Matryoshka Representation Learning \(MRL\) and a privacy\-aware tokenizer\. Because the benchmark subsets differ in size, we report balanced\-subset macro, query\-weighted micro, andforum\-only results\. Our model reaches 0\.5992 balanced macro nDCG@10, 0\.5044 micro, and 0\.4617 onforum, versus 0\.4932 macro for the baseline architecture and 0\.5404 for[google/embeddinggemma\-300m](https://huggingface.co/google/embeddinggemma-300m)\. Removing every benchmark example flagged by the conservative exact/13\-gram overlap audit leaves 0\.6011 balanced macro \(0\.5010 micro\), indicating that detected train–benchmark overlap does not explain the headline result\. MRL truncation to 256 dimensions preserves 99\.9% of retrieval quality while reducing dense\-index storage and exact similarity arithmetic by a factor of three\.

###### Keywords:

Code retrieval 1C:Enterprise BSL Dense retrieval Benchmark Matryoshka embeddings

## 1Introduction

Natural\-language code retrieval ranks code fragments in response to a text query\. For popular languages, sizeable datasets and trained retrievers exist; for 1C:Enterprise, open retrieval infrastructure has been scarce despite its role in Russian\-speaking enterprise software\.

The 1C domain differs from typical English\-language code search\. Queries are often Russian and may describe errors, platform objects, reports, registries, or accounting operations; relevant documents are BSL or 1C query fragments that mix Cyrillic keywords, Russian identifiers, SQL\-like clauses, and business terminology\. Without a domain benchmark, it is hard to tell whether general multilingual embeddings suffice or domain adaptation is needed\.

We formulate the task as single\-stage dense closed\-set retrieval, where queries and documents are represented by learned continuous vectors and ranked by vector similarity\. Given a corpusD=\{d1,…,dn\}D=\\\{d\_\{1\},\\ldots,d\_\{n\}\\\}and queryqq, the system ranks documents so that the labeled relevant fragmentd\+d^\{\+\}ranks as high as possible\. A bi\-encoder embeds queries and documents independently:

𝐞q\\displaystyle\\mathbf\{e\}\_\{q\}=Enc⁡\(q,search\_query\),\\displaystyle=\\mathrm\{Enc\}\\bigl\(q;\\;\\texttt\{search\\\_query\}\\bigr\),\(1\)𝐞d\\displaystyle\\mathbf\{e\}\_\{d\}=Enc⁡\(d,search\_document\),\\displaystyle=\\mathrm\{Enc\}\\bigl\(d;\\;\\texttt\{search\\\_document\}\\bigr\),and scores them by cosine similarity \(an inner product after L2 normalization\):

score⁡\(q,d\)=cos⁡\(𝐞q,𝐞d\)\.\\mathrm\{score\}\(q,d\)=\\cos\(\\mathbf\{e\}\_\{q\},\\mathbf\{e\}\_\{d\}\)\.\(2\)
This work establishes a starting point for 1C retrieval\. We do not propose a new architecture or loss; we contribute a benchmark, evaluation protocol, training resource, contamination audit, and reference domain model\.

The main contributions are:

1. 1\.[PruhaNLP/1C\-Ebench](https://huggingface.co/datasets/PruhaNLP/1C-Ebench), an open benchmark for retrieving 1C/BSL code from Russian questions \(3,413 pairs; subsetsforumandfastcode\)\.
2. 2\.[PruhaNLP/1C\-Code\-Train](https://huggingface.co/datasets/PruhaNLP/1C-Code-Train), 784,057 synthetic triplets\(q,d\+,d−\)\(q,d^\{\+\},d^\{\-\}\)with PII scrubbing\.
3. 3\.[PruhaNLP/1C\-RB](https://github.com/PruhaNLP/1C-RB), a reproducible evaluation harness for dense models and BM25Okapi\.
4. 4\.A train–benchmark contamination audit \(exact match and 13\-gram\); no exact duplicate pairs\.
5. 5\.A reference baseline suite and[PruhaNLP/USER2\-1C\-code](https://huggingface.co/PruhaNLP/USER2-1C-code), a domain\-adapted bi\-encoder with Matryoshka truncation that improves over[deepvk/USER2\-base](https://huggingface.co/deepvk/USER2-base)and strong multilingual embeddings\.

## 2Related Work

### 2\.1Code Retrieval Benchmarks

CodeSearchNet\[[12](https://arxiv.org/html/2608.19957#bib.bib1)\]introduced a large corpus and challenge for semantic code search over six mainstream languages\. CoSQA\[[10](https://arxiv.org/html/2608.19957#bib.bib2)\]moved closer to real search with web queries\. CodeXGLUE\[[19](https://arxiv.org/html/2608.19957#bib.bib3)\]aggregated program\-understanding tasks, including code search\. CoIR\[[18](https://arxiv.org/html/2608.19957#bib.bib4)\]covers text\-to\-code, code\-to\-code, and hybrid retrieval, while CodeRAG\-Bench\[[25](https://arxiv.org/html/2608.19957#bib.bib5)\]evaluates retrieval for code generation\. These resources cover widely used languages, but not 1C:Enterprise or Russian questions to BSL/1C query code\.

### 2\.21C:Enterprise Evaluation Resources

Open 1C evaluation has focused mainly on generation rather than retrieval\. The 1C Code Bench leaderboard\[[8](https://arxiv.org/html/2608.19957#bib.bib6)\]evaluates BSL function generation with compilation and correctness metrics; PRISM/GenLab\-1C develops executable BSL evaluation along similar lines\[[7](https://arxiv.org/html/2608.19957#bib.bib7)\]\. These resources are complementary to[PruhaNLP/1C\-Ebench](https://huggingface.co/datasets/PruhaNLP/1C-Ebench): they ask whether a model can*write*1C code, whereas we ask whether an embedding model can*find*a relevant fragment from a Russian query\.

### 2\.3Multilingual and Cross\-lingual Retrieval Context

Non\-English software\-engineering resources are increasingly needed as open\-source collaboration becomes more multilingual\[[2](https://arxiv.org/html/2608.19957#bib.bib22)\]\.[PruhaNLP/1C\-Ebench](https://huggingface.co/datasets/PruhaNLP/1C-Ebench)is not a classic cross\-language information retrieval \(CLIR\) benchmark, because queries and much of the code vocabulary are Russian or Cyrillic\-heavy rather than two separated natural languages; it still shares CLIR’s alignment problem of matching intent in one form to artifacts in another\[[9](https://arxiv.org/html/2608.19957#bib.bib23)\]\. Downstream, domain code retrievers can help retrieval\-augmented code generation\[[28](https://arxiv.org/html/2608.19957#bib.bib25)\];[PruhaNLP/1C\-Ebench](https://huggingface.co/datasets/PruhaNLP/1C-Ebench)can serve as a first\-stage testbed for 1C\. Generative retrieval and multilingual semantic compression\[[11](https://arxiv.org/html/2608.19957#bib.bib24)\]are complementary future directions\.

### 2\.4Dense Retrieval, BM25, and Compact Embeddings

Dense passage retrieval popularized scalable dual encoders\[[14](https://arxiv.org/html/2608.19957#bib.bib10)\]\. BM25 remains a strong lexical baseline when exact API names or error strings matter\[[22](https://arxiv.org/html/2608.19957#bib.bib11)\]\. Hard negatives from top\-ranked candidates strengthen dense training\[[26](https://arxiv.org/html/2608.19957#bib.bib12),[21](https://arxiv.org/html/2608.19957#bib.bib13)\]; code\-search work also studies false\-negative risk\[[17](https://arxiv.org/html/2608.19957#bib.bib14),[6](https://arxiv.org/html/2608.19957#bib.bib15)\]\. Hybrid BM25\+dense fusion via RRF\[[4](https://arxiv.org/html/2608.19957#bib.bib16)\]helps only when the lexical channel is complementary\.

### 2\.5Synthetic Queries and LLM\-as\-a\-Judge

When labeled query–code pairs are scarce, training often relies on synthetic queries, as in parts of CoIR\. Quality is commonly screened with LLM\-as\-a\-judge protocols\[[27](https://arxiv.org/html/2608.19957#bib.bib18)\]\. Strong judges can agree with humans after bias mitigation on chat tasks, but this does not imply expert agreement in specialized domains; we therefore treat LLM\-judge scores as an approximate signal, not ground truth\.

## 3Benchmark and Evaluation Protocol

### 3\.1Resource Stack

Table[1](https://arxiv.org/html/2608.19957#S3.T1)summarizes the resource stack\. The public benchmark, evaluation harness, and model are linked through their canonical Hugging Face and GitHub identifiers\. They enable comparisons without access to private 1C systems\. Provenance and upstream license metadata are preserved; we make no stronger claim about compatibility of all upstream licenses and exclude proprietary 1C platform distributions\.

Table 1:Public resource stack\.
### 3\.2The[PruhaNLP/1C\-Ebench](https://huggingface.co/datasets/PruhaNLP/1C-Ebench)Benchmark

[PruhaNLP/1C\-Ebench](https://huggingface.co/datasets/PruhaNLP/1C-Ebench)is built from real 1C questions and solutions rather than synthetic queries\. The main source is the public Hugging Face datasetarefaste/1C\_Forums\(19,041 records\)\[[1](https://arxiv.org/html/2608.19957#bib.bib8)\]\. We extract Markdown code fences fromsolutionand apply precision\-first filters for BSL or 1C query fragments\.

The benchmark\-construction branch in Figure[1](https://arxiv.org/html/2608.19957#S3.F1)summarizes the preparation pipeline: source records are normalized, candidate code is extracted, precision\-first validity filters and exact deduplication are applied, and the remaining examples undergo privacy processing\. In code, PII is replaced by placeholders such as\[EMAIL\],\[PHONE\],\[IP\],\[PATH\],\[PERSON\], and\[REDACTED\]\. Questions are not edited: residual PII triggers removal of the example\. The scrubber is a rule\-based, precision\-oriented risk\-reduction step over common surface forms, not a formal de\-identification guarantee; we do not report PII detection precision/recall or a residual manual privacy audit\.

The final benchmark has 3,413 examples in two subsets \(Table[2](https://arxiv.org/html/2608.19957#S3.T2)\)\. Each example hasid,question, andcode\. Within a subset, the corpus is allcodefields and relevance matchesid\. The query relevance judgments \(qrels\) assign one binary gold document to each query\. Thus, the protocol is closed\-set retrieval with a single gold; this can underestimate quality when an unannotated alternative ranks above gold\.

Table 2:[PruhaNLP/1C\-Ebench](https://huggingface.co/datasets/PruhaNLP/1C-Ebench)subsets\.
### 3\.3Evaluation Harness

[PruhaNLP/1C\-RB](https://github.com/PruhaNLP/1C-RB)loads[PruhaNLP/1C\-Ebench](https://huggingface.co/datasets/PruhaNLP/1C-Ebench)and buildscorpus,queries, andqrels\. The dense\-retrieval branch in Figure[1](https://arxiv.org/html/2608.19957#S3.F1)shows the complete evaluation pipeline: asymmetric query/document prompting, independent bi\-encoder inference, L2 normalization, exact cosine ranking over the subset corpus, and computation of nDCG@kk, Recall@kk, and MRR@kk\.

The headline metric is nDCG@10 for binary single\-relevant qrels\. Recall@kkis 1 iffd\+d^\{\+\}is in the topkk; MRR@kkis1/rank⁡\(d\+\)1/\\mathrm\{rank\}\(d^\{\+\}\)if that rank is≤k\\leq k, else 0\. Letsforums\_\{\\mathrm\{forum\}\}andsfastcodes\_\{\\mathrm\{fastcode\}\}denote the subset scores\. The balanced\-subset macro average assigns equal weight to the two retrieval regimes,macro=12​\(sforum\+sfastcode\)\\mathrm\{macro\}=\\tfrac\{1\}\{2\}\(s\_\{\\mathrm\{forum\}\}\+s\_\{\\mathrm\{fastcode\}\}\), while micro\-average weights by query volume \(nforum=2883n\_\{\\mathrm\{forum\}\}=2883,nfastcode=530n\_\{\\mathrm\{fastcode\}\}=530\)\. Thus, macro intentionally givesfastcode50% weight despite its 15\.5% query share; it estimates regime\-balanced rather than random\-query performance\. We report query\-weighted micro and per\-subset scores alongside macro\.

Figure 1:Construction of the public[PruhaNLP/1C\-Ebench](https://huggingface.co/datasets/PruhaNLP/1C-Ebench)benchmark and the dense\-retrieval evaluation pipeline implemented in[PruhaNLP/1C\-RB](https://github.com/PruhaNLP/1C-RB)\.
### 3\.4Train–Benchmark Contamination Audit

[PruhaNLP/1C\-Code\-Train](https://huggingface.co/datasets/PruhaNLP/1C-Code-Train)and[PruhaNLP/1C\-Ebench](https://huggingface.co/datasets/PruhaNLP/1C-Ebench)come from different sources: synthetic LLM queries over public GitHub 1C code versus real forum/FastCode questions\. We still audit overlap with criteria from contamination studies\. Dodge et al\.\[[5](https://arxiv.org/html/2608.19957#bib.bib19)\]distinguish input\-only, label\-only, and input\-and\-label matches after normalization; Brown et al\.\[[3](https://arxiv.org/html/2608.19957#bib.bib20)\]use a conservative wordNN\-gram dirty flag\. Our protocol applies NFKC, lowercasing, whitespace collapsing, exact matching on code/question/pairs, and 13\-gram dirty flags \(Table[3](https://arxiv.org/html/2608.19957#S3.T3)\)\.

Table 3:Train–benchmark overlap audit\.Under the strict exact\-pair criterion, no train–benchmark duplicate pairs remain, andfastcodehas no exact code matches\. Its 22\.83% dirty rate decomposes into 14\.15% label\-only, 6\.60% input\-only, and 2\.08% input\-and\-label cases\. All 46 exact\-question matches infastcodecontain at most 12 words and are generic catalog\-style titles, such as “Copy an array” or “Path to a file\.” For texts shorter than 13 tokens, the conservative 13\-gram criterion compares the entire text; consequently, the 8\.68% question 13\-gram rate identifies the same short questions as exact matching rather than an additional disjoint set\. The higherfastcodeoverlap therefore largely reflects short query templates and recurring BSL boilerplate\.

## 4Domain Adaptation:[PruhaNLP/USER2\-1C\-code](https://huggingface.co/PruhaNLP/USER2-1C-code)

### 4\.1Model

[PruhaNLP/USER2\-1C\-code](https://huggingface.co/PruhaNLP/USER2-1C-code)is a fine\-tuned[deepvk/USER2\-base](https://huggingface.co/deepvk/USER2-base)sentence\-transformers bi\-encoder with mean pooling and cosine scoring\. Asymmetric prompts use different role\-specific templates for queries and documents\. Their names aresearch\_queryandsearch\_document\(Table[4](https://arxiv.org/html/2608.19957#S4.T4)\)\.

We selected[deepvk/USER2\-base](https://huggingface.co/deepvk/USER2-base)because it combines a strong Russian retrieval prior with practical fine\-tuning\. The 149M\-parameter model, developed by DeepVK as a Universal Sentence Encoder for Russian, is based on[deepvk/RuModernBERT\-base](https://huggingface.co/deepvk/RuModernBERT-base)and supports contexts up to 8,192 tokens\. Its published training pipeline comprises retrieval\-oriented RetroMAE pretraining, weakly supervised English–Russian transfer, 50 million pairs mined from the Russiancultura\_ru\_educorpus, and supervised tuning on 4\.3 million examples that include multiple Russian retrieval datasets\. Its moderate size and sentence\-transformers interface make domain adaptation substantially more tractable than fine\-tuning a large generative model\.

Table 4:[PruhaNLP/USER2\-1C\-code](https://huggingface.co/PruhaNLP/USER2-1C-code)configuration\.For anonymized data we apply a privacy\-aware tokenizer patch:\[PATH\]/\[PERSON\]map to\[unused0\]/\[unused1\]; email/phone/IP placeholders use special USER2 tokens; new\-token embeddings are initialized from the mean of corresponding subtokens\.

### 4\.2Training Data

With no open supervised 1C retrieval set available, we build[PruhaNLP/1C\-Code\-Train](https://huggingface.co/datasets/PruhaNLP/1C-Code-Train)as a synthetic closed loop from public code to contrastive triplets \(Figure[2](https://arxiv.org/html/2608.19957#S4.F2)\)\. The design goal is coverage of both intent\-style Russian questions and lexical API lookup, without turning the generator into a verbatim copier of identifiers\.

Figure 2:Construction of[PruhaNLP/1C\-Code\-Train](https://huggingface.co/datasets/PruhaNLP/1C-Code-Train): from filtered public 1C code to synthetic queries, hard\-negative triplets, PII scrubbing, and fine\-tuning of[PruhaNLP/USER2\-1C\-code](https://huggingface.co/PruhaNLP/USER2-1C-code)\.Prompting is treated as a controlled sampling axis rather than a single fixed template: length and style hints diversify surface form, while a separate API\-name mode deliberately allows identifier copying for symbol search\. The remaining stages—hard\-negative mining, privacy scrubbing, and USER2\-style fine\-tuning—are standard dense\-retrieval engineering; Figure[2](https://arxiv.org/html/2608.19957#S4.F2)makes their order explicit\.

The source is the public Hugging Face corpusleongl/1c\_github\(3,038,637 lines\)\[[16](https://arxiv.org/html/2608.19957#bib.bib9)\]; after filtering short, comment\-only, and duplicate fragments, 784,058 unique documents remain\. For each document,[google/gemma\-4\-26B\-A4B\-it](https://huggingface.co/google/gemma-4-26B-A4B-it)emits a Russian query as structured output\. We mine hard negatives with[deepvk/USER2\-small](https://huggingface.co/deepvk/USER2-small)using FAISSIndexFlatIP\[[13](https://arxiv.org/html/2608.19957#bib.bib27)\]\. Specifically,d−=arg⁡maxd∈Top​\-​20​\(q\)∖\{d\+\}⁡cos⁡\(𝐞q,𝐞d\)d^\{\-\}=\\arg\\max\_\{d\\in\\mathrm\{Top\\text\{\-\}20\}\(q\)\\setminus\\\{d^\{\+\}\\\}\}\\cos\(\\mathbf\{e\}\_\{q\},\\mathbf\{e\}\_\{d\}\)\. Top hard negatives are standard\[[14](https://arxiv.org/html/2608.19957#bib.bib10),[26](https://arxiv.org/html/2608.19957#bib.bib12),[21](https://arxiv.org/html/2608.19957#bib.bib13)\]but risk false negatives from near duplicates or alternative solutions\[[21](https://arxiv.org/html/2608.19957#bib.bib13),[17](https://arxiv.org/html/2608.19957#bib.bib14)\]; we do not estimate the FN rate in[PruhaNLP/1C\-Code\-Train](https://huggingface.co/datasets/PruhaNLP/1C-Code-Train)\. These two counts are not the same quantity: 784,058 is the filtered unique\-document corpus, while 784,057 is the number of retained\(q,d\+,d−\)\(q,d^\{\+\},d^\{\-\}\)triplets after synthetic query generation \(one filtered document does not yield a retained pair\)\. PII anonymization changes 2,488 cells \(≈0\.106%\\approx 0\.106\\%\)\.

### 4\.3Synthetic Query Quality

We score a random sample of 300 triplets \(seed 42\) withgoogle/gemini\-3\.1\-pro\-previewvia OpenRouter\[[20](https://arxiv.org/html/2608.19957#bib.bib28)\], chosen because Gemini 3 Pro is strong on 1C Code Bench generation—an indirect, domain\-relevant argument that is not the same as developer agreement on query quality\. The judge rates relevance, naturalness, and clarity from 1 to 5 \(Table[5](https://arxiv.org/html/2608.19957#S4.T5)\); we treat the scores as an approximate usability signal, not human validation\.

Table 5:LLM\-as\-a\-judge scores for 300 synthetic query–code pairs\.
### 4\.4Objective and Hyperparameters

We follow the training recipe of[deepvk/USER2\-base](https://huggingface.co/deepvk/USER2-base)\. The objective isℒ=MatryoshkaLoss⁡\(CachedMNRL\)\\mathcal\{L\}=\\mathrm\{MatryoshkaLoss\}\(\\mathrm\{CachedMNRL\}\): InfoNCE\[[24](https://arxiv.org/html/2608.19957#bib.bib26)\]with one hard negative and in\-batch negatives, summed over MRL prefixes\{768,512,384,256,128,64,32\}\\\{768,512,384,256,128,64,32\\\}\. We keep this recipe for USER2 compatibility and Matryoshka truncation; the domain gain below is attributed to fine\-tuning on[PruhaNLP/1C\-Code\-Train](https://huggingface.co/datasets/PruhaNLP/1C-Code-Train), not to a novel loss \(Table[6](https://arxiv.org/html/2608.19957#S4.T6)\)\.

Table 6:Training hyperparameters\.

## 5Experimental Setup

##### Dense Models\.

We compare sentence\-transformers\-compatible models with fixed prompts \(Table[7](https://arxiv.org/html/2608.19957#S5.T7)\) under one[PruhaNLP/1C\-RB](https://github.com/PruhaNLP/1C-RB)protocol, without exhaustive per\-baseline tuning\.

Table 7:Dense baselines and prompts\. Empty means no prompt prefix\.
##### Lexical Baseline\.

BM25Okapi usesk1=1\.2k\_\{1\}=1\.2,b=0\.75b=0\.75, and identical Unicode tokenizationre\.findall\(r"\\w\+", text\.lower\(\)\)for queries and documents, with exact ranking over each subset corpus in[PruhaNLP/1C\-RB](https://github.com/PruhaNLP/1C-RB)\.

##### Hybrid RRF\.

We fuse BM25 and dense top\-100 ranks with Reciprocal Rank Fusion \(kRRF=60k\_\{\\mathrm\{RRF\}\}=60\):RRF⁡\(q,d\)=∑m∈\{BM25,dense\}1/\(kRRF\+rankm​\(q,d\)\)\\mathrm\{RRF\}\(q,d\)=\\sum\_\{m\\in\\\{\\mathrm\{BM25\},\\,\\mathrm\{dense\}\\\}\}1/\(k\_\{\\mathrm\{RRF\}\}\+\\mathrm\{rank\}\_\{m\}\(q,d\)\)\.

## 6Results

### 6\.1Main Leaderboard

Figure[3](https://arxiv.org/html/2608.19957#S6.F3)and Table[8](https://arxiv.org/html/2608.19957#S6.T8)show the main nDCG@10 leaderboard\.[PruhaNLP/USER2\-1C\-code](https://huggingface.co/PruhaNLP/USER2-1C-code)obtains the best balanced\-subset macro, 0\.5992, while also ranking first under query\-weighted micro \(0\.5044\) and onforumalone \(0\.4617\)\. Its macro gain is 0\.106 over[deepvk/USER2\-base](https://huggingface.co/deepvk/USER2-base)and 0\.0588 over[google/embeddinggemma\-300m](https://huggingface.co/google/embeddinggemma-300m)\.

Figure 3:nDCG@10 leaderboard on[PruhaNLP/1C\-Ebench](https://huggingface.co/datasets/PruhaNLP/1C-Ebench)\(balanced\-subset macro\)\.Table 8:nDCG@10 leaderboard on[PruhaNLP/1C\-Ebench](https://huggingface.co/datasets/PruhaNLP/1C-Ebench)\.BM25Okapi lags strong dense models, suggesting lexical overlap is insufficient for many Russian\-to\-BSL queries\. Untuned RRF hurts both dense systems, reducing balanced macro nDCG@10 from 0\.5992 to 0\.4300 for[PruhaNLP/USER2\-1C\-code](https://huggingface.co/PruhaNLP/USER2-1C-code)and from 0\.4932 to 0\.3908 for[deepvk/USER2\-base](https://huggingface.co/deepvk/USER2-base), with the largest drop onfastcode\. A likely cause is weak lexical complementarity under naive\\w\+tokenization, which poorly isolates BSL identifiers and error strings\. These results are for the tested untuned RRF setup and do not rule out stronger hybrids\.

Table[9](https://arxiv.org/html/2608.19957#S6.T9)gives full @10 metrics for[PruhaNLP/USER2\-1C\-code](https://huggingface.co/PruhaNLP/USER2-1C-code)and BM25\.forumis harder: longer contextual questions and single\-gold qrels can penalize unannotated alternatives\.

Table 9:Full @10 metrics for the reference dense model and BM25\.
### 6\.2Controlled Comparisons

#### Domain Adaptation\.

Comparing[deepvk/USER2\-base](https://huggingface.co/deepvk/USER2-base)and[PruhaNLP/USER2\-1C\-code](https://huggingface.co/PruhaNLP/USER2-1C-code)under the same architecture and prompts yields \+0\.106 balanced macro nDCG@10\. Paired bootstrap over queries \(10,000 resamples, stratified by subset\) gives 95% CI\[0\.087,0\.125\]\[0\.087,0\.125\]and one\-sidedP⁡\(Δ≤0\)<0\.001P\(\\Delta\\leq 0\)<0\.001\(Table[10](https://arxiv.org/html/2608.19957#S6.T10)\)\. Against[google/embeddinggemma\-300m](https://huggingface.co/google/embeddinggemma-300m), the balanced macro difference is \+0\.0588, with 95% CI\[0\.042,0\.075\]\[0\.042,0\.075\]and one\-sidedP⁡\(Δ≤0\)<0\.001P\(\\Delta\\leq 0\)<0\.001\. The same table shows untuned RRF vs\. dense\-only: significantly worse for[PruhaNLP/USER2\-1C\-code](https://huggingface.co/PruhaNLP/USER2-1C-code)on both subsets and for[deepvk/USER2\-base](https://huggingface.co/deepvk/USER2-base)onfastcode\.

Table 10:Paired\-bootstrap nDCG@10 comparisons\.
#### Contamination Sensitivity\.

We recompute nDCG@10 after removing every example flagged by the conservative audit through an exact or 13\-gram overlap in either the question or code\. On the resulting clean subsets,[PruhaNLP/USER2\-1C\-code](https://huggingface.co/PruhaNLP/USER2-1C-code)scores 0\.4653 onforum\(n=2702n=2702\) and 0\.7369 onfastcode\(n=409n=409\), giving 0\.6011 balanced macro and 0\.5010 query\-weighted micro\. The originalfastcodescore is 0\.7366, compared with 0\.7356 on flagged examples, 0\.7359 after excluding exact\-question matches, and 0\.7357 after excluding input\-and\-label cases\. Thus, the highfastcodescore is stable under all tested exclusions\. For comparison,[google/embeddinggemma\-300m](https://huggingface.co/google/embeddinggemma-300m)decreases from 0\.7084 to 0\.6903 on clean\-onlyfastcode; the clean\-only margin of[PruhaNLP/USER2\-1C\-code](https://huggingface.co/PruhaNLP/USER2-1C-code)is therefore larger, not smaller\.

### 6\.3Qualitative Examples

Fine\-tuning helps when a query uses 1C platform vocabulary whose gold answer is a specific metadata or UI API rather than a shared lexical token\. It can still fail on out\-of\-domain COM utilities and on short underspecified titles, where general multilingual encoders recover via surface cues\. Table[11](https://arxiv.org/html/2608.19957#S6.T11)shows two wins \([PruhaNLP/USER2\-1C\-code](https://huggingface.co/PruhaNLP/USER2-1C-code)at rank 1; every dense baseline in Table[7](https://arxiv.org/html/2608.19957#S5.T7)and BM25 outside the top 10\) and two failures\.

Table 11:Qualitative wins and failures \(original Russian query/code\)\. Gold rank: 1 = best;\>\>10 = outside top\-10\.
### 6\.4Matryoshka Truncation

MRL truncates a trained embedding to a prefix𝐞\(m\)=𝐞1:m\\mathbf\{e\}^\{\(m\)\}=\\mathbf\{e\}\_\{1:m\}and re\-normalizes\. Figure[4](https://arxiv.org/html/2608.19957#S6.F4)and Table[12](https://arxiv.org/html/2608.19957#S6.T12)show that 256\-d embeddings retain 99\.9% of 768\-d nDCG@10 at one third the dense\-index size\. Because cosine similarity between L2\-normalized embeddings is add\-dimensional dot product, reducingddfrom 768 to 256 also cuts the per\-comparison multiply–add count by a factor of three, yielding a corresponding theoretical3×3\\timesspeedup in exact similarity scoring; end\-to\-end retrieval latency is not measured here\.

Figure 4:Matryoshka truncation: balanced\-subset macro nDCG@10 by embedding dimension\.Table 12:MRL truncation on[PruhaNLP/1C\-Ebench](https://huggingface.co/datasets/PruhaNLP/1C-Ebench)\.

## 7Limitations and Future Work

#### Single\-Gold Qrels\.

[PruhaNLP/1C\-Ebench](https://huggingface.co/datasets/PruhaNLP/1C-Ebench)has exactly one gold document per query\. On forum data this can underestimate quality when alternatives exist\. In a manual check of 50 queries \(25 per subset, seed 42\), top\-10 mined neighbors contained multiple relevant fragments in 5 cases \(10%\), mostly near duplicates or equivalent recipes\. Incomplete judgments are well known in IR \(e\.g\., BEIR\[[23](https://arxiv.org/html/2608.19957#bib.bib21)\]\) and also affect hard\-negative mining; we neither estimate FN rate nor denoise in this version\.

#### Synthetic Data and Judge Validation\.

Synthetic query quality is screened with an LLM judge on 300 examples, without human–AI agreement in the specialized Russian\-to\-BSL domain\. Scores are a useful signal but not expert annotation\.

#### Privacy and Licensing\.

PII scrubbing is rule\-based over common surface patterns without measured precision/recall or a residual audit\. License metadata and provenance are documented, but we do not resolve every redistribution implication of heterogeneous upstream sources\.

#### Experimental Coverage\.

We do not evaluate rerankers, multi\-seed variance, hard\-vs\-random negatives, tokenizer\-patch ablations, ANN indexes, or latency at larger scale\. The train–benchmark domain gap remains, andnn\-gram contamination misses semantic paraphrases\. Future work includes multi\-gold qrels, human query judging, FN denoising, tuned hybrids/rerankers, and targeted ablations\.

## 8Conclusion

Domain adaptation matters:[PruhaNLP/USER2\-1C\-code](https://huggingface.co/PruhaNLP/USER2-1C-code)reaches 0\.5992 balanced macro nDCG@10, 0\.5044 query\-weighted micro, and 0\.4617 onforum, beating[deepvk/USER2\-base](https://huggingface.co/deepvk/USER2-base)by 0\.106 and[google/embeddinggemma\-300m](https://huggingface.co/google/embeddinggemma-300m)by 0\.0588 in balanced macro under paired bootstrap\. Removing every exact/13\-gram\-flagged example leaves 0\.6011 balanced macro \(0\.5010 micro\), while clean\-onlyfastcoderemains 0\.7369; detected overlap therefore does not explain the headline result\. Dense retrieval also substantially beats BM25Okapi, while naive RRF helps neither[PruhaNLP/USER2\-1C\-code](https://huggingface.co/PruhaNLP/USER2-1C-code)nor[deepvk/USER2\-base](https://huggingface.co/deepvk/USER2-base)in the tested setup\. Matryoshka truncation to 256\-d preserves 99\.9% quality at roughly one third storage and similarity arithmetic\.

#### Disclosure of Interests\.

The authors have no competing interests to declare that are relevant to the content of this article\.

## References

- \[1\]arefaste\(2025\)Arefaste: parsed dataset for 1c from popular forums\.Hugging Face\.Note:[https://huggingface\.co/datasets/arefaste/1C\_Forums](https://huggingface.co/datasets/arefaste/1C_Forums)Accessed: 2026\-07\-12Cited by:[§3\.2](https://arxiv.org/html/2608.19957#S3.SS2.p1.1)\.
- \[2\]M\. H\. M\. Bhuiyan, M\. K\. B\. Kumar, and C\. Staicu\(2026\)“Write in english, nobody understands your language here”: a study of non\-english trends in open\-source repositories\.CoRRabs/2602\.19446\.External Links:[Link](https://arxiv.org/abs/2602.19446)Cited by:[§2\.3](https://arxiv.org/html/2608.19957#S2.SS3.p1.1)\.
- \[3\]T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal,et al\.\(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 1877–1901\.External Links:[Link](https://arxiv.org/abs/2005.14165)Cited by:[§3\.4](https://arxiv.org/html/2608.19957#S3.SS4.p1.1)\.
- \[4\]G\. V\. Cormack, C\. L\. A\. Clarke, and S\. Buettcher\(2009\)Reciprocal rank fusion outperforms condorcet and individual rank learning methods\.InProceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 758–759\.External Links:[Document](https://dx.doi.org/10.1145/1571941.1572114)Cited by:[§2\.4](https://arxiv.org/html/2608.19957#S2.SS4.p1.1)\.
- \[5\]J\. Dodge, M\. Sap, A\. Marasović, W\. Agnew, G\. Ilharco, D\. Groeneveld,et al\.\(2021\)Documenting large webtext corpora: a case study on the colossal clean crawled corpus\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 1286–1305\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.98)Cited by:[§3\.4](https://arxiv.org/html/2608.19957#S3.SS4.p1.1)\.
- \[6\]Y\. Fan, C\. Li, J\. Ge, L\. Huang, and B\. Luo\(2025\)Effective hard negative mining for contrastive learning\-based code search\.ACM Transactions on Software Engineering and Methodology34\(3\),pp\. 1–35\.External Links:[Document](https://dx.doi.org/10.1145/3695994)Cited by:[§2\.4](https://arxiv.org/html/2608.19957#S2.SS4.p1.1)\.
- \[7\]GenLab\-1C\(2026\)PRISM: executable BSL code generation benchmark for 1C:Enterprise\.Note:[https://github\.com/genlab\-1c/prism](https://github.com/genlab-1c/prism)Accessed: 2026\-07\-12Cited by:[§2\.2](https://arxiv.org/html/2608.19957#S2.SS2.p1.1)\.
- \[8\]GigaCode R&D and Sber AI\(2026\)1C Code Bench: a benchmark for evaluating the ability of LLMs to write 1C code\.Note:Habr,[https://habr\.com/ru/companies/sberbank/articles/1040114/](https://habr.com/ru/companies/sberbank/articles/1040114/)In Russian\. Accessed: 2026\-08\-18Cited by:[§2\.2](https://arxiv.org/html/2608.19957#S2.SS2.p1.1)\.
- \[9\]R\. Goworek, O\. Macmillan\-Scott, and E\. B\. Özyiğit\(2025\)Bridging language gaps: advances in cross\-lingual information retrieval with multilingual llms\.CoRRabs/2510\.00908\.External Links:[Link](https://arxiv.org/abs/2510.00908)Cited by:[§2\.3](https://arxiv.org/html/2608.19957#S2.SS3.p1.1)\.
- \[10\]J\. Huang, D\. Tang, L\. Shou, M\. Gong, K\. Xu, D\. Jiang,et al\.\(2021\)CoSQA: 20,000\+ web queries for code search and question answering\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 5690–5700\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.442)Cited by:[§2\.1](https://arxiv.org/html/2608.19957#S2.SS1.p1.1)\.
- \[11\]Y\. Huang, S\. Wu, R\. Song, Y\. Xiang, Y\. Xian, S\. Gao,et al\.\(2025\)Multilingual generative retrieval via cross\-lingual semantic compression\.CoRRabs/2510\.07812\.External Links:[Link](https://arxiv.org/abs/2510.07812)Cited by:[§2\.3](https://arxiv.org/html/2608.19957#S2.SS3.p1.1)\.
- \[12\]H\. Husain, H\. Wu, T\. Gazit, M\. Allamanis, and M\. Brockschmidt\(2019\)CodeSearchNet challenge: evaluating the state of semantic code search\.CoRRabs/1909\.09436\.External Links:[Link](https://arxiv.org/abs/1909.09436)Cited by:[§2\.1](https://arxiv.org/html/2608.19957#S2.SS1.p1.1)\.
- \[13\]J\. Johnson, M\. Douze, and H\. Jégou\(2021\)Billion\-scale similarity search with GPUs\.IEEE Transactions on Big Data7\(3\),pp\. 535–547\.External Links:[Document](https://dx.doi.org/10.1109/TBDATA.2019.2921572)Cited by:[§4\.2](https://arxiv.org/html/2608.19957#S4.SS2.p3.1)\.
- \[14\]V\. Karpukhin, B\. Oguz, S\. Min, P\. Lewis, L\. Wu, S\. Edunov,et al\.\(2020\)Dense passage retrieval for open\-domain question answering\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 6769–6781\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.550)Cited by:[§2\.4](https://arxiv.org/html/2608.19957#S2.SS4.p1.1),[§4\.2](https://arxiv.org/html/2608.19957#S4.SS2.p3.1)\.
- \[15\]A\. Kusupati, G\. Bhatt, A\. Rege, M\. Wallingford, A\. Sinha, V\. Ramanujan,et al\.\(2022\)Matryoshka representation learning\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 30233–30249\.External Links:[Link](https://arxiv.org/abs/2205.13147)Cited by:[§2\.4](https://arxiv.org/html/2608.19957#S2.SS4.p2.1)\.
- \[16\]leongl\(2024\)1c\_github: 1c code corpus from GitHub\.Hugging Face\.Note:[https://huggingface\.co/datasets/leongl/1c\_github](https://huggingface.co/datasets/leongl/1c_github)Accessed: 2026\-07\-12Cited by:[§4\.2](https://arxiv.org/html/2608.19957#S4.SS2.p3.1)\.
- \[17\]H\. Li, X\. Zhou, L\. A\. Tuan, and C\. Miao\(2023\)Rethinking negative pairs in code search\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 12760–12774\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.786)Cited by:[§2\.4](https://arxiv.org/html/2608.19957#S2.SS4.p1.1),[§4\.2](https://arxiv.org/html/2608.19957#S4.SS2.p3.1)\.
- \[18\]X\. Li, K\. Dong, Y\. Q\. Lee, W\. Xia, H\. Zhang, X\. Dai,et al\.\(2025\)CoIR: a comprehensive benchmark for code information retrieval models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 22074–22091\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1072)Cited by:[§2\.1](https://arxiv.org/html/2608.19957#S2.SS1.p1.1)\.
- \[19\]S\. Lu, D\. Guo, S\. Ren, J\. Huang, A\. Svyatkovskiy, A\. Blanco,et al\.\(2021\)CodeXGLUE: a machine learning benchmark dataset for code understanding and generation\.CoRRabs/2102\.04664\.External Links:[Link](https://arxiv.org/abs/2102.04664)Cited by:[§2\.1](https://arxiv.org/html/2608.19957#S2.SS1.p1.1)\.
- \[20\]OpenRouter\(2026\)OpenRouter documentation\.Note:[https://openrouter\.ai/docs](https://openrouter.ai/docs)Accessed: 2026\-08\-18Cited by:[§4\.3](https://arxiv.org/html/2608.19957#S4.SS3.p1.1)\.
- \[21\]Y\. Qu, Y\. Ding, J\. Liu, K\. Liu, R\. Ren, W\. X\. Zhao,et al\.\(2021\)RocketQA: an optimized training approach to dense passage retrieval for open\-domain question answering\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 5835–5847\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.466)Cited by:[§2\.4](https://arxiv.org/html/2608.19957#S2.SS4.p1.1),[§4\.2](https://arxiv.org/html/2608.19957#S4.SS2.p3.1)\.
- \[22\]S\. Robertson and H\. Zaragoza\(2009\)The probabilistic relevance framework: BM25 and beyond\.Foundations and Trends in Information Retrieval3\(4\),pp\. 333–389\.External Links:[Document](https://dx.doi.org/10.1561/1500000019)Cited by:[§2\.4](https://arxiv.org/html/2608.19957#S2.SS4.p1.1)\.
- \[23\]N\. Thakur, N\. Reimers, A\. Rücklé, A\. Srivastava, and I\. Gurevych\(2021\)BEIR: a heterogeneous benchmark for zero\-shot evaluation of information retrieval models\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks,External Links:[Link](https://arxiv.org/abs/2104.08663)Cited by:[§7](https://arxiv.org/html/2608.19957#S7.SS0.SSSx1.p1.1)\.
- \[24\]A\. van den Oord, Y\. Li, and O\. Vinyals\(2018\)Representation learning with contrastive predictive coding\.CoRRabs/1807\.03748\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1807.03748)Cited by:[§4\.4](https://arxiv.org/html/2608.19957#S4.SS4.p1.1)\.
- \[25\]Z\. Z\. Wang, A\. Asai, X\. V\. Yu, F\. F\. Xu, Y\. Xie, G\. Neubig,et al\.\(2025\)CodeRAG\-Bench: can retrieval augment code generation?\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 3199–3214\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.176)Cited by:[§2\.1](https://arxiv.org/html/2608.19957#S2.SS1.p1.1)\.
- \[26\]L\. Xiong, C\. Xiong, Y\. Li, K\. Tang, J\. Liu, P\. Bennett,et al\.\(2021\)Approximate nearest neighbor negative contrastive learning for dense text retrieval\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=zeFrfgyZln)Cited by:[§2\.4](https://arxiv.org/html/2608.19957#S2.SS4.p1.1),[§4\.2](https://arxiv.org/html/2608.19957#S4.SS2.p3.1)\.
- \[27\]L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang,et al\.\(2023\)Judging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 46595–46623\.External Links:[Link](https://arxiv.org/abs/2306.05685)Cited by:[§2\.5](https://arxiv.org/html/2608.19957#S2.SS5.p1.1)\.
- \[28\]Q\. Zhu, J\. Cao, X\. Chen, W\. Zhang, Y\. Lu, H\. Lin,et al\.\(2026\)Across programming language silos: a study on cross\-lingual retrieval\-augmented code generation\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 24283–24296\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.1216)Cited by:[§2\.3](https://arxiv.org/html/2608.19957#S2.SS3.p1.1)\.