Teaching LLMs a Low-Resource Language: Enhancing Code Completion in Pharo
Summary
This paper presents an end-to-end pipeline for adapting LLMs to low-resource programming languages, using Pharo as a case study, achieving superior code completion with small models suitable for real-time IDE support.
View Cached Full Text
Cached at: 07/09/26, 07:50 AM
Paper page - Teaching LLMs a Low-Resource Language: Enhancing Code Completion in Pharo
Source: https://huggingface.co/papers/2607.04939
Abstract
Large language models can be adapted for low-resource programming languages through specialized training pipelines and benchmarks, achieving superior code completion performance compared to general-purpose models.
Large Language Models(LLMs) unlocked new possibilities in automated code writing, becoming the backbone of mostcode completiontools. While LLMs excel in mainstream languages, they often lack support for the so-called low-resource languages where training data is scarce. As a result, these languages lag behind in the quality ofcode completiontooling available to their communities. A concrete example is Pharo, a Smalltalk-inspired language whose IDE currently offers only single-token completion. In this work, we report on our experience bringing LLM-basedcode completionto Pharo. First, we describe an end-to-end pipeline that combines Pharo-specific data curation,continued pre-trainingandfine-tuningof open code LLMs. Second, we introduce a set of Pharocode completion benchmarksdesigned to evaluate whether models (i) learn Pharo’s syntax and (ii) accurately complete masked Pharo code from real-world GitHub repositories. Third, we show empirically that Pharo-specialized models substantially outperform their original base checkpoints and also exceed the accuracy of substantially larger code LLMs on Pharo completion. Overall, our case study demonstrates the feasibility of bringing strong LLM-basedcode completionto low-resource programming languages, with models small enough to provide ``real-time’’ in-IDE support.
View arXiv pageView PDFGitHub0Add to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.04939 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.04939 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.04939 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages
This tutorial paper provides an overview of building multilingual and multimodal LLMs for low-resource languages, covering data creation, model alignment, fine-tuning, and evaluation, with a focus on practical recipes and hands-on resources.
No Code LLM Finetuning
This article discusses making LLM finetuning accessible to non-coders, highlighting a video demonstration of how anyone can customize LLMs without coding.
HPC-LLM: Practical Domain Adaptation and Retrieval-Augmented Generation for HPC Support
This paper presents HPC-LLM, a retrieval-augmented and domain-adapted assistant for HPC workflows, fine-tuning Llama 3.1 8B with QLoRA on HPC documentation. It demonstrates performance comparable to larger general-purpose models with significantly lower resource requirements.
No Resource, No Benchmarks, No Problem? Evaluating and Improving LLMs for Code Generation in No-Resource Languages
This paper tackles code generation for no-resource programming languages by building benchmarks and proposing a method that combines further pre-training with weight difference transfer to create specialized instruction-following models at reduced cost.
Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language
This paper introduces PyLang, a programming language absent from all pretraining corpora, and shows that LLMs fine-tuned on it can learn syntax but fail to transfer algorithmic reasoning, resulting in an 'implementation fidelity gap' where models understand algorithms but cannot express them in an unfamiliar language.