LoRA on abliterated Qwen 3.8-27B for internal codebase recall[P]

Reddit r/MachineLearning 09/18/26, 10:58 AM Models

lora qwen fine-tuning abliterated internal-tooling codebase-recall hugging-face

Summary

A LoRA adapter was trained on an abliterated Qwen 3.8-27B model to enhance internal codebase recall, demonstrating superior performance over Claude models on private-repo-specific tasks in evaluations.

Trained a LoRA on top of huihui-ai/Huihui-Qwen3.8-27B-abliterated to get the model to recall internal codebase conventions (migration rules, deploy checks, secret handling) without touching the base weights cheap to train (r=16, ~80M trainable params, ~7.5 hours) and cheap to share compared to shipping a full fine-tuned checkpoint. Disclosure: part of a side project of mine (ClawHunt). Went with the abliterated base because refusal behavior gets in the way for internal-tooling use cases. That behavior is inherited from Huihui's fork, not something I added worth knowing if you build on top of it, since it changes what the model will and won't refuse. Where it's actually strong: ran it against Claude Sonnet 5 and Opus 5 on the same 96 prompts, and it wins clearly on private-repo-specific recall makes sense, since Claude never saw that codebase. More interesting to me, the two Claude models failed the hallucination-related gates in different ways from each other, which wasn't something I expected going in. On the eval against the previous adapter version, I'd rather report it straight than round it up: passed all 6 internal gates, but the gain over the prior version wasn't statistically significant (McNemar p=0.5), and the best score ties rather than beats an earlier benchmark. Figured that's more useful to share than pretending it was a bigger jump than it was. Not deployed anywhere yet, still experimental. Model card for reference: huggingface.co/Clawhunt-store/clawhunt-p6-candidate-20260909

Original Article

Similar Articles

Recover-LoRA for Aggressive Quantization: Reclaiming Accuracy in 2-Bit Language Models via Low-Rank Adaptation with Knowledge Distillation on Synthetic Data

arXiv cs.LG

Researchers from AMD propose Recover-LoRA, a method that uses low-rank adaptation with knowledge distillation on synthetic data to recover accuracy lost from aggressive 2-bit quantization of LLMs, achieving 80–95% accuracy recovery on 9 of 12 benchmarks for Qwen3-4B using only 10k synthetic samples.

Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution

Hugging Face Daily Papers

Code2LoRA introduces a hypernetwork that generates LoRA adapters from a repository in a single forward pass, allowing frozen code LLMs to adapt to repository context without extra tokens, and supporting evolving codebases efficiently. It also delivers RepoPeftBench, a benchmark for repo-conditioned code modeling.

AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning

arXiv cs.LG

AQLoRA is a zero-search method that accelerates quantized LoRA fine-tuning by adaptively keeping layers with high NF4 reconstruction error in fp16, achieving up to 11% faster training with minimal accuracy loss.

Real Continual Learning Model (Prototype)

Reddit r/ArtificialInteligence

A developer claims to have built a real continual learning model prototype using LoRA to give Qwen4B instant, generalizable memory without retraining, and is inviting independent researchers to validate the mechanism.

How Small Can You Go? LoRA Fine-Tuning 270M-8B Models for Merchant Information Extraction in Financial Transactions

arXiv cs.AI

This paper presents a deployment-focused study comparing LoRA fine-tuning of 24 model variants (270M–8B parameters) for merchant information extraction from financial transaction strings. The authors find that smaller models like Qwen 3.5 4B achieve 96.6% F1, within 0.35 points of the 8B baseline, while offering significant reductions in latency and cost.