Inference time abliteration

Reddit r/LocalLLaMA Tools
llama-cpp abliteration jailbreak open-source llm inference

Summary

AbliteratedOutTheBox 是一个 llama.cpp 分支，可在推理时对任意加载的模型实时执行 abliteration（去除对齐/拒绝机制），无需微调、无需额外文件，也不占用额外显存。

AbliteratedOutTheBox is a llama.cpp fork that abliterates any model you load, live at inference, with no fine-tuning and no extra files. This eliminates any hassle of abliteration without requiring any more vram. https://github.com/heterodoxin/AbliteratedOutTheBox
Original Article

Similar Articles

@NFTCPS: 4GB VRAM running 70B large model? It actually works! AirLLM did a clever trick — layered inference, not loading the whole model into VRAM at once, but layer by layer, compute and discard, squeezing the giant into a small GPU. The best part: 100% open source, freebie warning https://github.com/0xSo…

X AI KOLs Timeline

AirLLM is a fully open-source tool that uses layered inference (loading and releasing VRAM layer by layer) to enable 70B large language models to run on GPUs with only 4GB VRAM, without quantization, distillation, or pruning. It already supports running Llama3.1 405B on 8GB VRAM.

@NFTCPS: Attention to those running large models locally! Someone has transformed llama.cpp into a performance beast — BeeLlama.cpp. With the same VRAM, inference speed triples and context capacity expands 7.5x. This isn't a slide deck; it's real benchmark data. It stuffs three top-tier optimizations into one codebase: DFlash speculative decoding…

X AI KOLs Timeline

BeeLlama.cpp is a fork of llama.cpp that integrates DFlash speculative decoding, TurboQuant/TCQ KV-cache compression, and adaptive draft control, achieving up to 3x faster inference and 7.5x context expansion on the same hardware.

Liquid Inference

Product Hunt

Liquid Inference 是一款 LLM 路由器，让多家模型提供商针对每次 prompt 动态竞价，从而为用户提供最低推理成本，兼容 Claude Code、Cursor 等主流 agentic 编码工具，支持数百个开源与闭源模型。