Tag
The paper introduces 'All-in-One', a composable attack framework from ETH Zurich that trains LLMs to pass safety checks yet turn malicious after deployment-time transformations like quantization, pruning, or model merging, by approximating each transformation as weight perturbations jointly optimized in a single training objective.
A hobbyist reports progress running the 320B-parameter GLM-5.3-Flash MoE model at ~8-9 tok/s on dual Huawei Ascend 310P cards using vllm-ascend, including hot-swapping modules, YaRN for 1M context, and a hybrid 'Ascend Sidecard' idea to offload weights from CUDA GPUs to Ascend cards.
作者发布了 Qwen3.8-27B Uncensored (Heretic + MTP) 的 UD-IQ4_XS 量化版本，使其能在 16 GB 显存的 RTX 4080 上以 55-68 tok/s 运行，并提供 12 GB / 24 GB 版本及 KLD 质量对比与速度基准测试。
LlamAmpere 新版本发布，面向 12GB Ampere 显卡优化，可在 RTX 3060/3080/3090 上运行 200K+ 上下文的 Qwen3.8 27B 量化模型，并引入 Staged + Journaled KVaRN 变体将 KV 缓存压缩的 KLD 降低约 40%，性能达到 BF16 的约 85%。
autotrust/GEV-26B-Decide-NVFP4 is an NVFP4 4-bit quantized release of GEV-26B-Decide that reduces weights from 49.5 GB to 18 GB (−66% GPU memory) by quantizing only the 3,840 routed expert MLPs, with benchmark results within run-to-run noise of the bf16 model and support for native W4A4 inference on Blackwell GPUs via vLLM.
发布了一款基于 Qwen3.8-27B 经多轮 SFT 与 RLOO 后训练的编程模型，修复了原模型不停推理直至 94K 截断的问题，并提供 BF16、FP8、NVFP4、INT8/INT4、GGUF 等多种量化版本，支持 DFlash2/MTP 投机解码，推理速度最高约 180 tok/s。
The author provides a detailed introduction to using Claude Opus-5.5 for kernel development and AI work, sharing practical experiences with local model workflows, quantization tasks, and hardware selection.
A user benchmarks exllamav3 against llama.cpp for CPU-offloaded inference, showing exllamav3 is faster for Qwen models but slower for others, depending on hardware and model architecture.
An analysis suggests that AtomicChat's Qwen3.8-Flash-Next quant may be deceptive by misrepresenting IQ2_S tensors as Q4_K_M, highlighting potential transparency issues in model quantization.
The team quantized Qwen 3.8 27B into various GGUF formats and benchmarked them on an RTX 6000, finding similar performance across quants with AD-Q6_K recommended for safety.
Introducing colibri, an open-source inference engine written in pure C that unifies VRAM, RAM, and disk as a hierarchical structure to stream large model weights, supporting various cutting-edge MoE models to run locally on consumer hardware, lowering the barrier to using large models.
The article critiques the lack of systematic benchmarking for AI models in different quantization formats, highlighting discrepancies between benchmark results and real-world usage, and calls for more thorough evaluation.
A detailed guide on running the quantized NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B model on two RTX 3090s using vLLM with full 262K context, achieving high inference speeds without CPU offloading.
Poolside releases GGUF quantizations of the Laguna S 2.1 AI model, including a DFlash speculative decoding draft model, enabling efficient local inference with llama.cpp.
A user tested the unsloth quantized GLM-5.2 model on a high-end consumer-like system with dual RTX 5090, achieving 12 tokens per second.
This tutorial from NVIDIA walks through the end-to-end workflow of converting an FP8-quantized PyTorch model into a TensorRT inference engine for production deployment, covering ONNX export and performance profiling.
Ported the EXL3 LLM codec to run on Apple Silicon via Metal, achieving high prefill and generation speeds on M5 Max (e.g., ~600 tok/s prefill, 17-80 tok/s gen on various models).
Unsloth is uploading a GGUF quantized version of the MiniMax M3 model to Hugging Face.
This article is the middle part of the AI Engineering Landscape series, detailing core techniques such as inference optimization, model slimming (quantization, distillation, pruning, MoE), and speculative decoding, while reviewing the latest advances from hardware to the engineering stack.
A user shares their GGUF quantized version of Qwen 3.6 27B using AutoRound, claiming it performs better than other quants, and invites feedback.