Tag
Explains six parallelism strategies for running LLMs in production — data, tensor, pipeline, context, expert, and hybrid parallelism — highlighting each one's trade-offs and guidance on choosing based on measured bottlenecks like memory, TTFT, and TPOT.
A blog post introducing "Weight-Space Programming," where frontier LLM GPT-6 Astra natively reads and edits neural network weights—changing only 0.016% of Qwen3.5-0.8B's weights to boost MATH-500 from 7.9% to 48.1% and tripling IFEval, without gradients or training.
该论文提出了一个统一的“工作智能体”(work agents)框架,将基础模型与运行时框架结合以可靠执行复杂数字任务,引入持久性与开放性两个维度,并系统分析能力族、模型与框架优化及评估方法。
HF Viewer 创作者宣布推出 embedl.com,一个收录 1000+ 针对边缘设备优化模型的平台,用户可通过免费的真实边缘设备农场(如 NVIDIA Orin、AGX Thor、RTX GPU)直接运行模型并获得详细性能分析。
一位用户质疑 Strata 的高性能宣传:虽然 tokens per second 和针尖测试数据亮眼,但其剪枝专家(pruned experts)与量化是否牺牲实际输出质量尚未经过验证,建议用 llama.cpp 上的相同 GGUF 来对照测试。
Woosuk Kim congratulates Reflection AI on releasing Reflection-Beam-1.0, a 501B-total-parameter (23B active) MoE model with 1M-token context, hybrid attention, and Harmony-format reasoning, highlighting AI agents autonomously driving model optimization with tracked performance gains and Day-0 vLLM/SkyRL support.
本文系统分析了LLM评估中rubric设计的机制,涵盖标准组织、操作化与分数聚合,并提出ARTS-Hybrid方法(自适应树构建+语义优先的响应校准细化),在评估性能和rubric质量上均有提升,且生成的rubric可作为模型优化的有效信号。
Project Maya 是一个开源工具,可在本地 GPU 上运行 321B 参数的 GLM-5.3-Flash 模型,通过分层存储(GPU/RAM/NVMe)和自动调优实现单卡 30+ tokens/s 的推理速度,并提供 OpenAI/Anthropic 兼容 API。
Tibo announces a ~50% speed optimization for GPT-6 Astra and GPT-6.1 Sol across OpenAI products and third-party partners using Sign in With ChatGPT, and a user's test confirms a jump from ~20 tps to ~30 tps.
A 12GB GPU with 64GB of system RAM is enough to run a 100-billion-parameter large model: the open-source project Strata can run quantized versions of Qwen3.8-Flash-Next — about 94 tokens/s with the Q2_0 quantization and about 53 tokens/s with IQ3_S on an RTX 5070. The author remarks that the hammer is ready; the question is where the nail (i.e., the application scenario) is.
MIT 6.5940: TinyML and Efficient AI Computing, taught by MIT Associate Professor Song Han, has open-sourced all its slides and videos — covering model compression, TinyML deployment, LLM and diffusion acceleration, and efficient training, with five hands-on labs.
The author asks whether the extensive inference optimizations going into Qwen3.8 Flash Next will position Alibaba to quickly uplift those gains to Qwen4 when it releases.
ANEMLL-FORGE reports on applying int8 V-compression to the KV-cache of Qwen 3.8-27B during prefill on Apple's ANE M6, with compression and dequantization performed on ANE to yield memory I/O savings.
River AI promotes its frontier AI development stack, claiming it helped Ivo's contract review agent jump from 70% to 91% on the LAB benchmark at lower cost than competitors, with RL training and inference on optimized open-weight models via River Cloud or on-prem clusters.
A user shares early impressions of Dwarfstar, a tool offering bespoke quantization techniques for running large language models locally, reporting fast performance for Qwen3-8B on an M3 Ultra 96GB Mac Studio with leftover memory headroom.
The article argues that harnesses for AI models don't boost innate reasoning but steer models to maintain performance, with examples showing a significant improvement on ARC-AGI-3 through state retention and compaction.
The article explains when and how to finetune a specialized decision model classifier that outperforms Opus, is faster than Jev, and runs on user devices.
BeeLlama v0.4.7 is released, offering gigabytes of VRAM savings through independent drafter ubatch sizing for MTP and DFlash, along with major ROCm and CUDA performance enhancements for AI inference.
Pruna-Qwen-Image-2.1 is a set of LoRA adapters that significantly speed up image generation and editing in Qwen-Image-2.1, reducing steps from 40 to 5 or 8 for up to 6.3x faster performance.
Dynamic Quantiser is an open-source tool that enables users to create custom and high-quality dynamic quantizations of LLM GGUF files, minimizing cosine deviation for better accuracy compared to standard quants.