Tag
A pull request to llama.cpp refactors Intel XMX matrix engine support, delivering massive improvements to speculative decoding performance on Intel hardware.
该论文证明每个 Transformer 块都可精确编译为由固定仿射映射、路由及 sigmoid/逆平方根等基本算子构成的通用深度神经网络，量化分析了去除激活-激活乘积的代价，并提出多智能体注意力聚类方法将打分复杂度从 N² 降至 Nk，实验在 DeiT-Tiny、MambaIRv2-Light 和 Pythia-70M 上验证了编译模型与原模型一致且标准后训练量化可直接适用。
一篇系统综述，涵盖了一百多项关于在 FPGA 和集成电路平台上部署强化学习的研究，重点分析架构设计、性能度量与硬件-算法协同设计。该综述统一比较了 FPGA 与 IC 两种实现路径的资源利用率、延迟、吞吐量、功耗及性能增益，并指出了开放挑战与未来方向。
Google 与 Meta 联合发布开源项目 TorchTPU，为 Google TPU 提供原生 PyTorch 后端，支持真正的 eager 执行与 torch.compile 集成，并在 Llama 3 405B 和 Qwen 3 MoE 上实现近线性扩展。
Google open-sources TorchTPU, a native PyTorch backend for its TPUs, featuring custom ATen kernels, XLA compilation caching, torch.compile() integration, and distributed training support with examples for models like Llama, Qwen, and ResNet.
Hugging Face 宣布 Transformers、Diffusers 和 Kernels 原生支持 TorchTPU，使 TPU 成为 PyTorch 中的一个普通设备选项，无需再依赖 torch_xla 或 Optimum TPU。
Modular announced it is an official inference partner powering MiniMax's M3.1 Flash model, nicknamed Space Bunny. The partnership highlights Modular's MAX platform for serving new MiniMax models.
开发者正在为 AMD R9700 AI Pro 显卡构建专用推理引擎，支持单卡或双卡运行本地模型，并计划将优化整合到面向 Strix 与 Gorgon 的 Lucebox 引擎中。
Npunlock is a tool that enables running custom C kernels on Intel NPUs by reconstructing the path from C code to runnable kernels, with verification on Meteor Lake hardware.
A tweet explains the core components of inference engineering: loadable encodings, backend operations, and native hardware arithmetic, with a discussion on kernels and GPU efficiency.
Arm introduces the Mali G2-Ultra NX GPU, its first AI-native GPU that integrates neural acceleration into the graphics pipeline to deliver desktop-class mobile gameplay with AI-enhanced graphics.
The article discusses alternatives to the official llama.cpp for optimized performance on Strix Halo hardware, with specific forks achieving significantly higher throughput for AI inference tasks.
This megathread compiles a list of the latest open-source projects, research papers, and hardware innovations focused on optimizing inference, efficiency, and accessibility for open-source LLMs and related technologies.
WebLLM is a high-performance in-browser LLM inference engine that leverages WebGPU for hardware acceleration and is fully compatible with the OpenAI API, enabling local execution of open-source language models.
This paper proposes a Generalized Optimization Engine (GOE) to accelerate AI inference on resource-constrained edge devices by integrating various model compression techniques, demonstrating that the choice of compression method affects task accuracy for language models deployed on GPU-less CPUs.
At Hot Chips 2026, Samsung presented their Processing-in-Memory (PIM) technology, which integrates MAC units within LPDDR5X chips to exploit high internal bandwidth for enhanced compute efficiency in AI workloads.
This is a fork of NVIDIA's open GPU kernel modules that enables PCIe peer-to-peer communication between consumer GeForce GPUs (3090, 4090, 5090), significantly improving multi-GPU performance by allowing direct data transfer over PCIe.
The article presents Warnock, a technique that leverages GPU geometry amplification to improve vector graphics rendering, potentially boosting performance and visual quality.
AI coding agents write runtime adapters to bridge gaps between new model architectures and software stacks, enabling thousands of HuggingFace models to run on the IBM Spyre accelerator with day-one support.
Reverse-engineered the RK3588 NPU and built an open compiler and runtime to run GPT-2 and SigLIP from PyTorch, ONNX, and JAX without vendor SDK.