Tag
A practitioner roundup of which LLMs teams are running for real-time voice agents, covering dedicated vs. serverless trade-offs (Gemma 4 31B, Qwen3 235B, GLM 5.2, Nemotron 3 Ultra, GPT-4.1/5.6, Gemini Flash), along with deployment lessons on measuring latency, caching, tool testing and idle GPU cost.
用户推荐 MIT 6.5940 Fall 2026 课程《TinyML and Efficient AI Computing》，由 Song Han 团队讲授，涵盖剪枝、量化、蒸馏、神经架构搜索、LLM 量化与部署、扩散模型优化等高效 AI 计算技术。
该论文提出 Stepped MoE，一个将弹性结构与稀疏门控路由统一的框架，使单一 LLM 能同时适应部署约束和任务需求，在 1B-4B 参数规模间灵活切换并优于稠密模型 2-5%。
A developer got llama.cpp compiling with WebGPU, demonstrating a proof-of-concept of an LLM running fully inside the browser, with plans to turn it into an agent that can interact with web apps.
A practical guide covering common inference bottlenecks — including runtime selection (Ollama, llama.cpp, vLLM, SGLang), hardware-specific compilation flags, quantization choices (IQ vs Q_K vs NL), and model selection by task — with best practices for optimizing LLM inference within hardware constraints.
AgBench introduces a benchmark suite and open artifacts for evaluating agentic AI on personal devices, comparing local, hybrid, and cloud execution across task success, latency, API cost, and data exposure using over 162 million data points. The study finds no single architecture dominates, with local execution reducing cloud cost and privacy risk but generally lowering task success.
The article speculates on consumer interest in a hypothetical AI chip from Taalas that could run large language models like Qwen3.8-27B at high speeds, comparing it to past media formats.
This paper explores how prompt properties like cognitive load and phrasing pattern influence energy usage in on-device LLM inference, showing that cognitive load affects energy per token while phrasing impacts token usage, highlighting the need for model-aware prompt design for energy efficiency.
The author demonstrates how to run a 182B Qwen model on a 4080 GPU with 64GB DDR5 by offloading ngrams to SSD, achieving faster and more intelligent performance than a 27B model, making large model inference accessible on consumer hardware.
This article is a detailed guide on large language model deployment, covering key metrics such as latency, throughput, and memory usage, and illustrates how to optimize performance and choose hardware through practical cases.
A tweet sharing information or resources about the deployment of local large language models with GPUs.
This article discusses the trade-offs and considerations for developers deciding between using cloud-based AI services versus running large language models locally.
Fixed three bugs in a qMLX fork for running Qwen3.5-122B on Mac Studio, reducing prefill time from minutes to sub-seconds for long-context inference; open-sourced the fork and benchmark script.
Explored NVIDIA Dynamo, a tool for deploying LLMs across multiple GPU cluster nodes with features like model caching, autoscaling, multinode deployments, and Kubernetes integration.
Lilian Weng's blog post explores the concept of harness engineering as a key component for recursive self-improvement in AI systems, discussing design patterns, workflow automation, and the analogy to operating systems.
A comprehensive guide to running LLMs locally across various hardware and software setups is now available online for free, covering tools like llama.cpp, vLLM, and more.
This article provides a comprehensive overview of the complete technology stack for cloud deployment of Transformer inference, covering application scenarios, workload definition, models, inference engines, hardware, observability, and performance optimization, along with future trends.
This paper presents a two-stage methodology for end-to-end LLM deployment on spatial NPUs, progressing from human-guided development to an autonomous agent skill system. The system achieves speedups of 2.2x on prefill and 4.0x on decode for a reference model, and autonomously deploys eight additional LLMs on AMD XDNA 2 NPU with minimal human guidance.
A practitioner seeks advice on running AI agents 24/7 without high API costs, asking about local models, cloud GPUs, or hosted APIs, and wants cost-efficient setups balancing reliability and reasoning quality.
A lecture on LLM deployment techniques covering AWQ, vLLM, FlashAttention, quantization, and activation smoothing for efficient serving.