Tag
The author of Tensward used CUPTI-based probes to pinpoint the culprit behind poor throughput when serving hybrid-architecture models (such as Qwen3.8-27B) concurrently with llama.cpp: a synchronous copy of a 643MB state snapshot to host memory after every request. Disabling context checkpoints doubled concurrent throughput.
tamserve 是一个开源的实时多摄像头目标跟踪服务引擎,让单份 SAM 2 / EdgeTAM 模型同时追踪所有摄像头中的所有目标,显存占用降低约 4 倍、速度提升 11%-24%,且支持按名称跟踪与多 GPU 路由。
AMD Loom, an experimental inference stack with runtime, kernel compiler and server, added Radeon 9700 support and was benchmarked on 48 LLMs, showing an average 42% speedup over llama.cpp in prefill and decode.
vor.tech demoed a multiplayer game where each player's view is rendered by a world model running locally on their GPU, effectively a game engine that is just a neural net supporting 10 players at once. The model was trained on 3,958 hours of every-POV CS:GO footage plus full game state, and the dataset is being opened up.
audio.cpp 0.9.5 adds a real-time local avatar voice chat app in its WebUI, combining SoulX-FlashHead, OWSM v4, and LFM3-Audio to achieve ~850ms speech-to-avatar latency on an RTX 5090.
The author tested three local open-source models — DeepSeek-V4.1-Flash-1017, Qwen-3.8-Flash-Next, and Qwen-3.8-27b — on an RTX 5090, an RTX 4090, and dual V100s, comparing the real-world experience gap against cloud models like GPT-5.5 and Opus-4.8 through a Core-300 blind test and 200K+ long-context stress tests.
Openspline 是一个开源的实时数字人头像层,可通过 5 行代码接入现有语音 AI 智能体,在本地 GPU 上生成口型同步的实时视频,RTX4090 上可达 90+ FPS。
A free browser-based LLM cost calculator lets you compare local GPU inference (with RTX 4070 Ti Super / 5090 presets) against cloud API pricing, highlighting that cached-input token rates — often guessed at by other calculators — most determine where the local-vs-cloud crossover sits.
一位 Reddit 用户发布了 PS5LM 的分支,使 llama.cpp 能在越狱 PS5 上原生运行并利用 GPU 和更多内存,从而在本地运行 Qwen3.8-27B 等大型模型,速度超过 100 tokens/s。
介绍一种基于768维嵌入空间上的吸收马尔可夫链(Absorbing Markov Chain)的语义分流引擎,通过在60节点领域流形上做随机游走来替代朴素余弦相似度路由,实现<2ms的零token意图分类,从而在唤醒LLM之前完成路由。文中给出JevBench基准测试结果:四层级联系统(Gevva0)在240个条目上达到100%召回、89.17%精度,平均p50延迟46ms,并配备否定/语法、流形邻近OOD和熵与边界三重防护网。
有人在越狱的 PlayStation 5 GPU 上运行 Qwen 27B 大模型,速度约为每秒 20 个 token,展示了在游戏主机硬件上进行本地 LLM 推理的可行性。
rdna35-perf branch of strix-llama.cpp 为 Strix Halo (Radeon 8060S) 带来新的 RDNA3.5 flash attention 与 GEMM 内核优化,prompt processing 比 Vulkan 快 21–54%,在 128k 长上下文下比原版 ROCm 最快达 2.7 倍,且输出质量与原版 ROCm 基本一致。
zzpanic shares the final revision of his vllm-radiance build tuned for a single AMD R9700 GPU running qwen3.8-27b, adding startup caching and a KV-offload feature (vram-sysram-disk) with memory thinning to enable multi-agent workloads on one card. He notes the project is winding down as users migrate to StillDeadcode's newer radiance inference engine.
Google AI Edge 开源了 ML Drift,一个高性能、跨平台的 GPU 加速 AI/ML 推理引擎,支持 Android、iOS、Web(WebGPU)和服务器端,可高效运行大型生成式模型。
作者开源了其 prompt-to-game 网站 Game Summoner 的完整技术栈(MIT 协议),包含基于多智能体设计文档流程的游戏生成架构、SQL 队列 + 自动扩缩容的 GPU worker 系统,以及本地运行脚本。整个系统可在 5090 等本地 GPU 或 RunPod/AWS 托管 GPU 上部署。
Strata is an open-source tool that uses MoE heterogeneous parallelism to run the 125B-parameter Qwen3.8-Flash-Next model locally on a gaming PC with just 12GB VRAM, reaching over 60-94 tokens/s on NVIDIA RTX 5070 and AMD RX 9070 XT GPUs. The project claims full local, private inference with support for NVIDIA and AMD cards on Windows and Linux.
A quantized and abliterated Qwen3.8-27B CODER model variant (IQ4_XS with imatrix) released in a GGUF build optimized by LexiPanel to fit a single 24 GB GPU with a 262k context window, featuring an MTP head for speculative decoding at 85% draft acceptance on an RX 7900 XTX.
A user benchmarks the new Ninfer inference backend on a Blackwell MaxQ GPU, reporting 130-180 t/s on qwen3.8-27b-nvfp4 with minimal slowdown under parallel decoding — claiming vLLM-level concurrency and far faster throughput than llama-server, even after installing it on Windows 10.
A hands-on benchmark on a single RTX 3090 shows llama.cpp is 2-3x faster and much quicker to first token when the MoE model fits in VRAM, while FreeToken wins 7x on time-to-first-token with 32 concurrent users on gpt-oss-120b (63 GB) that overflows VRAM.
An educational walkthrough of the LLM serving scheduling loop, explaining how inference requests move through waiting queues, batch construction, prefill, decode, and continuous batching to keep GPUs saturated as requests finish at different times.