gpu-inference

Tag

Cards List
#gpu-inference

@mylifcc: If you use llama.cpp to deploy one of the recently popular hybrid-architecture large language models (such as Qwen3.8-27B) in production or locally with multiple concurrent requests, you may encounter an extremely counterintuitive phenomenon: despite having a top-of-the-line H100 GPU installed and running 16 concurrent clients, total throughput is stuck at a measly 58 to…

X AI KOLs Timeline ↗ · yesterday Cached

The author of Tensward used CUPTI-based probes to pinpoint the culprit behind poor throughput when serving hybrid-architecture models (such as Qwen3.8-27B) concurrently with llama.cpp: a synchronous copy of a 643MB state snapshot to host memory after every request. Disabling context checkpoints doubled concurrent throughput.

0 favorites 0 likes
#gpu-inference

Show HN: Open-source, real-time object tracking for live video feeds

Hacker News Top ↗ · 3d ago Cached

tamserve 是一个开源的实时多摄像头目标跟踪服务引擎,让单份 SAM 2 / EdgeTAM 模型同时追踪所有摄像头中的所有目标,显存占用降低约 4 倍、速度提升 11%-24%,且支持按名称跟踪与多 GPU 路由。

0 favorites 0 likes
#gpu-inference

AMD Loom Fun: Added Radeon 9700 support over the weekend. Benchmarked 48 models with an average 42% speedup over llama.cpp.

Reddit r/LocalLLaMA ↗ · 3d ago

AMD Loom, an experimental inference stack with runtime, kernel compiler and server, added Radeon 9700 support and was benchmarked on 48 LLMs, showing an average 42% speedup over llama.cpp in prefill and decode.

0 favorites 0 likes
#gpu-inference

We made a multiplayer game where each player's view is generated by a world model running on their GPU [P]

Reddit r/MachineLearning ↗ · 4d ago Cached

vor.tech demoed a multiplayer game where each player's view is rendered by a world model running locally on their GPU, effectively a game engine that is just a neural net supporting 10 players at once. The model was trained on 3,958 hours of every-POV CS:GO footage plus full game state, and the dataset is being opened up.

0 favorites 0 likes
#gpu-inference

[audio.cpp] SoulX-FlashHead + LFM3-Audio + OWSM v4 = Real-Time Local Avatar Chat

Reddit r/LocalLLaMA ↗ · 4d ago

audio.cpp 0.9.5 adds a real-time local avatar voice chat app in its WebUI, combining SoulX-FlashHead, OWSM v4, and LFM3-Audio to achieve ~850ms speech-to-avatar latency on an RTX 5090.

0 favorites 0 likes
#gpu-inference

@servasyy_ai: https://x.com/servasyy_ai/status/2117209604174311760

X AI KOLs Timeline ↗ · 2026-11-02 Cached

The author tested three local open-source models — DeepSeek-V4.1-Flash-1017, Qwen-3.8-Flash-Next, and Qwen-3.8-27b — on an RTX 5090, an RTX 4090, and dual V100s, comparing the real-world experience gap against cloud models like GPT-5.5 and Opus-4.8 through a Core-300 blind test and 200K+ long-context stress tests.

0 favorites 0 likes
#gpu-inference

Openspline - host a realtime talking avatar on a commercial GPU (90+ FPS on RTX4090)

Reddit r/LocalLLaMA ↗ · 2026-10-17

Openspline 是一个开源的实时数字人头像层,可通过 5 行代码接入现有语音 AI 智能体,在本地 GPU 上生成口型同步的实时视频,RTX4090 上可达 90+ FPS。

0 favorites 0 likes
#gpu-inference

Where does your local vs cloud cost line sit? (3 presets, 3 API rates, the cached-token gap most calculators guess at) :)

Reddit r/LocalLLaMA ↗ · 2026-10-17

A free browser-based LLM cost calculator lets you compare local GPU inference (with RTX 4070 Ti Super / 5090 presets) against cloud API pricing, highlighting that cached-input token rates — often guessed at by other calculators — most determine where the local-vs-cloud crossover sits.

0 favorites 0 likes
#gpu-inference

Jailbroken PS5s can now run local AI models at over 100+ tokens per second — fork of PS5LM goes native, hijacks the CPU, and accesses the GPU and more RAM

Reddit r/artificial ↗ · 2026-10-16 Cached

一位 Reddit 用户发布了 PS5LM 的分支,使 llama.cpp 能在越狱 PS5 上原生运行并利用 GPU 和更多内存,从而在本地运行 Qwen3.8-27B 等大型模型,速度超过 100 tokens/s。

0 favorites 0 likes
#gpu-inference

Why naive cosine routing fails: Using Absorbing Markov Chains over a 768D embedding space for <2ms zero-token intent triage (before waking the LLM)

Reddit r/LocalLLaMA ↗ · 2026-10-16

介绍一种基于768维嵌入空间上的吸收马尔可夫链(Absorbing Markov Chain)的语义分流引擎,通过在60节点领域流形上做随机游走来替代朴素余弦相似度路由,实现<2ms的零token意图分类,从而在唤醒LLM之前完成路由。文中给出JevBench基准测试结果:四层级联系统(Gevva0)在240个条目上达到100%召回、89.17%精度,平均p50延迟46ms,并配备否定/语法、流形邻近OOD和熵与边界三重防护网。

0 favorites 0 likes
#gpu-inference

Someone is running Qwen 27B at ~20 tok/s on a jailbroken PS5's GPU

Reddit r/LocalLLaMA ↗ · 2026-10-13

有人在越狱的 PlayStation 5 GPU 上运行 Qwen 27B 大模型,速度约为每秒 20 个 token,展示了在游戏主机硬件上进行本地 LLM 推理的可行性。

0 favorites 0 likes
#gpu-inference

strix-llama new branch: prompt processing +21–54% over Vulkan and up to 2.7× over stock ROCm kernel at 128k, same outputs quality

Reddit r/LocalLLaMA ↗ · 2026-10-12

rdna35-perf branch of strix-llama.cpp 为 Strix Halo (Radeon 8060S) 带来新的 RDNA3.5 flash attention 与 GEMM 内核优化,prompt processing 比 Vulkan 快 21–54%,在 128k 长上下文下比原版 ROCm 最快达 2.7 倍,且输出质量与原版 ROCm 基本一致。

0 favorites 0 likes
#gpu-inference

Final revision of zzpanic vllm-radiance for 1 x R9700 GPU, qwen3.8-27b

Reddit r/LocalLLaMA ↗ · 2026-10-11

zzpanic shares the final revision of his vllm-radiance build tuned for a single AMD R9700 GPU running qwen3.8-27b, adding startup caching and a KV-offload feature (vram-sysram-disk) with memory thinning to enable multi-agent workloads on one card. He notes the project is winding down as users migrate to StillDeadcode's newer radiance inference engine.

0 favorites 0 likes
#gpu-inference

GitHub - google-ai-edge/ml-drift: GPU-Accelerated AI/ML Inference

Reddit r/LocalLLaMA ↗ · 2026-10-09 Cached

Google AI Edge 开源了 ML Drift,一个高性能、跨平台的 GPU 加速 AI/ML 推理引擎,支持 Android、iOS、Web(WebGPU)和服务器端,可高效运行大型生成式模型。

0 favorites 0 likes
#gpu-inference

Open sourcing Game Summoner, my prompt to game site

Reddit r/LocalLLaMA ↗ · 2026-10-08

作者开源了其 prompt-to-game 网站 Game Summoner 的完整技术栈(MIT 协议),包含基于多智能体设计文档流程的游戏生成架构、SQL 队列 + 自动扩缩容的 GPU worker 系统,以及本地运行脚本。整个系统可在 5090 等本地 GPU 或 RunPod/AWS 托管 GPU 上部署。

0 favorites 0 likes
#gpu-inference

@Lonely__MH: Holy crap?! Is it really that badass? It's said that thanks to "MoE heterogeneous parallelism," you can run the 125B pa…

X AI KOLs Timeline ↗ · 2026-10-05 Cached

Strata is an open-source tool that uses MoE heterogeneous parallelism to run the 125B-parameter Qwen3.8-Flash-Next model locally on a gaming PC with just 12GB VRAM, reaching over 60-94 tokens/s on NVIDIA RTX 5070 and AMD RX 9070 XT GPUs. The project claims full local, private inference with support for NVIDIA and AMD cards on Windows and Linux.

0 favorites 0 likes
#gpu-inference

Tuned/abliterated Qwen3.8-27b into a 24gb card 262k guff using the newest unreleased version of LexiPanel. It's fast with reliable draft acceptance. Made for 7900xtx but should work on whatever 24gb card with this setup and headless. Doesn't get dumber while coding like most of the other fine-tunes.

Reddit r/LocalLLaMA ↗ · 2026-10-02

A quantized and abliterated Qwen3.8-27B CODER model variant (IQ4_XS with imatrix) released in a GGUF build optimized by LexiPanel to fit a single 24 GB GPU with a 262k context window, featuring an MTP head for speculative decoding at 85% draft acceptance on an RX 7900 XTX.

0 favorites 0 likes
#gpu-inference

Ninfer Stahp

Reddit r/LocalLLaMA ↗ · 2026-10-02

A user benchmarks the new Ninfer inference backend on a Blackwell MaxQ GPU, reporting 130-180 t/s on qwen3.8-27b-nvfp4 with minimal slowdown under parallel decoding — claiming vLLM-level concurrency and far faster throughput than llama-server, even after installing it on Windows 10.

0 favorites 0 likes
#gpu-inference

FreeToken vs llama.cpp on one RTX 3090: llama.cpp is 2–3× faster when the MoE fits in VRAM. On gpt-oss-120b (63 GB), FreeToken gets the first token out 7× faster at 32 concurrent users.

Reddit r/LocalLLaMA ↗ · 2026-10-01 Cached

A hands-on benchmark on a single RTX 3090 shows llama.cpp is 2-3x faster and much quicker to first token when the MoE model fits in VRAM, while FreeToken wins 7x on time-to-first-token with 32 concurrent users on gpt-oss-120b (63 GB) that overflows VRAM.

0 favorites 0 likes
#gpu-inference

@_avichawla: The LLM scheduling loop, clearly explained: (bookmark this) Whenever a user sends a prompt to an LLM, the resulting inf…

X AI KOLs Timeline ↗ · 2026-10-01 Cached

An educational walkthrough of the LLM serving scheduling loop, explaining how inference requests move through waiting queues, batch construction, prefill, decode, and continuous batching to keep GPUs saturated as requests finish at different times.

0 favorites 0 likes
Next →
← Back to home

Submit Feedback