model-inference

Tag

Cards List
#model-inference

@Lonely__MH: Holy crap?! Is it really that badass? It's said that thanks to "MoE heterogeneous parallelism," you can run the 125B pa…

X AI KOLs Timeline ↗ · 2026-10-05 Cached

Strata is an open-source tool that uses MoE heterogeneous parallelism to run the 125B-parameter Qwen3.8-Flash-Next model locally on a gaming PC with just 12GB VRAM, reaching over 60-94 tokens/s on NVIDIA RTX 5070 and AMD RX 9070 XT GPUs. The project claims full local, private inference with support for NVIDIA and AMD cards on Windows and Linux.

0 favorites 0 likes
#model-inference

@Gorden_Sun: Strata: Running a 100-billion-parameter model with just 12GB VRAM On a regular computer with 12GB VRAM, locally running…

X AI KOLs Timeline ↗ · 2026-10-04 Cached

Strata 项目实现在 12GB VRAM + 64GB RAM 的普通电脑上本地运行 125B 参数的 Qwen3.8-Flash-Next 模型,通过让常用专家模型常驻 GPU、全模型放入系统内存,并在 GPU、内存与 SSD 之间协同推理和投机采样来突破显存限制。

0 favorites 0 likes
#model-inference

Recovering Off-Policy Supervision for Speculative Decoding

arXiv cs.CL ↗ · 2026-10-01 Cached

This paper proposes a rollout-based training framework (Anchor-Label Relabelling and In-Rollout Anchors) to recover full supervision for speculative decoding block drafters trained on off-policy corpora, boosting greedy accepted length by up to 36.5% over DFlash without modifying the training text.

0 favorites 0 likes
#model-inference

Announcing our partnership with OpenAI (4 minute read)

TLDR AI ↗ · 2026-09-30 Cached

Baseten announces a partnership with OpenAI, making Baseten-powered open models available to OpenAI customers so organizations can build multi-model agentic coding workflows, routing tasks to the best-fit model across intelligence, cost, latency, and capacity.

0 favorites 0 likes
#model-inference

vulkan: fuse qwen4exp's SCALE -> SIGMOID -> SCALE -> hc_post chain by fxgsell · Pull Request #29520 · ggml-org/llama.cpp

Reddit r/LocalLLaMA ↗ · 2026-09-28 Cached

This pull request fuses a chain of operations in the Vulkan backend for the qwen4exp model to optimize inference performance in llama.cpp.

0 favorites 0 likes
#model-inference

The model fills the blank. Nobody gave it that authority.

Reddit r/AI_Agents ↗ · 2026-09-18

The article discusses the risk of AI models inferring missing data in agent systems, potentially leading to unauthorized executions, and proposes separating decision authority from models to ensure only explicitly declared instructions are followed.

0 favorites 0 likes
#model-inference

Dual AMD Radeon AI Pro R9700 or dual NVIDIA or RTX 3090.

Reddit r/LocalLLaMA ↗ · 2026-09-15

The author is building a dual GPU system for running large language models, evaluating AMD Radeon AI Pro R9700 versus NVIDIA RTX 3090 options while aiming to reduce AI subscription costs.

0 favorites 0 likes
#model-inference

Don't Sleep on EXL3 Quants

Reddit r/LocalLLaMA ↗ · 2026-08-30

The author shares their experience running a 30B parameter model with EXL3 quantization on a 12GB VRAM GPU, achieving efficient performance and speed for coding and agent tasks.

0 favorites 0 likes
#model-inference

It's unbelievable! I used the mmap function in llama.cpp to fit Qwen3.8-Flash-Next IQ3_XSS into 16G+64G RAM, and the speed still reached 26t/s.

Reddit r/LocalLLaMA ↗ · 2026-08-28

A user successfully used the mmap function in llama.cpp to fit the Qwen3.8-Flash-Next IQ3_XSS model into 16GB+64GB RAM, achieving a speed of 26 tokens per second, which outperforms a larger non-MOE 30B model.

0 favorites 0 likes
#model-inference

Spent a day seeing how far extreme MoE models can be pushed on a 4070 Ti + 32GB RAM. Kimi K3, DeepSeek V4 Flash, and Qwen3.5-122B results + research paper🔧

Reddit r/LocalLLaMA ↗ · 2026-08-25

This article details experiments with extreme Mixture-of-Experts models on consumer hardware using a custom runtime CRANE V2, and presents a research paper with results from Kimi K3, DeepSeek V4 Flash, and Qwen3.5-122B models.

0 favorites 0 likes
#model-inference

@no_stp_on_snek: Check out Buun's work, he cookin.

X AI KOLs Following ↗ · 2026-08-19 Cached

A user highlights Buun's work on optimizing AI models, achieving high-speed inference of Qwen 3.6 on a single 3090 GPU and developing DFlash2 for Qwen 3.8.

0 favorites 0 likes
#model-inference

Qwen 3.8 27b with DSH(DeepSeek Harness) is Amazing!! Experiences so far and perfomance.

Reddit r/LocalLLaMA ↗ · 2026-08-16

A user shares positive experiences using the Qwen 3.8 27b model with DeepSeek Harness, praising its stability and long-context handling, but mentions speed limitations and hopes for future model releases.

0 favorites 0 likes
#model-inference

@TheAhmadOsman: Prediction We’re gonna get Kimi K3 equivalent intelligence running on a single RTX PRO 6000 in less than 18 months How?…

X AI KOLs Following ↗ · 2026-08-15 Cached

A Twitter user predicts that AI intelligence comparable to Kimi K3 will run on a single RTX PRO 6000 GPU within 18 months, later noting that Opus 4.6 Max quality already fits on a single RTX 5090.

0 favorites 0 likes
#model-inference

@victormustar: I deployed FREE public endpoint for Qwen3.8-27B no token needed, OpenAI-compatible, light rate limiting. Powered by Hug…

X AI KOLs Following ↗ · 2026-08-15 Cached

A free public endpoint for the Qwen3.8-27B AI model has been deployed, offering an OpenAI-compatible API with vision support, tool calls, and a large context window, powered by Hugging Face Inference Endpoints for at least 72 hours.

0 favorites 0 likes
#model-inference

@TheAhmadOsman: I built this AI Server in 2023 with models like Qwen 3.8 27B and beyond in mind

X AI KOLs Timeline ↗ · 2026-08-15

A user built an AI server in 2023 designed to run models such as Qwen 3.8 27B.

0 favorites 0 likes
#model-inference

I implemented the YOLO26n model inference from scratch using ARM64 Assembly Language (No framework) [P]

Reddit r/MachineLearning ↗ · 2026-07-26

The author implemented YOLO26n model inference from scratch using ARM64 assembly language without any external frameworks, demonstrating low-level AI inference techniques.

0 favorites 0 likes
#model-inference

@DanKornas: Building video-generation workflows is easier when inference, model configs, and integration paths live in one place. L…

X AI KOLs Timeline ↗ · 2026-07-24 Cached

LTX-Video is an open-source Python repository by Lightricks for generating and conditioning videos locally using LTX-Video models, with support for text/image inputs, multi-condition workflows, and integration with ComfyUI and Diffusers.

0 favorites 0 likes
#model-inference

@Youssofal_: 72+ TPS on Qwen 3.6 27B on a Macbook pro M5 max. MTPLX V2 out now! The fastest way to run models on MLX.

X AI KOLs Following ↗ · 2026-07-07 Cached

MTPLX V2 is released, claiming 72+ tokens per second on Qwen 3.6 27B running on a Macbook Pro M5 Max via MLX.

0 favorites 0 likes
#model-inference

@yishan: Meta really fumbled this guy.

X AI KOLs Timeline ↗ · 2026-07-07 Cached

John Carmack comments on memory cost and capacity issues for AI accelerators, noting that model inference can have deterministic memory access patterns, contrasting with game rendering.

0 favorites 0 likes
#model-inference

DFlash support merged into llama.cpp

Reddit r/LocalLLaMA ↗ · 2026-06-28 Cached

DFlash support has been merged into llama.cpp, improving inference performance for compatible models.

0 favorites 0 likes
Next →
← Back to home

Submit Feedback