Tag
Strata is an open-source tool that uses MoE heterogeneous parallelism to run the 125B-parameter Qwen3.8-Flash-Next model locally on a gaming PC with just 12GB VRAM, reaching over 60-94 tokens/s on NVIDIA RTX 5070 and AMD RX 9070 XT GPUs. The project claims full local, private inference with support for NVIDIA and AMD cards on Windows and Linux.
Strata 项目实现在 12GB VRAM + 64GB RAM 的普通电脑上本地运行 125B 参数的 Qwen3.8-Flash-Next 模型,通过让常用专家模型常驻 GPU、全模型放入系统内存,并在 GPU、内存与 SSD 之间协同推理和投机采样来突破显存限制。
This paper proposes a rollout-based training framework (Anchor-Label Relabelling and In-Rollout Anchors) to recover full supervision for speculative decoding block drafters trained on off-policy corpora, boosting greedy accepted length by up to 36.5% over DFlash without modifying the training text.
Baseten announces a partnership with OpenAI, making Baseten-powered open models available to OpenAI customers so organizations can build multi-model agentic coding workflows, routing tasks to the best-fit model across intelligence, cost, latency, and capacity.
This pull request fuses a chain of operations in the Vulkan backend for the qwen4exp model to optimize inference performance in llama.cpp.
The article discusses the risk of AI models inferring missing data in agent systems, potentially leading to unauthorized executions, and proposes separating decision authority from models to ensure only explicitly declared instructions are followed.
The author is building a dual GPU system for running large language models, evaluating AMD Radeon AI Pro R9700 versus NVIDIA RTX 3090 options while aiming to reduce AI subscription costs.
The author shares their experience running a 30B parameter model with EXL3 quantization on a 12GB VRAM GPU, achieving efficient performance and speed for coding and agent tasks.
A user successfully used the mmap function in llama.cpp to fit the Qwen3.8-Flash-Next IQ3_XSS model into 16GB+64GB RAM, achieving a speed of 26 tokens per second, which outperforms a larger non-MOE 30B model.
This article details experiments with extreme Mixture-of-Experts models on consumer hardware using a custom runtime CRANE V2, and presents a research paper with results from Kimi K3, DeepSeek V4 Flash, and Qwen3.5-122B models.
A user highlights Buun's work on optimizing AI models, achieving high-speed inference of Qwen 3.6 on a single 3090 GPU and developing DFlash2 for Qwen 3.8.
A user shares positive experiences using the Qwen 3.8 27b model with DeepSeek Harness, praising its stability and long-context handling, but mentions speed limitations and hopes for future model releases.
A Twitter user predicts that AI intelligence comparable to Kimi K3 will run on a single RTX PRO 6000 GPU within 18 months, later noting that Opus 4.6 Max quality already fits on a single RTX 5090.
A free public endpoint for the Qwen3.8-27B AI model has been deployed, offering an OpenAI-compatible API with vision support, tool calls, and a large context window, powered by Hugging Face Inference Endpoints for at least 72 hours.
A user built an AI server in 2023 designed to run models such as Qwen 3.8 27B.
The author implemented YOLO26n model inference from scratch using ARM64 assembly language without any external frameworks, demonstrating low-level AI inference techniques.
LTX-Video is an open-source Python repository by Lightricks for generating and conditioning videos locally using LTX-Video models, with support for text/image inputs, multi-condition workflows, and integration with ComfyUI and Diffusers.
MTPLX V2 is released, claiming 72+ tokens per second on Qwen 3.6 27B running on a Macbook Pro M5 Max via MLX.
John Carmack comments on memory cost and capacity issues for AI accelerators, noting that model inference can have deterministic memory access patterns, contrasting with game rendering.
DFlash support has been merged into llama.cpp, improving inference performance for compatible models.