Tag
A new technique called JEVfire enables existing LLMs like Qwen to behave more like Jev by modifying decision-making processes without retraining, resulting in significantly faster JSON generation and enabling local AI agents to run efficiently on consumer hardware.
CoVeR is a coverage-based routing method that reduces LLM verifier calls by 62-68% in agentic retrieval systems while maintaining accuracy on multi-hop QA benchmarks.
ClusterFewshot is a novel method for improving few-shot demonstration selection in LLM workflows by integrating semantic clustering and utility scoring, which reduces optimization costs and enhances accuracy in DSPy-based pipelines.
OptiSkill is a framework that builds a hierarchical and evolving SkillBank for LLM-based operations research modeling, improving formulation accuracy by storing and validating reusable skills across problems.
This article provides 10 steps to optimize AI agent decision-making by using the langchain-typesafe tool, reducing LLM call costs to $0.042 per million tokens and latency to 70-500ms while preventing hallucinations.
D-Quant is a flexible KV cache quantization framework using driftable entropy coding to convert variable-length representations into fixed-size bitstreams, enabling efficient parallel dequantization in attention kernels and achieving up to 7x memory reduction and 3.5x throughput improvement while preserving near BF16 performance.
COBRA-Skills introduces a method for optimizing agent skills using contextual bandits and evolutionary operators, achieving 55-58% lower optimization costs across multiple AI agent benchmarks.
NVIDIA's paper introduces a method to transfer KV cache between AI models, allowing target models to skip prefill entirely and achieve 2.7 to 25x faster conversions, which could improve efficiency in multi-model applications.
This megathread compiles a list of the latest open-source projects, research papers, and hardware innovations focused on optimizing inference, efficiency, and accessibility for open-source LLMs and related technologies.
Apple's Lily engine optimizes on-device LLM inference for Apple silicon by leveraging unified memory and hardware, outperforming MLX-LM, and is tuned for the Qwen3.6-35B-A3B model's architecture.
OpenFreedom's Dynamic Thinking feature reduces task costs by 86% and execution time by 80% compared to OpenClaw, with 74% fewer LLM calls, while maintaining quality.
LLM4LLM introduces a deployment-aware closed-loop optimization framework to bridge kernel benchmarks and real LLM inference, achieving up to 6.98x speedups on H100 GPUs.
The author describes a token routing system to optimize LLM usage, reducing API costs by up to 70% through techniques like cost-based routing, prompt caching, and output discipline.
LambLabs introduces Woolly, a post-training technique that accelerates Qwen3-8B by 2-3x for math and coding tasks, enabling it to run on their chip, with a live demo available.
Inco AI announces DFlash 2, a next-generation decoding technique that achieves up to 4.6× speed increase for Qwen3.8-27B on M5 Max MacBook Pro, improving efficiency without altering output.
A Stanford and MIT research paper shows that optimizing the Python harness around LLMs can yield up to a 6x performance gap without changing model weights, with systems like Meta-Harness automating context evolution.
This paper proposes a reinforcement learning method for bootstrapping niche multilingual code translation with execution-based supervision, introducing a new benchmark HumanEval-X++ and demonstrating significant improvements over baselines using Qwen-3.5 models.
A developer created a task-aware GGUF quantization pipeline that uses tensor-level bit allocation to improve Gemma 4 12B Q3 coding performance by 8.55% over a hand-tuned imatrix while increasing model size by only 0.119%.
Explains why agent API bills grow quadratically with context length because each turn re-reads the full history, and shares practical techniques like expiring tool results, shrinking tool schemas, and compacting context to cut costs.
Proposes a two-stage structured pruning framework for LLMs that jointly optimizes latency and model size using multi-objective depth pruning and parallel Bayesian optimization, achieving favorable trade-offs for edge deployment.