llm-routing

Tag

Cards List
#llm-routing

@rohanpaul_ai: A model decision in under 90 ms is fast enough to put a router in front of almost any LLM call. @NaceAI just launched D…

X AI KOLs Following ↗ · yesterday Cached

Nace AI launched Drex Router, a 12B Gemma-based model that selects the best LLM/SLM and reasoning effort per agent step in 57–87 ms, scoring 87.6% on 1,400 exam questions versus rival Jev Router's 86.4% at a fraction of the cost, and returns per-candidate probabilities with explanations.

0 favorites 0 likes
#llm-routing

FinVeriQA: Financial Research Agents for Question Answering: Routing, Retrieval, MCP Tool Use, and Provenance-Aware Evidence Verification

arXiv cs.CL ↗ · 2d ago Cached

FinVeriQA is an open-source multi-agent financial research prototype over SEC filings and live market data that shows a compact typed router beats LLMs, Python beats LLM arithmetic, and provenance-aware verification independent of writer evidence is essential — shared-evidence checks catch far fewer errors than checks against original passages.

0 favorites 0 likes
#llm-routing

Switching LLMs mid-conversation is a ski-rental problem (30 minute read)

TLDR AI ↗ · 3d ago Cached

Edgee's Agent Gateway frames mid-conversation LLM switching as a ski-rental problem, combining a difficulty tier map with a switching policy that accounts for prompt-cache rebuild costs, and learns both via weekly replays of past traffic. The post explains why routing must optimize not just which model handles a turn, but when a session should switch models without losing its warm cache.

0 favorites 0 likes
#llm-routing

When the route hops host mid session the cache starts over, keeping it on one

Reddit r/openclaw ↗ · 2026-11-14

Prompt caches live on the backend that served the last request, so route hops re-bill long prefixes as fresh input — e.g. a 700k Kimi K3 context costs ~$2 per call on a hop vs ~$0.20 with a warm cache. The post discusses pinning providers, single-host multi-model setups, and error-only fallbacks to keep caches warm.

0 favorites 0 likes
#llm-routing

@rohanpaul_ai: New Meta paper finds that AI agents improve more from their past attempts when each new problem is matched to notes fro…

X AI KOLs Timeline ↗ · 2026-11-02 Cached

A new Meta paper shows AI agents learn better from past attempts when experiences are routed into per-problem-type memory clusters rather than a single shared history, boosting performance by 15-30% of the max score while cutting API costs by 65-72%.

0 favorites 0 likes
#llm-routing

@dair_ai: Brilliant paper on agent memory from FAIR at Meta. (bookmark it) A single global history is the usual way to let agents…

X AI KOLs Following ↗ · 2026-10-31 Cached

FAIR at Meta 提出 Routed Continual Learning:将过往轨迹按簇划分、每个簇搭配固定记忆库中的一种记忆系统,并由 LLM 路由器决定复用或新建簇,在交错任务流上相对全局 ICL 历史提升 15–30% 归一化奖励并降低 65–72% API 成本。

0 favorites 0 likes
#llm-routing

@_avichawla: Prefix-aware LLM routing, clearly explained! (trick to get first LLM token 170x sooner) Production prompts start with t…

X AI KOLs Timeline ↗ · 2026-10-29 Cached

文章清晰解释了前缀感知的 LLM 路由机制:通过按块哈希提示词并维护块到 worker 的索引,让请求路由到持有共享前缀 KV 缓存的 GPU,实测在 8 个 vLLM Pod(16 张 H100)上将 p90 首 token 延迟从 92.5 秒降至 0.54 秒。

0 favorites 0 likes
#llm-routing

SkillEval: Learning Interpretable Ability Profiles of LLMs via Cognitive Diagnosis Models

arXiv cs.CL ↗ · 2026-10-22 Cached

The paper introduces SkillEval, a framework that combines LLM-discovered interpretable skills with cognitive diagnosis models to produce fine-grained ability profiles of LLMs across benchmarks, improving on average-score evaluation and enabling LLM routing and adaptive testing.

0 favorites 0 likes
#llm-routing

Notion Auto

Product Hunt ↗ · 2026-10-20 Cached

Notion Auto がリニューアルし、エージェントハーネスが再構築されて 24 時間以上のタスク実行が可能に。タスクごとに最適なモデルへ自動ルーティングし、Opus 5.5 相当の性能を約 90% 低コストで提供するほか、ChatGPT プラン接続、Notion MCP の刷新、新しいクレジット制を導入した。

0 favorites 0 likes
#llm-routing

Chance-Constrained Personalized LLM Routing with Online Preference Learning

arXiv cs.LG ↗ · 2026-10-19 Cached

该论文提出一种基于约束上下文老虎机的个性化 LLM 路由框架,通过 primal-dual GLM-UCB 算法在线学习用户偏好,在最小化推理成本的同时以高概率保证响应质量满意度。

0 favorites 0 likes
#llm-routing

Ramp’s $1.5 Million Week of Spending Leads to Glass AI Tool

Reddit r/artificial ↗ · 2026-10-14 Cached

Ramp 宣布将内部构建的 AI 工具 Glass 对外开放给其他公司使用。Glass 源于 Ramp 一周内烧掉 150 万美元 AI token 的经历,旨在将企业专属信息转化为全组织可用的工具,并通过 Ramp Router 自动选择合适模型,使账单下降 30%。

0 favorites 0 likes
#llm-routing

Navige

Product Hunt ↗ · 2026-10-14 Cached

Navige 是一款在 AI agent 运行时降低成本的工具,通过 Sufficiency Intelligence 将请求智能路由到更小的模型、缓存重复读取并精简工具输出,同时提供预算设置、失控循环检测和敏感操作审批功能。

0 favorites 0 likes
#llm-routing

@FeitengLi: Teams from Tsinghua and others have released TokenRouter, which helps small and large AIs collaborate on writing the sa…

X AI KOLs Timeline ↗ · 2026-10-12 Cached

清华大学等团队发布了 TokenRouter,一个支持小模型与大模型在单次回答中进行 token 级协作的推理服务引擎,通过减少模型切换的等待开销,在双卡 A100 上将 8 路并发的总吞吐从约 135 tokens/s 提升到 372 tokens/s(2.76 倍)。代码与论文均已开源公开。

0 favorites 0 likes
#llm-routing

Evaluating Escalation Signals for LLM Routing: Targets, Controls, and Five Ways to Fool Yourself

arXiv cs.AI ↗ · 2026-10-08 Cached

Cisco researchers evaluate semantic entropy as an escalation signal for routing queries from small to large LLMs across three benchmarks, showing it works on GSM8K but can be misleadingly matched by naive difficulty heuristics. The paper's main contribution is a checklist of evaluation pitfalls—question-only baselines, ambiguous success metrics, benchmark saturation, and hidden sampling costs—for assessing routing signals.

0 favorites 0 likes
#llm-routing

TokenRouter: Efficient Serving System for Token-Level LLM Routing

Hugging Face Daily Papers ↗ · 2026-10-08 Cached

TokenRouter 是一个面向 token 级 LLM 路由的高效推理服务系统,采用「请求中心编程、模型中心执行」架构与延迟批调度策略,实现较现有系统 2.01–64.15 倍的解码吞吐提升。

0 favorites 0 likes
#llm-routing

Liquid Inference

Product Hunt ↗ · 2026-10-07 Cached

Liquid Inference 是一款 LLM 路由器,让多家模型提供商针对每次 prompt 动态竞价,从而为用户提供最低推理成本,兼容 Claude Code、Cursor 等主流 agentic 编码工具,支持数百个开源与闭源模型。

0 favorites 0 likes
#llm-routing

Together Link

Product Hunt ↗ · 2026-10-05 Cached

Together AI launches Together Link, a developer tool that connects coding agents to open models on Together AI, allowing developers to keep their existing workflows while reducing model costs.

0 favorites 0 likes
#llm-routing

Dynamic LLM Routers are Often Misguided

arXiv cs.AI ↗ · 2026-10-05 Cached

The paper analyzes six commercial dynamic LLM routers across 14 settings and finds none outperforms a router that randomly selects between two well-chosen models at matched cost, attributing failures to difficulty blindness, length reversal, semantic matching, and roster suboptimality — patterns the standard evaluation objective actually rewards.

0 favorites 0 likes
#llm-routing

FlexRouter: Learning Complementary Model Sets for Flexible LLM Routing

arXiv cs.LG ↗ · 2026-10-02 Cached

FlexRouter 提出一种显式建模模型互补性的 LLM 路由框架,将路由建模为基于覆盖的子集选择问题,并用 Determinantal Point Processes 对模型能力与冗余进行联合建模,同时通过边缘化失败集合的目标函数直接优化答案覆盖率。在 RouterEval 基准上,该方法在域内与域外任务上以更低冗余实现了更高的覆盖率,且推理成本灵活可控。

0 favorites 0 likes
#llm-routing

Collective Bias Mitigation via Model Routing and Collaboration

Hugging Face Daily Papers ↗ · 2026-10-02 Cached

The paper introduces Collective Bias Mitigation (CBM), a framework that reduces LLM bias by fine-tuned model behavior and knowledge sharing among diverse LLMs, using Debating and Committee routing topologies to balance bias reduction and inference cost.

0 favorites 0 likes
Next →
← Back to home

Submit Feedback