llm-efficiency

Tag

Cards List
#llm-efficiency

When Do Typed Decision Models Work in Transportation? Evidence from Travel Choice and Traffic Signal Control

arXiv cs.CL ↗ · 3d ago Cached

This paper evaluates typed decision models on travel-mode choice prediction and traffic signal control, finding they offer lower latency and cost than chat-based LLMs but lag behind traditional discrete choice models and strongest baselines in accuracy, with performance sensitive to input framing and task difficulty.

0 favorites 0 likes
#llm-efficiency

Co-Designing Sparse Attention for Efficient Long-Context Inference (17 minute read)

TLDR AI ↗ · 4d ago Cached

NVIDIA's developer blog presents a practical primer on co-designing model-level sparse attention for efficient long-context inference, using an FP8 kernel to sweep indexer design knobs and compare token-level vs. block-level core attention at 128K token contexts.

0 favorites 0 likes
#llm-efficiency

@akshay_pachaar: The next step after Jev: A decision model like Jev bills and computes the full context on every call, so a 40-message c…

X AI KOLs Following ↗ · 2026-11-02 Cached

Levanto Labs introduces Sage2, a streaming decision model that computes each token of a conversation only once, reusing cached context so each new message adds only its own token cost. It delivers typed answers with calibrated probabilities and claims 3x faster decisions on progressive context, plus extraction that is 166x cheaper than Opus 5.5.

0 favorites 0 likes
#llm-efficiency

@skytearss: Karpathy fired shots (former OpenAI researcher and former head of AI at Tesla): "Agents aren't magic; they're large-scale distillation. 99.99% of your model's capacity is wasted on data it doesn't need at all." In a 16-minute conversation, he laid out the tech stack that actually works: small models, the right tools, a…

X AI KOLs Timeline ↗ · 2026-11-01 Cached

In a 16-minute conversation, Karpathy, formerly of OpenAI and former head of AI at Tesla, said that agents are not magic but the result of large-scale distillation. He advocates building a truly effective agent stack using small models paired with the right tools and closed-loop feedback, rather than relentlessly pursuing larger models.

0 favorites 0 likes
#llm-efficiency

DeFT-RAG: Decomposing Multi-Hop Questions into Self-Contained Sub-Questions with Reusable Answers

Hugging Face Daily Papers ↗ · 2026-10-30 Cached

DeFT-RAG introduces a RAG framework that decomposes multi-hop questions into self-contained sub-questions with reusable cached answers, offloading reasoning to rule-based updates and a lightweight decision model. It achieves the best accuracy among thirteen baselines on MuSiQue, 2WikiMultiHopQA and HotpotQA while reducing LLM calls.

0 favorites 0 likes
#llm-efficiency

Can Test-Time Memory Evolution Outperform Test-Time Training for Scientific Discovery?

arXiv cs.AI ↗ · 2026-10-26

论文提出 Test-Time Memory Evolution (TTME),一种即插即用的测试时适应方法,通过持续更新外部记忆来指导科学发现而无需更新模型权重。实验表明 TTME 在五个数学任务上全面优于测试时训练 (TTT),同时将计算成本降低 82.8–84.6% FLOPs 和 47.7–58.4% 运行时间。

0 favorites 0 likes
#llm-efficiency

How Much Does the Hint Matter? Scope Conditions for Hindsight Self-Distillation in Financial Reasoning

Hugging Face Daily Papers ↗ · 2026-10-26 Cached

该论文提出 Fin-SD(hindsight self-distillation),一种无需外部教师模型的本地化自蒸馏方法,用于提升小型开放权重模型的多步金融推理能力;实验发现增益主要来自监督位置而非提示内容,且与基于完整参考解答的自蒸馏精度相当但少生成 9.5% token。

0 favorites 0 likes
#llm-efficiency

Terminal X matches senior investor judgment 88.6% with GPT-6.1

OpenAI Blog ↗ · 2026-10-20 Cached

Terminal X 的 Capital Memory 系统利用 GPT-6.1 Sol 与 GPT-6 Luna 将资深投资者的判断编码为可审计的代理工作流,Eval Agent 以 88.6% 的匹配度复现资深投资人判断,并在同等流程下将单次查询成本降低约 30%。

0 favorites 0 likes
#llm-efficiency

@dair_ai: Interesting paper from Microsoft on context-guided model routing. They found a nice trick to achieve cost-aware model r…

X AI KOLs Timeline ↗ · 2026-10-16 Cached

微软的研究论文提出在用户轮次边界进行上下文感知的模型路由,发现将会话历史提供给路由器可在相同卸载率下提升多轮电子表格智能体的任务通过率,并分析了提示缓存切换的不对称成本与廉价模型的成本优势衰减。

0 favorites 0 likes
#llm-efficiency

Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules

arXiv cs.CL ↗ · 2026-10-15 Cached

Titan-Llama 提出一种后训练方法,通过分层记忆架构和测试时参数更新,将预训练注意力模型改造为具备外部神经记忆模块的长上下文高效推理模型,在 QuALITY 和 LongHealth 上显著提升准确率并降低推理延迟与显存占用。

0 favorites 0 likes
#llm-efficiency

Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming

arXiv cs.LG ↗ · 2026-10-15 Cached

提出LayerTrim——一种无需训练的结构化剪枝方法,通过保留注意力机制和解码器深度、选择性修剪冗余层的MLP特征,显著缓解了推理语言模型在长链推理任务上的性能崩溃,在Qwen3-14B上实现1.24倍预填充加速和19.7%峰值显存降低。

0 favorites 0 likes
#llm-efficiency

Trajectory-Level Speculation with Semantic Verification for Tool-Using Agents

Hugging Face Daily Papers ↗ · 2026-10-15 Cached

SpecAgent 提出一种协作协议,让弱模型起草并执行只读工具调用,强模型通过单次 prefill 并行验证并修复,在 τ²-Bench 上几乎保持强模型成功率的同时将延迟与内存流量减半。

0 favorites 0 likes
#llm-efficiency

Asana says cache changes cut a browser agent's cost 76x

Reddit r/artificial ↗ · 2026-10-09

Asana says it reduced a browser-agent workflow's model cost by 76x (from $36.21 to $0.47 per run) and cut runtime from 22.5 to about 4 minutes by stabilizing page history to enable prompt caching and pruning screenshots in batches, rather than changing the underlying model.

0 favorites 0 likes
#llm-efficiency

REMORY: Learning Residual Memory for Context Compaction

arXiv cs.CL ↗ · 2026-10-09 Cached

Remory 提出一种神经记忆网络,用有限的软记忆 token 补充文本摘要,使冻结的 LLM 在仅用 5.2% 输入位置的情况下逼近全上下文表现,并在 BrowseComp 和 Terminal-Bench 2.1 等长时程 agent 基准上带来一致提升、显著减少重复工具输出和错误。

0 favorites 0 likes
#llm-efficiency

Sparse Attention Is Matrix Approximation, Not Choosing from a Bag of Values

arXiv cs.CL ↗ · 2026-10-09 Cached

论文提出 MASA,主张稀疏注意力应被视为矩阵逼近问题而非简单挑选大注意力值,并通过闭式打分度量每个稀疏单元对矩阵乘积误差的削减,在多种稀疏注意力方法和 LLM 上取得一致精度提升。

0 favorites 0 likes
#llm-efficiency

Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training

arXiv cs.LG ↗ · 2026-10-08 Cached

Q-PACE 提出一种基于二阶敏感度模型的动态精度分配方法,用于量化感知训练(QAT),在预训练和监督微调实验中(模型规模达4B参数)相比现有混合精度训练方案取得了更优效果,并在更低内存预算下达到可比拟的损失。研究还发现量化敏感度可通过深度和层类型高度预测,从而支持低成本的低频重校准。

0 favorites 0 likes
#llm-efficiency

LRCC: Generalizing Low-Rank Compression with Conditional Computation

arXiv cs.CL ↗ · 2026-10-08 Cached

LRCC proposes token-dependent compute allocation by training lightweight routers over nested low-rank paths in frozen pretrained LLMs, improving perplexity and downstream accuracy over static low-rank compression at matched active-parameter budgets and decoding latency on Llama and Qwen models.

0 favorites 0 likes
#llm-efficiency

Why isn't the industry freaking out about DeepSeek 4.1 Flash?

Hacker News Top ↗ · 2026-10-08 Cached

作者基于一个月的重度使用体验,认为 DeepSeek 4.1 Flash 在编程任务上已与 Claude Opus 等前沿模型难以区分,但成本低几个数量级,业界却未因此震动。文章指出 KV 缓存缩减约 437 倍使全天会话成本不到一美元,预示便宜且'足够好'的中国模型正在改变开发者的工作方式。

0 favorites 0 likes
#llm-efficiency

@mem0ai: https://x.com/mem0ai/status/2107873474517889201

X AI KOLs Following ↗ · 2026-10-07 Cached

Jev-Mem:一篇来自UT Dallas的论文提出用TypeSafe AI的Jev结构化输出模型替代LLM来处理智能体记忆系统中的大量小型决策(如记忆分类、关联判断、检索停止条件),仅在最终生成答案时才调用LLM,从而显著降低token开销。

0 favorites 0 likes
#llm-efficiency

Beyond the Context Window: An Adaptive Entropy-Based Routing Framework for Hybrid Retrieval and Long-Context Language Models

arXiv cs.CL ↗ · 2026-09-30 Cached

The paper proposes EDAR, an entropy-driven adaptive routing framework that decides at inference time whether to answer a query with retrieval-augmented generation or escalate it to full long-context processing. EDAR retains 97.4% of the pure long-context baseline's accuracy while cutting token cost by 70.7%, escalating only 18.2% of queries on LongBench v2 and Infinity-Bench.

0 favorites 0 likes
Next →
← Back to home

Submit Feedback