Tag
LangChain is hosting a virtual workshop on Oct 28th about LLM-as-a-judge reliability, trajectory scoring, and wiring evaluations into development loops to catch regressions before shipping.
该论文提出 MISCONCEPTDIALOGUE 框架，发现基准式精确匹配评分与人工校准的 LLM-as-judge 在相同对话上相差 39.2 个百分点，并指出 LLM 数学导师几乎不会依据所提供的诊断信息调整教学。
该论文提出 DiagSafe 诊断式评估框架，用提示级评分标准（prompt-level rubrics）替代二元安全标签，构建了中英双语、基于 84 类安全分类体系的基准，使 LLM-as-a-judge 与人工标注的一致性达到 95.34%（Cohen's κ=0.864），并揭示了主流模型在中文与多轮对话场景下的独特漏洞。
We present AuditKIT, a source-available evaluation library that unifies benchmark accuracy, judge-based assessment, runtime performance, tool-use behavior, and retrieval quality in a single result representation with direction-aware metric contracts and extensible backends. Seven case studies show how answer extraction, runtime behavior, and metric selection can change evaluation outcomes for LLMs and agentic systems.
该论文提出了一种名为"语义重构博弈"的免训练评估方法，用于更准确地评估 LLM 越狱攻击的成功率（ASR）。研究发现仅更换评判 LLM 就可导致 ASR 报告值相差高达 82 个百分点，而新方法通过将合规性判断分解为受限语义比较，显著降低了相对于人工标注的平均误差。
A talk shared by LangChain: every agent builder should be able to answer three questions—where the agent fails, how to reliably reproduce the failure, and how to prevent it from happening again. The talk introduces a systematic approach to improving agents through production tracing, specialized judge models, and a "world spec" simulation environment.
This paper shows that a single-character colon edit in a candidate answer can sharply increase a decision model judge's false acceptance rate (from 1% to 26%) depending on how the structured judging request is presented, demonstrating that judging competence can coexist with presentation-dependent vulnerabilities.
Walmart Global Tech 研究员探讨在电商搜索页面中使用语言模型离线评估页面级布局决策（如插入次级产品栈的位置），结果表明基于表示的方法在预测用户参与度方面持续优于直接提示判断。
This paper tests whether stacked runtime gates (deterministic rules plus LLM judges) for agent tool calls fail independently, finding that rule+judge layers compose to ~1.9-2.1 multiplication-equivalent layers while two LLM judges only compose to ~1.2-1.5 due to correlated failures.
MIT 的 Neil Thompson 提出 O*NET-BENCH 审计套件，评测 33 种 LLM judge 配置，发现排序一致性虽高但接受率估计偏差极大（3.0%–97.9% vs 工人实际的 61.1%），表明仅凭排序一致性选择 judge 不足以支撑职业层面的 AI 测量。
A Red Hat Developer article benchmarks TypeSafe AI's Jev "System One" decision models against LLM-as-a-judge guardrails and traditional zero-shot text classifiers, concluding that decision models don't clearly outperform the established alternatives despite their schema-safe, low-latency, zero-shot benefits.
The paper introduces ARCagent, an adaptive retrieval calibration agent for clinical question answering in ME/CFS, where diagnostic guidelines actively contradict each other. It contributes a conflict-aware retrieval re-ranking pipeline and an LLM-as-Judge benchmark, achieving 95.3% accuracy and outperforming base LLMs.
@wonder is building an AI agent to automate meal planning and delivery, integrating LangSmith for debugging and scoring using LLM-as-judge.
This paper systematically evaluates pseudo-labeling for adapting pretrained ASR models like Whisper and Qwen3-ASR to noisy police audio, introducing an LLM-as-a-judge filtering method and a cross-model paradigm to reduce word error rates.
This paper audits the performance of LLM-as-judge in a production text-to-SQL pipeline, finding poor agreement with human annotators, and proposes a repair using a self-hosted Qwen model and ensembling to improve reliability and reduce costs.
This paper introduces two taxonomies—decomposing retrieval-stage errors into five quality dimensions and verifier-reasoning errors into six steps—induced automatically via LLM-as-Judge to diagnose where retrieval-based medical factuality evaluation fails. Stress-tests across 4 retrieval methods and 6 frontier verifiers show these failure modes are fundamental limitations of the retrieve-then-verify paradigm, not artifacts of outdated systems.
This blog post discusses how to benchmark LLMs for specific production use cases, proposing a comprehensive evaluation process that includes task-specific metrics, trade-offs between quality and cost, and analysis of failure cases.
CrossAudit introduces a git-native protocol for auditing autonomous research pipelines using AI agents from different vendors to ensure oversight and integrity in agentic science.
This paper presents a unified benchmark comparing outline-driven frameworks for long-form generation with LLMs, evaluating outlines independently and finding that performance varies with framework-granularity match, supporting the decoupling of outline and writing stages.
The paper proposes information satisfaction as a reader-centered axis for summarization evaluation, showing that current metrics fail to capture user-specific informational needs and agree poorly with human judgment.