Tag
一个跨 4 个产品团队的实践观察：在 12800 个行程预订会话（每会话 5 轮，共 64000 轮）中，单轮相关性高达 97.6%，但约 7.4%（947 个会话）在第 4 轮前因压缩对话状态丢失用户拒绝记录，重新推荐已被拒绝的行程。作者讨论如何划分会话级评分与行为规范，并询问如何区分真正的偏好反转与丢失的对话状态。
The article discusses the challenge of evaluating LLM-based agents that perform multi-step reasoning, noting that scoring only the final output is insufficient because agents may take wrong paths and recover by accident, and raises questions about how to evaluate the trajectory without manual review.