open-ended-tasks

FrontierCS 2 introduces a benchmark of 48 open-ended computer science R&D tasks for LLM agents, placing them inside real research repositories with hidden evaluation suites; the best agent, Claude Opus 5.5, passes only about 58% of its runs.

华为openJiuwen团队提出Reward Engine，通过构建由测试用例和评分细则组成的冻结RewardPack，为开放式工具智能体提供证据驱动的奖励信号，用于自修订与强化学习，在WorkBuddy Bench Office和Claw-Eval上显著提升通过一致性。

本文系统分析了LLM评估中rubric设计的机制，涵盖标准组织、操作化与分数聚合，并提出ARTS-Hybrid方法（自适应树构建+语义优先的响应校准细化），在评估性能和rubric质量上均有提升，且生成的rubric可作为模型优化的有效信号。

该论文提出Station——一个多智能体开源环境，用于评估AI代理进行开放式科学发现的能力。通过添加Supervisor和Meta Reflection机制，Station在重现ICLR oral论文发现的基准上平均达到62.7%，显著超越Codex Multiagent-v2（15.4%）和AI Scientist-v2（14.4-20.6%）。

This paper introduces Elo-per-token analysis to study how LLM agents allocate test-time compute, revealing that agents initially outperform independent sampling but slow down over time, with parallel sessions offering performance gains.

Zero-WAM is a causal video-action model that enables zero-shot robotic manipulation of unseen tasks by conditioning on in-context human video guidance, with the HumanGen dataset and a future-chunk prediction objective to improve generalization.

This paper analyzes LLM outputs over three years using open-ended tasks and creativity assessments, finding a statistically significant decrease in diversity that suggests creative substance is converging across models, which could diminish human agency in co-creative work.

A factorial experiment reveals that persona detail does not monotonically increase LLM opinion diversity; interaction architectures explore non-overlapping opinion regions; low-cost interventions like temperature scaling have negligible effects.

Introduces CriPO, a method that enhances rubric-based reinforcement learning for LLMs via on-policy self-distillation, addressing unexplored and suppressed criteria. Experiments on medicine and science benchmarks show stronger final performance with roughly 2x fewer optimization steps.

This paper introduces Experiential Learning (EL), a method that repurposes an LLM-as-a-Judge into an LLM-as-a-Coach to provide rich textual feedback instead of scalar rewards, improving performance and generalization on open-ended non-verifiable tasks.

VIBE is a framework that evaluates generative bias in Large Audio-Language Models using open-ended tasks with human-recorded speech, revealing systematic biases triggered by gender and accent cues.

This paper proposes a prompt-level reward specification framework that separates reward specification from computation, constructing reusable task-adaptive rubrics and executable constraint checkers offline to produce a hybrid reward for open-ended post-training without requiring human annotations or separate reward models.

SCOPE is a self-play framework for open-ended tasks that co-evolves a Challenger and Solver policy, achieving up to +10.4 points on benchmarks without external supervision.

ARES proposes a framework for automatically constructing rubric-based RL data from pretraining documents, generating question-answer pairs and weighted rubrics to enable instance-level reward supervision for open-ended LLM responses, outperforming existing methods on multi-dimensional open-ended tasks.

This paper introduces POW3R, a policy-aware rubric reward framework for reinforcement learning with verifiable rewards (RLVR). It shows that static rubric aggregation misallocates learning signal, and POW3R achieves faster convergence and better performance across multiple settings.

Cornell researchers propose POP, a self-play framework that lets an LLM generate its own rubrics and training pairs for open-ended tasks, boosting Qwen-2.5-7B on healthcare QA, creative writing and instruction following without human labels.