Tag
A position paper arguing that AI development should move beyond treating a single ground truth as definitive, instead modeling the space of plausible human judgments while distinguishing meaningful ambiguity from annotation noise.
Adobe paper proposes evaluating agents against code that recomputes the correct answer from live data, with an AI grader matching human experts 29% better and using 16% fewer tokens than written descriptions on 53 test cases.
The Era by Eon Benchmark generates a complete fictional enterprise with simulators and databases to evaluate LLM agents on enterprise tools, providing exact ground truth for accurate grading.
A developer shares the challenge of creating a gold standard evaluation dataset for an AI product with no users, considering synthetic data generation and adversarial testing to avoid post-launch restructuring.
This position paper argues that ground truth datasets in machine learning are not objective truths but human constructions shaped by choices, and advocates for articulating these choices to improve reliability, transparency, and accountability.
Introduces a generative framework that uses LLM agents to inject behavioral anomalies into simulated trajectories and applies kinematic and map constraints to produce realistic anomalous mobility data with ground truth.