I built EvalSeal: reproducibility receipts for LLM evals
Summary
EvalSeal is an open-source tool that enhances LLM evaluation reproducibility through multiple runs, flip rate measurement, and tamper-evident sealing, highlighting instability in LLM judges.
Similar Articles
How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure
This paper audits the reproducibility of LLM evaluation conclusions, finding that small-sample evaluations are unstable and can mislead, and it recommends reporting additional metrics for reliability.
@DanKornas: LLM behavior changes too easily to rely on a few manual spot checks. DeepEval is an open-source Python framework for ev…
DeepEval is an open-source Python framework for evaluating LLM applications, offering ready-made metrics, Pytest-style workflows, and CI/CD support.
JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation
JudgeArena is an open-source framework that unifies major LLM-judge benchmarks under a single interface, enabling systematic study of judge choices and offering open-model judges that match or outperform closed models, with the ability to simulate LMArena Elo scores.
MAWILE: Multi-Axis Workbench for Inspecting LLM Evaluators
MAWILE is a developer workbench for auditing the sensitivity of LLM judges to perturbations across prompts, rubrics, inputs, and outputs, ensuring robust and meaningful evaluations.
@dair_ai: // Your LLM judge disagrees with the experts // LLM Judges can be tricky to build. Here is an interesting showcasing wh…
The paper presents UPHELD, a benchmark with extensive human annotations for evaluating conversational LLMs, and a Mixture-of-Judges framework that enhances evaluation accuracy by 30%.