Open-source procurement rubric for agentic AI vendors, I scored 5 of them and want feedback on the methodology

Reddit r/AI_Agents Tools
open-source procurement agentic-ai rubric vendor-evaluation documentation-scoring

Summary

The author created an open-source rubric tool to evaluate agentic AI vendor documentation on tool-call correctness, loop termination, and multi-step state coherence, scored five vendors (Anthropic, OpenAI, LangGraph, Sierra, Salesforce), and requests feedback on methodology and potential bias toward public documentation depth.

I built a tool that scores agentic AI vendor documentation against a 15-question rubric covering tool-call correctness, loop termination, and multi-step state coherence. Drop a folder of a vendor's public docs in, get back a structured report with verbatim evidence quotes for every score. I ran 5 well-known vendors through it using only their public material: \- Anthropic Claude Agent SDK: 3.27 / 5 \- OpenAI Agents SDK: 3.20 / 5 \- LangGraph: 2.67 / 5 \- Sierra: 2.47 / 5 \- Salesforce Agentforce: 2.27 / 5 Salesforce came last despite having the broadest enterprise compliance footprint (SOC 2, ISO 27001, PCI-DSS, HIPAA, FedRAMP, GDPR). The rubric penalized them for not publishing quantitative red-team data, not exposing buyer-visible step / cost ceilings, and keeping the Atlas Reasoning Engine source closed. Foundation model platforms won because they publish more "show your work", turns out to matter more than compliance breadth for procurement. Where I want pushback: 1. The vendor folders (vendors/\*/) — am I being fair to each, or missing things their docs actually cover 2. The "show your work" bias — is rewarding public documentation depth the right signal for procurement, or am I penalizing legitimately-proprietary vendors? Github link is in the comments.
Original Article

Similar Articles

Benchmarking Agentic Review Systems

arXiv cs.AI

This paper benchmarks agentic review systems for peer review, evaluating open-source and proprietary systems on research papers. The best configuration achieves 83.0% pairwise accuracy and catches 71.6% of injected errors, but user feedback highlights issues with false positives and nitpicks.