Agent and Model Evaluations in Gemini Enterprise Agent Platform are now Generally Available (GA)!

Measure, test, and monitor your AI agents in dev and production—using one engine with consistent metric definitions.

20+ pre-built & custom metrics Adaptive rubrics & simulation tools Built-in online monitoring

Get started: https://goo.gle/4w5dU6b

Agent and Model Evaluations in Gemini Enterprise Agent Platform are now GA

Source: https://developers.googleblog.com/agent-and-model-evaluations-in-gemini-enterprise-agent-platform-are-now-ga/ Agent quality must be measured during development against the cases you wrote, and after launch against the tasks the agent actually performed.Agent and model evaluationsin Agent Platform are now generally available, letting you measure and compare agents and models at both stages, on one engine with consistent metrics. When you use consistent quality scoring on local experiments and live traffic, a drift in production points to a problem with the agent rather than with the way it was measured.

What’s generally available

• **Metrics.**Start from more than 20pre-built metricsspanning quality, safety, grounding, agent tool use and trajectory, and reference-based scoring for tasks like summarization and translation.Adaptive rubricstailor the judging criteria to each case instead of applying one brittle llm-as-judge prompt across inputs that don’t deserve the same questions. You can also define your own code-based orLLM-as-a-judge metricsand store them in one versioned, org-wide place so scoring stays consistent and comparable over time.
• **Experiments.**Run them client- or server-side. Server-side keeps every artifact in Cloud Storage, so runs are auditable and reproducible. Experiments integrate withcase generationto bootstrap an evaluation dataset, auser simulatorto play out multi-turn cases without scripting each reply, and anenvironment simulatorto stand in for the systems the agent calls, so you can emulate a failing or slow backend without affecting production.
• Online monitors and telemetry integrations.Continuous evaluation on live production trafficgrades the traces you already collect and producesscore-over-time charts and drift alerts, without needing to set up custom data processing pipelines.

You can reach evaluations from theAgent Platform SDK,agents-cli,theEvaluationsection in Agent Platform Google Cloud console, and directly fromADK.

Experiments

The primary unit of work in evaluation is theexperiment, a dataset of eval cases and a set of metrics and scores to run and review as you iterate. The evaluation service comes with a flexible UI to define metrics, kick off new runs, and review the results, all the way down to a single failure, where you can open the agent’s full trace and session log to see exactly what happened.

Sorry, your browser doesn’t support playback for this video

Evals Worksheet — review and run agent evaluations.

You can run experimentslocallyfor fast iterations, or, if your agent is already deployed on Agent Platform with telemetry enabled, you can grade existingsessions and traces, or run the agent with the user simulator enabled to create traces before your users produce them. Every experiment artifact is stored transparently inCloud Storage, so you can version and audit your runs for compliance and posterity.

For large evaluation jobs, the system supportsissue clustering: it groups eval failures into interpretable, actionable clusters against your owntaxonomyof failure reasons. If you haven’t developed a taxonomy yet, you can use a pre-built one we have for adaptive rubrics, which covers the common ways agents go wrong.

Evaluation metrics

More than 20pre-built metricsship with the service.Computation-based metricsscore deterministically against a ground-truth reference:ROUGEfor summarization,BLEU,MetricX, andCOMETfor translation, exact match for extractive QA.

Beyond those, anadaptive rubricis an advanced LLM-judge metric workflow co-developed with our research partners at Google DeepMind. It creates case-specificpass/fail tests(rubrics) from the eval case definition, the developer instruction, and the tool declarations. It then grades the traces against these rubrics, providing the verdict and rationale per rubric.

Adaptive rubric with criteria generated per evaluation case.

We designed and calibrated variants for what you usually want to know about an agent.Task Successgrades goal fulfillment across a conversation from observable outcomes and confirmations in the agent’s responses.Tool Use Qualityevaluates tool selection, argument correctness, and schema compliance.Safetyscores the response against content policies spanning hate speech, harassment, dangerous content, sexually explicit material, and PII, returning the policies violated.Trajectory Quality, Final Response Quality, Hallucination, Grounding, and image and video quality metrics are also available. Natural-language guidelinessteer rubrics generationtoward criteria you specify, with the resulting rubric group reviewable and reusable across agents.

When the pre-built metrics don’t fit, you can bring your own. Acode-based metricis a Python function, covering exact text matches, JSON-shape checks, and anything else expressible in code. AnLLM-as-a-judge metriccarries your own criteria, rating scale, and judge model. When running locally you can use any model from any provider, and server-side evaluation runs support anyModel Gardenmodel, including all Gemini and Anthropic models.

However a metric is defined, it lands in the same versioned, org-wide registry, running unchanged in offline experiments and online monitors. We’re actively expanding coverage to more specialized tasks and input modalities, and we welcome your feedback.

Online monitors and telemetry integrations

An agent that’s green across your whole test suite can still drift a week after launch, when faced with inputs nobody wrote a case for. That’s not a testing failure so much as a fact of production where real tasks are bigger and stranger than any suite you can write up front.

The metrics you built when developing your agent should keep running after your agent launches. If you alreadycollect traces and sessions through Cloud Trace, you can evaluate them directly with just one click in the traces UI. Better still is to set up anonline monitorto grade the live traffic as it arrives. To avoid grading every request and keep costs manageable, you can use the built-in sampling and targeted filters. When monitors are active you can see scores over time in built-in dashboards and even set up drift alerts for email, Slack, or other channels.

Online monitors show eval scores over on production traces.

Case generation and simulation

Every evaluation needs test cases to run against, and writing them all by hand is slow and tends to miss the non-obvious ones. The evaluation service can generate them for you, with acase generator, auser simulator, and anenvironment simulator.

Thecase generatorseeds synthetic eval cases from the agent’s instructions and tools, so you’re not starting from a blank page or capping your coverage at what you thought to type out. For ADK agents, it can pair with auser simulatorto reach the scenarios that are hard to produce by hand: you define a persona and a short conversation plan, and the simulator plays that user across a full multi-turn exchange, so you can evaluate real back-and-forth without scripting each reply.

Theenvironment simulatorstands in for the systems your agent calls. Point it at a tool, give it the response you want — mocked data, a forced error, added latency — and it intercepts that call during the run, so you can test how the agent handles a failing or slow backend without touching production.

Pricing and availability

Agent and model evaluations are generally available. For supported regions and enterprise security features seeregions and security features table. You pay standard rates for the model calls behind LLM-as-a-judge and the other model-based metrics – plus Cloud Storage for the artifacts a server-side run keeps. Code-based and computation metrics add no additional cost. Your datasets and traces always stay in your project.

Get started

Evaluation meets you in whatever tool you already build agents with. It’s part of theAgent Platform SDK, with the same operations available over theREST APIfor the languages and pipelines the SDK doesn’t reach.

If you work from the command line,agents-climakes eval a first-class command next to the ones you already use to deploy agents and inspect telemetry (ADK-Python agents today). You can even run the loop straight from your coding agent: a reusable skill walks Claude Code and similar tools throughthe full agent-quality flywheel. For teams building onADK, evaluation is built into the framework — define eval sets and run them against your agent locally as you develop, including inside pytest for CI. And when you’d rather review a run without writing code, theEvals Worksheetweb UI is the grid covered above.

• Run your first evaluation:Tutorial — Agent Platform SDK
• Evaluate an agent end-to-end:Evaluate agents with the GenAI Client
• Run the loop from your coding agent:the agent-quality flywheel

We can’t wait to see what you ship.