@akshay_pachaar: Jev vs. LLM as Judge, clearly explained. Imagine a support agent says, “Done. I issued your refund.” The trace shows th…

X AI KOLs Timeline Tools
ai-evaluation llm-judges dev-tools structured-evaluation open-source

Summary

The article explains the differences between Jev and LLM as Judge for evaluating AI agent responses, highlighting when to use each based on the need for open-ended reasoning versus structured, parallel judgments.

Jev vs. LLM as Judge, clearly explained. Imagine a support agent says, “Done. I issued your refund.” The trace shows that the agent looked up the order, but never completed the refund. An evaluator now needs to decide whether the answer is grounded, honest, and useful. Both an LLM judge and Jev can make those judgments. The difference is how they produce them. → Both need the same evidence The customer request, policy, tool results, and agent answer must be included. A judge cannot evaluate evidence it has not seen. → An LLM judge generates an evaluation You describe the rubric in a prompt and specify the desired output. The model then generates a verdict, score, structured JSON, or written explanation one token at a time. This works well when the evaluation needs detailed reasoning, an explanation, or an answer outside a predefined set. → Jev returns structured decisions You provide the evidence once, then define separate questions with predefined answer formats. For a yes-or-no criterion, Jev returns the probability that the statement is true. For an ordered criterion, it returns a rating and a separate confidence value. The Jev calls these formats Noul and Score, but the core idea is simple. The possible answers are defined before the evaluation runs. Jev evaluates independent questions against the shared evidence in parallel. Your code receives values it can immediately use for thresholds, routing, or review. → The workload determines the better interface Use an LLM judge when you need open-ended reasoning or a written rationale. Use Jev when you repeatedly need focused judgments such as whether an answer is grounded, whether an action claim is honest, or how actionable the response is. Multiple LLM judge calls can also run concurrently. The important distinction is not concurrency between calls. It is token-by-token generation versus parallel structured judgments within one request. That distinction matters at scale. When every agent run needs several checks, a faster and cheaper evaluator can help you inspect more traces, measure more dimensions, and catch failures earlier. Jev does not replace every LLM judge. It gives bounded evaluation workloads a more direct interface. The agent generates the answer. The judge should only generate when the judgment requires it. I also published the complete evaluation workflow using Jev and Comet’s open-source Opik platform. The repository includes frozen agent traces, structured rubrics, result mapping, and the experiment runner. You can explore and run the code here: http://github.com/patchy631/jev-as-judge… I wrote the full breakdown. The article is quoted below.
Original Article
View Cached Full Text

Cached at: 09/24/26, 02:25 PM

Jev vs. LLM as Judge, clearly explained.

Imagine a support agent says, “Done. I issued your refund.”

The trace shows that the agent looked up the order, but never completed the refund. An evaluator now needs to decide whether the answer is grounded, honest, and useful.

Both an LLM judge and Jev can make those judgments. The difference is how they produce them.

→ Both need the same evidence

The customer request, policy, tool results, and agent answer must be included. A judge cannot evaluate evidence it has not seen.

→ An LLM judge generates an evaluation

You describe the rubric in a prompt and specify the desired output. The model then generates a verdict, score, structured JSON, or written explanation one token at a time.

This works well when the evaluation needs detailed reasoning, an explanation, or an answer outside a predefined set.

→ Jev returns structured decisions

You provide the evidence once, then define separate questions with predefined answer formats.

For a yes-or-no criterion, Jev returns the probability that the statement is true. For an ordered criterion, it returns a rating and a separate confidence value.

The Jev calls these formats Noul and Score, but the core idea is simple. The possible answers are defined before the evaluation runs.

Jev evaluates independent questions against the shared evidence in parallel. Your code receives values it can immediately use for thresholds, routing, or review.

→ The workload determines the better interface

Use an LLM judge when you need open-ended reasoning or a written rationale.

Use Jev when you repeatedly need focused judgments such as whether an answer is grounded, whether an action claim is honest, or how actionable the response is.

Multiple LLM judge calls can also run concurrently. The important distinction is not concurrency between calls. It is token-by-token generation versus parallel structured judgments within one request.

That distinction matters at scale. When every agent run needs several checks, a faster and cheaper evaluator can help you inspect more traces, measure more dimensions, and catch failures earlier.

Jev does not replace every LLM judge. It gives bounded evaluation workloads a more direct interface.

The agent generates the answer. The judge should only generate when the judgment requires it.

I also published the complete evaluation workflow using Jev and Comet’s open-source Opik platform. The repository includes frozen agent traces, structured rubrics, result mapping, and the experiment runner.

You can explore and run the code here: http://github.com/patchy631/jev-as-judge…

I wrote the full breakdown. The article is quoted below.

patchy631/jev-as-judge

Source: https://github.com/patchy631/jev-as-judge

Jev-as-a-Judge with Comet Opik

Jev — LLM as judge

A small, auditable example of evaluating refund-support traces with Jev and recording the results as an Opik experiment.

What this project does

  • Replays ten synthetic, frozen support traces, including correct answers, invented policies, false action claims, ambiguous wording, and an evaluator-injection attempt.
  • Runs deterministic checks before paying for semantic evaluation.
  • Evaluates three yes/no criteria and one ordered helpfulness rubric in a single Jev request.
  • Keeps raw model answers, the resolved model identifier, and a rubric version for auditability.
  • Maps values to separate Opik metrics through a custom BaseMetric.
  • Provides a fully offline demo, mocked transport tests, and real-SDK adapter tests.

This is an evaluator tutorial, not a native Jev plugin or production guardrail. It does not issue refunds. Review outcomes are recorded; automated escalation and a background worker are not implemented.

1. Run without an account

From this folder, with Python 3.10 or later:

python -m jev_judge.cli --mode demo
python -m unittest discover -s tests -v

No dependencies or network calls are needed for the core demo. The optional Opik tests are skipped if the SDK is absent. Demo probabilities are hand-authored and explicitly labeled DEMO-hand-authored-not-Jev. They test plumbing, not model accuracy. Do not use demo timings as inference latency.

2. Evaluate the traces with real Jev

Obtain a TypeSafe API key and export it in your shell:

export TYPESAFE_API_KEY="your-key"
export JEV_MODEL="jev-latest"
python -m jev_judge.cli --mode live --output results/live.jsonl

This sends the synthetic request, policy, tool results, and final answer to TypeSafe and may incur API charges. The sample makes nine initial judge requests: the empty-answer case fails a deterministic check. Retries can increase charges. It does not log to Opik.

jev-latest can change. The resolved response model is recorded. Use a supported pinned model identifier for longitudinal comparisons. Never turn an API error into a quality score; failures are marked judge_error, routed to review, and produce a nonzero CLI exit code.

3. Run an Opik experiment

python -m venv .venv
source .venv/bin/activate
pip install -e '.[opik]'
opik configure
export TYPESAFE_API_KEY="your-key"
python -m jev_judge.opik_eval --project jev-support-judge

This separately calls Jev again and creates a new dataset and experiment in your configured Opik workspace. Each invocation uses a unique name to avoid silently mixing fixture revisions. Choose Opik Cloud or self-hosting during configuration. No external account was modified while building this project.

The adapter was tested with Opik 2.2.71 and Python 3.12 using a fake Jev transport. Authenticated Jev calls and actual Opik uploads were not run. The public API contracts were checked against official documentation on September 20, 2026. Dependency versions do not freeze server behavior.

OPIK_TRACK_DISABLE=true is used only in tests. Do not set it for a real experiment. .env.example documents the variables; .env is not automatically loaded.

Understanding the scores

MetricMeaning
groundedEstimated probability that all factual claims have supplied support
addresses_requestEstimated probability that the reply addresses the actual request
action_honestEstimated probability that completed-action claims match successful tool evidence
helpfulnessProbability-weighted rubric value, normalized from 0–2 to 0–1
helpfulness_confidenceProvider’s uncertainty summary for the ordered Score answer
acceptedApplication verdict equals pass, not a correctness guarantee
needs_reviewApplication verdict is uncertain

A Noul value of 0.98 is not “98% of claims are grounded.” It is the model’s probability of a yes to the whole proposition. Score.confidence is not the probability that the judge is correct. The tutorial’s 0.2/0.8 boundaries and 0.6 confidence threshold are illustrative and uncalibrated.

Use your own traces

Pass --data path/to/cases.jsonl to either runner. Each line needs:

{
  "id": "case-001",
  "request": "Can I return this order?",
  "policy": "The authoritative policy relevant to this request.",
  "tool_calls": [{"name": "lookup_order", "result": {"eligible": true}}],
  "final_answer": "The answer being evaluated."
}

The demo mode additionally requires hand-authored demo fields, as in the supplied fixtures. Live mode never sends these to Jev. Optional expected_verdict fields are tutorial annotations, not independently collected human labels; they are excluded from the judge state. Collect real labels before reporting accuracy.

Redact personal data before submission. Restrict Opik retention and access according to your needs. The instruction boundary in the rubric reduces ambiguity but is not a proven defense against prompt injection. Test hostile trace content separately.

Files

  • article.md: publication draft, with source links and code excerpts.
  • jev_judge/rubric.py: atomic questions and allowlisted state projection.
  • jev_judge/client.py: documented HTTP contract, validation, timeout, bounded retries.
  • jev_judge/core.py: deterministic checks, score mapping, verdict policy.
  • jev_judge/cli.py: local demo/live runner and JSONL audit results.
  • jev_judge/opik_eval.py: live custom metric and experiment entry point.
  • tests/: network-free tests, including the real Opik SDK with mocked calls.
  • scripts/build_figures.py: reproducible architecture exports.
  • scripts/build_reading_copy.py: standalone HTML article builder.
  • VALIDATION.md: verification scope and publication checks.

Rebuild the graphics and reading copy

pip install -e '.[figures]'
python scripts/build_figures.py
npm install --no-save playwright
npx playwright install chromium
node scripts/render_illustrations.cjs
python scripts/build_reading_copy.py

Alternatively, install weasyprint and Poppler (pdftoppm), then run python scripts/render_illustrations.py in place of the Playwright steps. The four illustrations are editable HTML/CSS in assets/; their PNG exports support the Markdown article.

Before sharing on GitHub

Review the article, choose your repository URL, and choose a license before publishing. No license is imposed here. Commit the source, data, tests, article, and figures; do not commit .env, credentials, environments, or private result files. The included CI workflow runs only offline tests.

Sources

Similar Articles