@akshay_pachaar: Jev for RAG, clearly explained! Hybrid search gives you a shortlist. It does not decide which passages contain evidence…

X AI KOLs Following Tools
rag retrieval-augmented-generation reranking llm-judge answerability observability open-source

Summary

介绍 Jev 如何作为 RAG 管道中的“裁判”，在检索与生成之间评估候选段落的相关性、可回答性和潜在提示注入，并附上基于 Comet Opik 的开源 Jev-as-a-Judge 项目。

Jev for RAG, clearly explained! Hybrid search gives you a shortlist. It does not decide which passages contain evidence, which are merely adjacent, or whether the evidence is strong enough to answer. That missing judgment is where Jev fits. Jev sits between retrieval and generation. It does not replace BM25, embeddings, or the LLM. It evaluates the candidates they produce before those candidates enter the context window. → Retrieve wide Combine dense and keyword search, then merge the results with reciprocal rank fusion. This gives Jev a broad candidate pool, such as the top 20 passages shown in the visual. Retrieval still sets the ceiling. If the right passage is missing from this shortlist, Jev cannot recover it. → Judge every candidate together Send the query as Jev’s state. For each candidate, ask a typed yes-or-no question such as “Does C7 help answer this query?” Jev evaluates every question in one packed request and returns a calibrated probability for each passage. This avoids making a separate model call for every query-passage pair. → Let code apply the threshold Your application compares each probability with a threshold. Candidates above it continue to the LLM. Everything below it is removed before generation. Jev makes the fuzzy judgment. Code remains responsible for the actual decision. → Gate the entire answer The same request can check whether the retained passages make the query answerable. It can also flag signs of prompt injection inside a candidate. If answerability falls below the threshold, the application can skip the LLM and return “not in the documents.” The injection score should remain a filtering signal, not a security boundary. The result is a cleaner division of work. Hybrid search retrieves broadly. Jev reranks, filters, and decides whether sufficient evidence exists. The LLM writes only from the passages that survive. Jev’s value here is not simply moving passages up or down a list. It turns relevance into an explicit probability that your application can inspect, threshold, and act on. To summarise: - Retrieval finds the candidates. - Jev decides what deserves context. - The LLM writes the grounded answer. ---- I also built an open-source project showing how to use Jev as a judge for AI observability with Comet Opik. It evaluates support traces for groundedness, request coverage, action honesty, and helpfulness, then records the results as an auditable experiment. You can explore the project here: http://github.com/patchy631/jev-as-judge… My article on how Jev works is quoted below.
Original Article
View Cached Full Text

Cached at: 09/30/26, 04:09 PM

Jev for RAG, clearly explained!

Hybrid search gives you a shortlist. It does not decide which passages contain evidence, which are merely adjacent, or whether the evidence is strong enough to answer.

That missing judgment is where Jev fits.

Jev sits between retrieval and generation. It does not replace BM25, embeddings, or the LLM. It evaluates the candidates they produce before those candidates enter the context window.

→ Retrieve wide

Combine dense and keyword search, then merge the results with reciprocal rank fusion. This gives Jev a broad candidate pool, such as the top 20 passages shown in the visual.

Retrieval still sets the ceiling. If the right passage is missing from this shortlist, Jev cannot recover it.

→ Judge every candidate together

Send the query as Jev’s state. For each candidate, ask a typed yes-or-no question such as “Does C7 help answer this query?”

Jev evaluates every question in one packed request and returns a calibrated probability for each passage. This avoids making a separate model call for every query-passage pair.

→ Let code apply the threshold

Your application compares each probability with a threshold. Candidates above it continue to the LLM. Everything below it is removed before generation.

Jev makes the fuzzy judgment. Code remains responsible for the actual decision.

→ Gate the entire answer

The same request can check whether the retained passages make the query answerable. It can also flag signs of prompt injection inside a candidate.

If answerability falls below the threshold, the application can skip the LLM and return “not in the documents.” The injection score should remain a filtering signal, not a security boundary.

The result is a cleaner division of work.

Hybrid search retrieves broadly. Jev reranks, filters, and decides whether sufficient evidence exists. The LLM writes only from the passages that survive.

Jev’s value here is not simply moving passages up or down a list. It turns relevance into an explicit probability that your application can inspect, threshold, and act on.

To summarise:

  • Retrieval finds the candidates.
  • Jev decides what deserves context.
  • The LLM writes the grounded answer.

I also built an open-source project showing how to use Jev as a judge for AI observability with Comet Opik.

It evaluates support traces for groundedness, request coverage, action honesty, and helpfulness, then records the results as an auditable experiment.

You can explore the project here: http://github.com/patchy631/jev-as-judge…

My article on how Jev works is quoted below.

patchy631/jev-as-judge

Source: https://github.com/patchy631/jev-as-judge

Jev-as-a-Judge with Comet Opik

Jev — LLM as judge

A small, auditable example of evaluating refund-support traces with Jev and recording the results as an Opik experiment.

What this project does

  • Replays ten synthetic, frozen support traces, including correct answers, invented policies, false action claims, ambiguous wording, and an evaluator-injection attempt.
  • Runs deterministic checks before paying for semantic evaluation.
  • Evaluates three yes/no criteria and one ordered helpfulness rubric in a single Jev request.
  • Keeps raw model answers, the resolved model identifier, and a rubric version for auditability.
  • Maps values to separate Opik metrics through a custom BaseMetric.
  • Provides a fully offline demo, mocked transport tests, and real-SDK adapter tests.

This is an evaluator tutorial, not a native Jev plugin or production guardrail. It does not issue refunds. Review outcomes are recorded; automated escalation and a background worker are not implemented.

1. Run without an account

From this folder, with Python 3.10 or later:

python -m jev_judge.cli --mode demo
python -m unittest discover -s tests -v

No dependencies or network calls are needed for the core demo. The optional Opik tests are skipped if the SDK is absent. Demo probabilities are hand-authored and explicitly labeled DEMO-hand-authored-not-Jev. They test plumbing, not model accuracy. Do not use demo timings as inference latency.

2. Evaluate the traces with real Jev

Obtain a TypeSafe API key and export it in your shell:

export TYPESAFE_API_KEY="your-key"
export JEV_MODEL="jev-latest"
python -m jev_judge.cli --mode live --output results/live.jsonl

This sends the synthetic request, policy, tool results, and final answer to TypeSafe and may incur API charges. The sample makes nine initial judge requests: the empty-answer case fails a deterministic check. Retries can increase charges. It does not log to Opik.

jev-latest can change. The resolved response model is recorded. Use a supported pinned model identifier for longitudinal comparisons. Never turn an API error into a quality score; failures are marked judge_error, routed to review, and produce a nonzero CLI exit code.

3. Run an Opik experiment

python -m venv .venv
source .venv/bin/activate
pip install -e '.[opik]'
opik configure
export TYPESAFE_API_KEY="your-key"
python -m jev_judge.opik_eval --project jev-support-judge

This separately calls Jev again and creates a new dataset and experiment in your configured Opik workspace. Each invocation uses a unique name to avoid silently mixing fixture revisions. Choose Opik Cloud or self-hosting during configuration. No external account was modified while building this project.

The adapter was tested with Opik 2.2.71 and Python 3.12 using a fake Jev transport. Authenticated Jev calls and actual Opik uploads were not run. The public API contracts were checked against official documentation on September 20, 2026. Dependency versions do not freeze server behavior.

OPIK_TRACK_DISABLE=true is used only in tests. Do not set it for a real experiment. .env.example documents the variables; .env is not automatically loaded.

Understanding the scores

MetricMeaning
groundedEstimated probability that all factual claims have supplied support
addresses_requestEstimated probability that the reply addresses the actual request
action_honestEstimated probability that completed-action claims match successful tool evidence
helpfulnessProbability-weighted rubric value, normalized from 0–2 to 0–1
helpfulness_confidenceProvider’s uncertainty summary for the ordered Score answer
acceptedApplication verdict equals pass, not a correctness guarantee
needs_reviewApplication verdict is uncertain

A Noul value of 0.98 is not “98% of claims are grounded.” It is the model’s probability of a yes to the whole proposition. Score.confidence is not the probability that the judge is correct. The tutorial’s 0.2/0.8 boundaries and 0.6 confidence threshold are illustrative and uncalibrated.

Use your own traces

Pass --data path/to/cases.jsonl to either runner. Each line needs:

{
  "id": "case-001",
  "request": "Can I return this order?",
  "policy": "The authoritative policy relevant to this request.",
  "tool_calls": [{"name": "lookup_order", "result": {"eligible": true}}],
  "final_answer": "The answer being evaluated."
}

The demo mode additionally requires hand-authored demo fields, as in the supplied fixtures. Live mode never sends these to Jev. Optional expected_verdict fields are tutorial annotations, not independently collected human labels; they are excluded from the judge state. Collect real labels before reporting accuracy.

Redact personal data before submission. Restrict Opik retention and access according to your needs. The instruction boundary in the rubric reduces ambiguity but is not a proven defense against prompt injection. Test hostile trace content separately.

Files

  • article.md: publication draft, with source links and code excerpts.
  • jev_judge/rubric.py: atomic questions and allowlisted state projection.
  • jev_judge/client.py: documented HTTP contract, validation, timeout, bounded retries.
  • jev_judge/core.py: deterministic checks, score mapping, verdict policy.
  • jev_judge/cli.py: local demo/live runner and JSONL audit results.
  • jev_judge/opik_eval.py: live custom metric and experiment entry point.
  • tests/: network-free tests, including the real Opik SDK with mocked calls.
  • scripts/build_figures.py: reproducible architecture exports.
  • scripts/build_reading_copy.py: standalone HTML article builder.
  • VALIDATION.md: verification scope and publication checks.

Rebuild the graphics and reading copy

pip install -e '.[figures]'
python scripts/build_figures.py
npm install --no-save playwright
npx playwright install chromium
node scripts/render_illustrations.cjs
python scripts/build_reading_copy.py

Alternatively, install weasyprint and Poppler (pdftoppm), then run python scripts/render_illustrations.py in place of the Playwright steps. The four illustrations are editable HTML/CSS in assets/; their PNG exports support the Markdown article.

Before sharing on GitHub

Review the article, choose your repository URL, and choose a license before publishing. No license is imposed here. Commit the source, data, tests, article, and figures; do not commit .env, credentials, environments, or private result files. The included CI workflow runs only offline tests.

Sources

Similar Articles

@yibie: https://x.com/yibie/status/2102914640707567798

X AI KOLs Timeline

This article tests the application of the Jev model in RAG retrieval, evaluating its effects on accelerating retrieval and reducing costs. Results show advantages in reranking and judging answerability.