Found a great production use case for Jev.

I used Jev to organize ~2.3K AI research papers.

The total cost was $0.14, and it took about 83 seconds.

The process:

The papers already had old tags, which I ran through a previous open model (DeepSeek V4 Flash).

However, I wasn’t confident in the classifications, and I didn’t want to spend more on tokens unless I spent time tuning it into a good LLM classifier via few-shot (more expensive). Too tedious, too costly, and unsustainable.

Luckily for us, we now have Jev to help us with organizing papers better.

So Jev first went through all the papers and retagged them. It agreed with 75% of the previous tags.

Jev found about 579 high-confidence topic changes.

I evaluated reliability by manually labeling 30 disagreements and accepted all of them. I was astonished by Jev’s classification capabilities. We applied and verified all changes in production.

I’m much more satisfied with the classifications, but I think there is still room for improvement.

Check out the papers here: https://academy.dair.ai/papers

I will experiment with Jev more. The takeaway is that pipelines can be significantly improved by carefully combining System One and System Two models.

Jev clearly unlocks more interesting ways to organize papers and offer a more useful discovery layer for research papers. More updates on that soon.

AI Papers of the Week | DAIR.AI Academy

Source: https://academy.dair.ai/papers

Skill-based Agentic Evaluation for Real-time Data Science Tasks

Storing a fixed reference answer for every eval case breaks when the underlying data changes daily, so Adobe researchers write each reference answer as a Python function that runs against the live system at evaluation time. An LLM judge then splits the agent’s response and the computed answer into atomic facts and scores precision and recall regardless of output format, raising agreement with expert labels from an MCC of 0.331 to 0.427 while cutting token cost per case by 16%. A judge given no ground truth scored an MCC of -0.379, which is worse than chance.

02Agents

Verifiable Social Reasoning for LLM Assistants

People ask assistants for social advice constantly, and the assistant only hears the user’s version of events, which makes it hard to check whether it read the situation correctly. Google Research builds that ground truth by simulation, with a target agent holding a hidden motive while a user agent relays events to the assistant, which then has to infer the motive. Across 24k human annotations validating the simulations and 12 LLMs tested, biased framing from the user shifted the assistant’s answer, and longer conversations with room for clarifying questions did not reliably help.

03Evaluation

Mo’ Models, Mo’ Problems: How to best select model pools when designing Multi-Agent Systems

NVIDIA compared eight strategies for choosing which models go into a multi-agent system, based on size, accuracy, answer diversity, and error diversity, across routing, majority vote, and LLM-as-judge setups on hard science benchmarks. Larger pools of different open models raised the theoretical best-case accuracy while achieved accuracy often fell below the single best model in the pool, and using several copies of one model worked better. Majority vote over the best single model raised HLE accuracy from 29.4% to 32.2%, so measure what another model adds before putting it in the router.

04Agents

Divide, Consult, Conquer: Capability Laundering Through Aligned LLMs

Safety evaluations usually ask whether a model refuses a harmful request. Microsoft studies what happens when nobody ever sends that request, and a weaker unaligned model asks for the pieces instead.

05Safety

Stellar Colosseum: A Many-Agent Harness for Long-Horizon Research in Mathematics and Theoretical Computer Science

Long mathematical proofs break the usual agent loop, since a single wrong step early on invalidates everything after it. Google Research built a many-agent harness for this setting, and it produced new results on open problems from FOCS and JMLR papers.

07Agents

GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents

The standard way to compare task agents is to have an LLM user simulator talk to each one and an LLM judge score the transcript. Amazon audits that gate against verifiable rewards across 25 agents from six providers, and finds two specific failures.

08Evaluation

Is Bash All You Need? An Empirical Study of Tool Interfaces for Enterprise Digital Worker Agents

Deciding which tools to hand an enterprise agent usually means writing typed tool definitions for every system it touches. Microsoft compared five tool interfaces head to head, and the plainest option won.

09Agents

Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models

Byte-level language models drop the tokenizer and read raw bytes, which removes a preprocessing step that no one likes but also costs accuracy at small scale. Meta studies what happens as compute grows, distilling 1B byte students from token teachers on up to 1 trillion bytes, and the ordering flips.

10Training