i built a benchmark to test whether LLMs can understand and create jokes
Summary
A developer built a benchmark called lolbench to evaluate LLMs on understanding and creating jokes, revealing models are strong at explaining real jokes but struggle with failed ones, and incorporating human voting for preference testing.
Similar Articles
Humor Arena: Which LLM is the funniest?
The article presents a comparison of 20 LLM versions on humor generation using a fine-tuned judge, with Fable 5 scoring highest in a joke benchmark.
HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models
HumorRank introduces a tournament-based leaderboard using pairwise evaluations and Bradley-Terry MLE to rank LLMs on humor generation, showing humor quality depends on comedic mastery rather than scale.
Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges
This survey examines computational humor understanding in multimodal LLMs, covering methods, datasets, evaluation protocols, and challenges such as shortcut-prone evaluation and weak evidence grounding.
@dair_ai: // Your LLM judge disagrees with the experts // LLM Judges can be tricky to build. Here is an interesting showcasing wh…
The paper presents UPHELD, a benchmark with extensive human annotations for evaluating conversational LLMs, and a Mixture-of-Judges framework that enhances evaluation accuracy by 30%.
Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges
A comprehensive survey on multimodal humor understanding using large language models, covering methods, datasets, evaluation protocols, and challenges in interpreting humor in memes, cartoons, and comics.