mmlu

Alibaba's Qwen3.8-27B model's safety restrictions have been completely removed, with the V2 version achieving zero rejection, and the MMLU score improved to 86.3%, using complementary ablation blending to achieve uncensored and stronger performance.

This paper investigates how the tone of prompts affects both the accuracy and the inference cost (output token consumption) of large language models, finding that output token length can vary by up to 44.3% across tones while accuracy changes are smaller, and identifies optimal tones for different models.

This paper reports on a collaboration between the European Commission's Directorate-General for Translation and the European Master's in Translation network to localize the MMLU dataset into 11 European languages, creating a more inclusive benchmark for LLM evaluation while providing authentic training for translation students.

This paper analyzes sources of ranking uncertainty in LLM benchmarks like MMLU, proposing modifications to hypothesis tests for constructing rank confidence intervals, and shows that variability across subjects is substantial.

A tiny ~10K parameter router called tinyrouter learns which open model to use per question on MMLU, outperforming individual models by optimizing allocation.

A study on MMLU accuracy shifts across prompt formats found that the headline "93% answer-flip" rate was partly due to parsing artifacts. The author recommends pre-registering parse-failure handling and reporting parse-corrected metrics to separate genuine model sensitivity from parser brittleness.

This paper introduces a controlled protocol to evaluate answer stability in large language models by challenging correct answers with plausible counterarguments, revealing large variation in flip rates across models that accuracy metrics alone do not capture. The authors release the protocol, challenge records, and a curated MaxFlip challenge set to support stability evaluation.

HRM-Text introduces a Hierarchical Recurrent Model that decouples computation into slow and fast layers, enabling efficient pretraining from scratch on only 40 billion tokens and a $1,500 budget, achieving competitive performance with larger models.

TransformerLab is an open-source platform that orchestrates GPUs across clouds and provides pre-built templates for AI training and evaluation workflows like LoRA, DPO, and MMLU.

Introduces Capability Conditioned Scaffolding, a framework for LLM collaboration that adapts intervention based on user expertise domains to prevent Professional Domain Drift, with pilot evaluation on MMLU subsets.

This study presents a 33-model atlas analyzing domain-level metacognitive monitoring in frontier LLMs using MMLU benchmarks, revealing significant variations in confidence calibration across different knowledge domains that are obscured by aggregate metrics.