Tag
The author distills decisions from LLMs into small heads on a frozen ModernBERT encoder, improving accuracy on the banking77 task from 67.5% to 76.0% by fixing the option budget, and provides insights on token reading and teacher agreement.
This paper identifies a vocabulary gap as the root cause why advanced encoders like ModernBERT underperform in learned sparse retrieval, and proposes Vocabulary Transfer (VT), a model-agnostic framework that migrates encoders to sparse-friendly vocabularies, achieving state-of-the-art on the BEIR benchmark.
This paper introduces BERTomelo, a next-generation monolingual encoder pre-trained for Portuguese using the ModernBERT architecture, achieving superior performance on downstream tasks like STS and NER compared to previous Portuguese and multilingual models.
This paper explores domain adaptation of ModernBERT models in the legal domain by further pre-training on US court opinions, achieving significant improvements over the vanilla model and releasing the checkpoints publicly.
This paper systematically compares fine-tuned encoder classifiers (ModernBERT family) against decoder-based safety judges for LLM adversarial evaluation, finding that encoders can offer a cost- and latency-efficient alternative without significant performance loss.
Introduces LOCUS, a comprehensive corpus of U.S. local ordinance codes designed to enable machine-readable legal AI research, covering codes from 9,239 cities and counties with ModernBERT-based classifiers for analysis.
EZFurigana is a free, privacy-focused tool that uses Sudachi and ModernBERT to add context-aware furigana to Japanese text, supporting various input formats and customization options.
ACL-Verbatim introduces a family of lightweight extractive models for grounded RAG that return exact text spans from source, outperforming larger LLM-based extractors.
Introducing the Ettin Reranker family: six new state-of-the-art CrossEncoder rerankers at various sizes, built on ModernBERT encoders, with open-source data and training recipe.
HyDRA is a hybrid dynamic routing architecture for heterogeneous LLM pools that predicts fine-grained capability requirements per query and selects the cheapest capable model via shortfall matching, achieving up to 72.5% cost savings with quality maintained. It is deployed in GitHub Copilot's VS Code Chat auto-mode and decouples routing from model catalog, requiring no retraining when models change.
This paper demonstrates that switching from Masked Language Modeling to Causal Language Modeling during encoder adaptation improves downstream performance on biomedical texts. The authors release ModernBERT-bio and ModernCamemBERT-bio as state-of-the-art biomedical encoders.