Tag
The paper introduces Latin Modern BERT (LAMB), a ModernBERT-based encoder pretrained on over 74B Latin tokens with morphologically informed masking, alongside two new Latin corpora (Acervus and Corpus Trinum), and shows it outperforms 15 existing encoders on 16 of 17 benchmark subtasks.
The paper introduces Verdict-MQ, a 395M open encoder that encodes an input's state once and answers many typed questions about it via a structured attention mask, outperforming the 421M Laya cross-encoder on calibration and accuracy for typed decisions while trading off performance on RACE and BoolQ.
UltraBERT revisits bidirectional encoder pre-training design choices, showing performance saturation beyond a few trillion tokens, masking-ratio trade-offs between classification and retrieval, and proposes a dual-LM-head distillation approach achieving state-of-the-art on GLUE (91.0) and BEIR (47.7).
DecisionTune 1.0 is a 395M-parameter encoder (ModernBERT-large plus a scoring head) that scores options offline for small agent decisions, running in ~10 ms per decision on MLX with Apache-2.0 weights and backends for PyTorch, MLX, and ONNX.
A tweet discusses the resurgence of encoder models in AI, referencing BERT and GPT, and questions if this marks a Renaissance of the Encoder.
NeoMME presents a family of efficient multimodal encoders pretrained with masked discrete diffusion, achieving strong performance in visual document retrieval and high embedding compression while outperforming smaller models on benchmarks.
Liquid AI fine-tuned their LFM2.5-Encoder models (230M and 350M) to perform multi-label classification in a single forward pass, eliminating the need for decoding loops or parsing. This demonstrates efficient label scoring for NLP tasks.
Liquid AI releases LFM2.5-Encoder-230M and LFM2.5-Encoder-350M, bidirectional encoders optimized for non-generative tasks like classification and retrieval, offering fast CPU inference at long context.
Liquid AI released two new encoder models, LFM2.5-Encoder-230M and LFM2.5-Encoder-350M, that are fast, easy to train, and strongly multilingual, with speed benchmarks showing over 3.7x improvement on CPU compared to ModernBERT-base.
Liquid AI releases LFM2.5-Encoders (230M and 350M), efficient encoder models optimized for long-context inference on CPU, matching or beating larger encoders on benchmarks with 3.7x speedup over ModernBERT-base.
Liquid AI releases LFM2.5-Encoder-350M, a multilingual bidirectional encoder built on the LFM2 architecture, offering strong quality for its size, 8k context, and efficient on-device performance across 15 languages.
CrossBERT decouples representation learning from token reconstruction, enabling higher masking ratios and better sample efficiency, outperforming BERT on MTEB and GLUE benchmarks.
This paper introduces LeanGuard, a lightweight bidirectional encoder-based safety guardrail that matches the accuracy of larger reasoning-based guardrails while being approximately 100x faster, challenging the assumption that chain-of-thought reasoning is necessary for effective moderation.
The article describes libdeflate's new level 13, a deliberately slow DEFLATE compression level that achieves marginally better compression (0.134% on Silesia) at the cost of being 56x slower than level 12, designed for scenarios where data is compressed once and decompressed many times.
This paper introduces m3BERT, a multilingual bidirectional encoder with a novel pretraining strategy that jointly optimizes representations across transformer layers and multiple embedding dimensions, enabling a single model to be adapted to varied resource constraints. It significantly outperforms state-of-the-art models on the Bing-Click industrial retrieval dataset.