bert

UltraBERT revisits bidirectional encoder pre-training design choices, showing performance saturation beyond a few trillion tokens, masking-ratio trade-offs between classification and retrieval, and proposes a dual-LM-head distillation approach achieving state-of-the-art on GLUE (91.0) and BEIR (47.7).

这篇论文在极小规模（8.7M 参数、4 层）的中文 BERT 上严格控制变量，对比了 MLM、WWM 和 MacBERT 三种预训练策略，发现小规模下的策略排名（MLM > WWM ≫ MacBERT）与大规模结论相反，并指出 MacBERT 在同义词词典覆盖不足时会出现困惑度严重退化、训练损失不可靠的问题。

The paper investigates neurons in frozen BERT that drive AI-text detection using sparse probing and activation patching on the RAID benchmark, identifying a small set of causally relevant neurons that generalize across generator families.

A study or model with only 4 million parameters, fine-tuned on 10,000 examples, achieves performance surpassing opus and kimi in certain benchmarks.

This paper introduces a multi-agent system called CB-MAS for document sensitivity classification, which addresses the limitation of fixed input length in transformer models like BERT by using iterative consultation and channel boosting.

This study examines blame attribution in the Danish Parliament from 1997 to 2026 using the BlameBERT classifier, revealing ideological asymmetries and a banana-shaped trajectory in political discourse.

A nostalgic reflection on BERT's significant impact in 2020, with Hugging Face documenting it as 'BERTology', and a debate on what constitutes a large language model.

A tweet discusses the resurgence of encoder models in AI, referencing BERT and GPT, and questions if this marks a Renaissance of the Encoder.

Laya is an open-source non-autoregressive decision model that provides typed answers with calibrated probabilities, designed for tasks like email triage and conversational AI, showing significant performance improvements over existing models.

The paper presents fine-tuned pre-trained models for automatic relation extraction from biomedical text in the variant-phenotype domain, showing that DeBERTa and Gemini Pro 1.0 achieve state-of-the-art performance.

The author questions whether Jev is a generalized version of BERT, noting its capabilities as an intelligent classifier with custom prompt and criteria instruction reading features.

This paper studies confidence estimation and selective prediction for financial named entity recognition under domain shift, evaluating BERT and LoRA-tuned Qwen models to enhance reliability across different input distributions like SEC filings and social media.

A comparative study of BART, BERT, and RoBERTa for text summarization, examining their architectures and suitability for extractive and abstractive summarization tasks.

This paper evaluates the reliability of BERT-based QA models (RoBERTa, ALBERT, DistilBERT) under Monte Carlo Dropout and input paraphrasing, finding RoBERTa more consistent and validating MCD as a reliability metric.

Presents STCAD, a scalable framework using BERT-based encoding and CURE clustering to perform trajectory clustering and anomaly detection on terabyte-scale AIS maritime data, demonstrating stable clusters and clear separation of anomalous vessel behavior.

This paper presents a machine learning pipeline for detecting self-introductions in legislative committee testimony, using features like bag-of-words and BERT probabilities. XGBoost achieves the best F1 score of 0.9747, improving further with BERT-augmented features.

This paper proposes CNM, a lightweight augmentation that injects discrete compositional structure of Chinese characters into BERT via Ideographic Description Sequences, improving performance on rare and out-of-vocabulary characters while preserving general NLU accuracy.

This paper proposes Joint Affine Spectral Shaping (JRI), which extends weight-only spectral optimizers like Muon to jointly update weight and bias in affine layers, showing small but consistent accuracy improvements on a BERT-mini IMDb classification task.

A tweet argues that masked language modeling was unnecessary and that autoregressive models would have sufficed, with a nod to BERT.

This paper proposes an automated multilabel classification system for Mpox research articles using BERT, achieving 97% accuracy, and employs SHAP for explainability. The system aims to help researchers and healthcare workers quickly find relevant information.