@jxmnop: we didn't ever need to invent Masked Language Modeling, I don't think. it was a bit silly by construction. in most alte…
Summary
A tweet argues that masked language modeling was unnecessary and that autoregressive models would have sufficed, with a nod to BERT.
Similar Articles
Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL [R]
This paper proposes using Masked Diffusion Language Models (MDLMs) as text-based world models for agentic reinforcement learning, showing that their any-order denoising objective avoids prefix mode collapse and leads to stronger performance than autoregressive baselines.
@_jasonwei: When language models first started using tools well, I was sympathetic to the narrative that instead of scaling up lang…
The author argues that while tool use allows smaller language models to perform tasks effectively, larger models remain crucial for speed, reliability, and internalized knowledge, emphasizing the ongoing need for scaling in AI.
Ad Headline Generation using Self-Critical Masked Language Model
Proposes a method using reinforcement learning on masked language models to generate advertising headlines for e-commerce, outperforming baselines and human submissions.
Masked Language Flow Models
This paper introduces Masked Language Flow Models (MLFMs), which incorporate masking into flow-based language models to enable continuous flow for conditional generation and allow pretrained Masked Diffusion Models to be converted. The authors propose a novel sampler that alternates continuous denoising with discrete unmasking, demonstrating for the first time that flow-based language models can scale to downstream reasoning and instruction-following tasks.
@maximelabonne: BERT was so incredibly big in ~2020 that @huggingface dedicated a page to "BERTology" in their docs. Lots of nostalgia …
A nostalgic reflection on BERT's significant impact in 2020, with Hugging Face documenting it as 'BERTology', and a debate on what constitutes a large language model.