OpenESM: Energy-Steered Models Are Scalable Foundation Learners
Summary
OpenESM 提出能量引导模型（ESM），将能量引导的潜状态优化与 token 生成统一到单一架构中，在大规模语料上进行能量引导预训练，表现出良好的扩展性并在多数下游任务上超越 GPT，同时完全开源模型权重与训练基础设施。
View Cached Full Text
Cached at: 10/28/26, 01:52 AM
Paper page - OpenESM: Energy-Steered Models Are Scalable Foundation Learners
Source: https://huggingface.co/papers/2610.33005
Abstract
Energy-steered models integrate energy-guided latent optimization with token generation to improve foundational language modeling and scale effectively across diverse tasks.
Energy-based modelshave shown promising properties for domain-specific tasks but remain challenging in modeling general-purpose scenarios. To bridge this gap, we designenergy-steered models(ESMs) for foundational language modeling. ESMs integrate energy steering and token generation within a single architecture to leverage the energy-guided search to refine token generation. Specifically, ESMs first optimize alatent stateover theenergy landscape, then leverages the refinedlatent statefor token generation. By performing suchenergy-guided pre-trainingon large-scale corpora, ESMs learn generally effectiveenergy landscapewhere favorable tokens correspond to lower energy values. To explore thescaling behaviorof ESMs, we pre-train ESMs on three large-scale corpora, then evaluate across 26 downstream tasks spanning a broad range of domains. Our results demonstrate that ESMs have promisingscaling behaviors with increasing model size and training data, and outperform GPTs on most downstream tasks. These findings highlight ESMs as a promising architecture for large-scale pre-training and general purpose problem solving. We fully release the model weights and training infrastructure to enable the community to completely reproduce and extend our results. GitHub: https://github.com/datamllab/openesm HuggingFace: https://huggingface.co/collections/guan-wang/openesm
View arXiv pageView PDFProject pageGitHub179Add to collection
Get this paper in your agent:
hf papers read 2610\.33005
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2610.33005 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2610.33005 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2610.33005 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
🚀 OpenEBFM: Can Energy-Based Models Scale as Foundation Models? Code and Weights Released
OpenEBFM introduces Energy-Based Foundation Models that separate generation into energy steering and token generation, showing better data, parameter, and compute scaling than matched GPT baselines at up to 1B parameters, with full code, checkpoints, and an EBFM chatbot released as open source.
Implicit generation and generalization methods for energy-based models
OpenAI presents implicit generation and generalization methods for energy-based models (EBMs) that use Langevin dynamics for iterative refinement to generate samples without explicit generator networks. The approach offers advantages including adaptive computation time, flexibility in learning disconnected data modes, and built-in compositionality through product of experts.
ShellElastic: Scalable Elastic Pretraining with Simultaneous Maximal Updates and Hyperparameter Transfer
提出ShellElastic，一种将隐藏维度组织为稠密核心与递增外壳的弹性Transformer架构，使单次全宽前向即可同时包含所有子网络的精确隐藏状态；并引入μE扩展最大化更新参数化（μP），保证子网络在超网络加宽或增加外壳时仍被最大化更新且超参数可迁移，性能优于MatFormer等弹性基线并逼近分别训练的模型。
Scaling Transformer Energy-based Models For High Quality Image Synthesis
该论文提出 Transformer Energy-based Model (TEM)，通过在潜空间中构建可扩展的 Transformer 能量函数并多层能量聚合，实现了无需 classifier-free guidance 的高质量图像生成（ImageNet 256×256 FID-50K 达 3.82），验证了 EBM 的组合生成、零样本修复与分布外检测能力。
Introducing gpt-oss
OpenAI releases gpt-oss-120b and gpt-oss-20b, two state-of-the-art open-weight language models under Apache 2.0 license that achieve near-parity with proprietary models while being optimizable for consumer hardware and edge devices. Both models demonstrate strong reasoning and tool-use capabilities with comprehensive safety evaluations.