Tag
HiUni explores extending a pretrained vision-language model into a unified multimodal model via extensive parameter sharing, reusing a single dense backbone for both understanding and generation with a PixelDiffusionHead and HiGate pathway to address optimization and spatial-detail challenges. Experiments show it matches the pretrained VLM's understanding score while achieving competitive image generation and editing performance.
该论文提出因果潜在工作区 Transformer(causal latent workspace Transformer),通过在共享深度的循环块之间加入可读写的前缀专属潜在槽位,提升了循环 Transformer 的语言建模困惑度,并在三个字节级语料上的多数设置中优于五个基线对照。
论文提出 LatentLoop,通过非对称计算分配(压缩第一次循环访问的序列分辨率、保留第二次全分辨率并将节省的算力用于增大隐藏维度)来扩展循环 MoE 模型,在 1.3B–7B 参数范围内一致降低验证 NLL,相比原始 MoE 和循环基线分别节省约 19.0% 和 7.9% 的计算量。
The paper proposes CS-MoE, a novel Transformer architecture that shares neural experts across layers to improve parameter utilization, achieving lower perplexity with only 55% of parameters activated.
This paper introduces Looped GPT-BERT, which uses depth-wise parameter sharing to train a small language model with fewer parameters, achieving comparable performance to baselines in the BabyLM 2026 Strict-small setting.