Tag
The paper proposes LatentMoE, which moves expert computation into a lower-dimensional latent space to treat expert input dimension and intermediate width as explicit design choices, showing that near-square experts and intrinsic-dimension-guided layerwise capacity allocation improve validation loss and downstream performance over layer-homogeneous MoE baselines.