The Galaxy's Guide to the Tokenizer: A Benchmark for Scientific Foundation Models

Hugging Face Daily Papers Papers
tokenization foundation-models astronomical-images benchmark transformer representation-learning

Summary

This paper compares four tokenization methods (Affine, AIM, JetFormer, VQ-VAE) for astronomical images within a unified transformer framework, using 640,000 galaxy images to evaluate reconstruction quality, physical property prediction, and morphological preservation. It finds that no single method excels across all tasks, highlighting trade-offs in representation learning.

Tokenization is central to adapting scientific data for transformer-based foundation models, yet its impact on learned representations remains poorly understood. We compare four tokenization strategies, Affine, AIM, JetFormer, and VQ-VAE, within a unified transformer framework for astronomical imaging. Using 640,000 galaxy images from the DESI Legacy Survey and a shared AstroPT backbone, we evaluate each method on reconstruction fidelity and prediction of physical properties. Our results reveal trade-offs across approaches. The flow-based JetFormer achieves higher reconstruction quality, while VQ-VAE yields strong probe performance for galaxy physical properties. Affine and AIM better preserve localized morphological information. We find that reconstruction and representation quality are decoupled, and no single method consistently performs best across the tasks considered here. By grounding our evaluation in independently measured physical quantities, we hope this study serves to highlight the potential of scientific data as a basis for constructing interpretable benchmarks for foundation models.
Original Article
View Cached Full Text

Cached at: 06/29/26, 06:04 PM

Paper page - The Galaxy’s Guide to the Tokenizer: A Benchmark for Scientific Foundation Models

Source: https://huggingface.co/papers/2606.25610

Abstract

Four tokenization methods for astronomical images show distinct strengths in reconstruction quality, physical property prediction, and morphological preservation, with no single approach excelling across all tasks.

Tokenizationis central to adapting scientific data fortransformer-based foundation models, yet its impact on learned representations remains poorly understood. We compare fourtokenizationstrategies, Affine, AIM, JetFormer, andVQ-VAE, within a unified transformer framework forastronomical imaging. Using 640,000 galaxy images from theDESI Legacy Surveyand a sharedAstroPT backbone, we evaluate each method onreconstruction fidelityand prediction ofphysical properties. Our results reveal trade-offs across approaches. Theflow-basedJetFormer achieves higher reconstruction quality, whileVQ-VAEyields strong probe performance for galaxyphysical properties. Affine and AIM better preserve localizedmorphological information. We find that reconstruction and representation quality are decoupled, and no single method consistently performs best across the tasks considered here. By grounding our evaluation in independently measured physical quantities, we hope this study serves to highlight the potential of scientific data as a basis for constructing interpretable benchmarks for foundation models.

View arXiv pageView PDFGitHub48Add to collection

Get this paper in your agent:

hf papers read 2606\.25610

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.25610 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.25610 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.25610 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

From 2D Grids to 1D Tokens: Reforming Shared Representations for Multimodal Image Fusion

Hugging Face Daily Papers

This paper introduces a multimodal image fusion method that uses a 1D token interface from a pretrained image tokenizer to enhance global appearance coherence while preserving local details through selective token editing (STE). Experiments on four benchmarks show state-of-the-art performance in both global coherence and local fidelity.

Identifiable Token Correspondence for World Models

arXiv cs.LG

This paper introduces Identifiable Token Correspondence, a method that models token correspondence across time frames to improve temporal consistency in transformer-based world models for visual reinforcement learning, achieving state-of-the-art results on multiple benchmarks.