representation-learning

This paper uses Sparse Autoencoders to analyze the geometry of LoRA-induced representations in language models, finding that LoRA updates occupy partially distinct feature structures not fully captured by pretrained interpretability dictionaries.

TaxDistill proposes a knowledge distillation framework using a 500M parameter genomic foundation model (GenomeOcean) as a teacher to improve metagenomic taxonomic annotation by reducing label noise from similarity search tools, achieving significant F1 improvements on CAMI2 datasets.

DOMINO is a novel framework that learns minimal sufficient domain representations from reference examples to synthesize domain-specific data for LLMs, improving code benchmark performance without requiring explicit domain descriptions.

Introduces Representation Forcing (RF), a technique that enables unified multimodal models to perform both perception and generation end-to-end without external VAE latent spaces, matching state-of-the-art VAE-based models in image generation while improving understanding.

DynaFLIP is a dynamics-aware multimodal pre-training framework that integrates motion understanding into visual perception for robot manipulation. It uses image-language-3D flow triplets and geometric regularization to improve representation learning, achieving significant gains in out-of-distribution scenarios.

This paper investigates the role of inductive bias in time-series pretraining for clinical data, proposing PathoFM, an encoder-centric transformer pretrained on multivariate gait windows. The study compares different pretraining objectives and finds that dynamics-centric mixtures yield the most balanced transfer across classification and regression tasks.

This paper presents a systematic frozen-feature probing study comparing vision-language models (VLMs) and video generation models (VGMs) on spatial intelligence tasks. It finds that VLMs excel at semantic tagging and instance grouping, while VGMs provide better dense geometry and camera motion signals, and a naive fusion of both yields strong performance across all axes.

This paper systematically compares reconstruction-based and semantic latent spaces for action-conditioned latent diffusion world models in robotics. It finds that semantic encoders like V-JEPA 2.1 generally outperform reconstruction encoders on policy-relevant metrics, advocating for semantic latent spaces as a stronger foundation for robotics world models.

This paper applies successor representations from reinforcement learning to natural language, training a neural network to predict the expected distribution of future words. It shows that linguistic categories like parts of speech and lexical subclasses emerge spontaneously without explicit supervision.

The paper introduces a novel task of fact generation for hyper-relational knowledge graphs (HKGs) and proposes KREPE, a generative representation learning method using masked discrete diffusion that unifies link prediction and fact generation, achieving state-of-the-art performance.

This paper introduces a bifurcation theory of representation dynamics to detect when neural networks acquire structured representations during training, using a Hessian analysis of a GMM probe. The resulting ratio β/β_c serves as a label-free phase coordinate that predicts the onset of usable structure and can forecast feature interpretability in sparse autoencoders early in training.

This paper investigates whether positional bias in dense retrievers originates from architecture or training data, finding that training data distribution strongly influences bias and that balanced training can reduce sensitivity by up to 87% while maintaining retrieval performance.

SpatialBench is a comprehensive benchmark for evaluating spatial foundation models across diverse domains and tasks, revealing limitations in current models and introducing DA-Next-5M and DA-Next to advance spatial representation learning.

PilotWiMAE introduces a self-supervised framework that directly ingests noisy pilot observations for wireless channel representation learning, removing the unrealistic full-CSI assumption and enabling robust cross-frequency beam selection and channel estimation that beats supervised baselines.

This paper introduces LOES (Layer-wise Optimal Embedding Selection) and GeoReg (Geometric Regularization Loss), methods that select and fuse task-relevant intermediate layers from deep models to improve transfer learning performance, demonstrating consistent gains across architectures and modalities.

RADAR is a geometrically grounded metric that estimates cross-domain transferability in foundation models by analyzing layer-wise angular and distance changes in representations, using KL divergence between within-domain and cross-domain trajectory distributions.

This paper proposes Human-Centered Learning Mechanics (HCLM), a dynamical and information-theoretic framework for studying open and controlled learning systems. It formalizes entropy regularization through effective information force, derives convergence and generalization results, and provides a conditional interpretation of scaling-law behavior.

Next Implicit Token Prediction (NITP) enhances language model pre-training by adding dense continuous supervision in representation space, improving generalization and performance across model sizes with minimal computational overhead.

Introduces the Temporal Contrastive Transformer (TCT), a self-supervised framework for learning temporal embeddings from financial transactions for fraud detection. Achieves AUC 0.8644 with embeddings alone but does not improve over strong engineered features (AUC 0.9205 vs 0.9245), indicating learned representations overlap with existing features.

This paper proposes an operational criterion for interpretable text representations based on inter-annotator agreement and label disentanglement, and introduces LLM-assisted Feature Discovery (LFD), a method that uses cross-LLM agreement screening and residual predictive gain to select clear, label-disentangled features. Experiments show LFD matches predictive performance while producing more interpretable features, validated by human audits.