transformer

MACRO is a framework that learns task-specific execution routes over frozen LLM layers using Markov chain-based routing, improving reasoning accuracy without modifying model weights. It outperforms prior routing approaches while reducing search time significantly.

This paper introduces Spectral Aliasing Pretext (SAP), a self-supervised learning method for fault diagnosis in rotating machinery. By deliberately undersampling vibration signals and training a Transformer to reconstruct the original spectrum, SAP learns discriminative frequency-domain representations that achieve strong classification performance with limited labeled data.

This paper proposes a spatiotemporal graph Transformer framework for traffic forecasting in edge computing, combining graph neural networks for spatial correlations and Transformer self-attention for long-range temporal dependencies. Experiments on real-world cellular data show it outperforms recurrent graph-based baselines like GCN-LSTM and GCN-GRU.

Discusses the bottlenecks of using Transformer for large-scale inference and reviews the evolution of inference optimization technologies for large models from 2023 to 2026, including KV cache quantization, speculative decoding, architectural innovation, and hardware-software co-design.

Qapla' is a project that demonstrates training a small transformer language model from scratch on an $8 ESP32-S3 microcontroller, showing that on-device training is feasible without a GPU or datacenter.

This paper studies a self-supervised task for generating single-cell gene expression vectors using an autoregressive transformer with a quantized VAE tokenizer. It reports scaling laws and a compute-optimal frontier for single-cell foundation models, with potential fine-tuning for perturbation prediction.

This paper introduces the Customer-Based Multi-task Transformer (CBMT), a model that forecasts revenue by jointly predicting customer acquisition, repeat orders, and average order value. It claims improved forecast accuracy over existing benchmarks and analyzes when coordinated forecasting helps.

Introduces Maglev, a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training. It uses a prefiller and decoder with a memory consistency loss, improving validation loss and downstream benchmarks over baselines.

This paper presents a multimodal auto-regressive transformer surrogate that models variable well operations and geological uncertainty for geological carbon storage, achieving accurate predictions and enabling uncertainty quantification via MCMC data assimilation.

SeDeM is a selective decompression framework that stores long-context hidden states in a compact memory bank and decompresses only query-relevant blocks for decoder conditioning, improving QA accuracy and efficiency over compression baselines.

This paper identifies a numerical underflow failure mode in ALiBi positional encodings that can zero out attention weights, analyzes its impact, and evaluates mitigation strategies for training models with ALiBi.

This tutorial covers how to use a transformer model from training to inference, focusing on autoregressive generation, prefill vs. decode phases, and key-value caching for efficient inference.

This paper investigates evaluation protocols for federated pre-trained models, showing that downstream fine-tuning does not reliably preserve pre-training quality rankings, while direct next-token prediction strongly aligns with pre-training perplexity.

Documentation reference for MSLK 1.3.0, a library of fused GPU kernels for transformer workloads including attention, quantization, GEMM, and MoE routing, supporting CUDA and ROCm with PyTorch integration.

The author trained an encoder-only transformer to predict future blood glucose levels from past glucose, carbs, insulin, and future meal/insulin inputs, releasing the MIT-licensed source code with pretrained weights.

The paper proposes Source-Centered State Evolution (SCSE), a method for looped Transformers that reconciles input conditioning with reference-preserving shared recurrence, improving recurrent quality frontiers across multiple benchmarks.

This paper proposes a novel framework for causal discovery in multivariate time series using inverted self-attention mechanisms, introducing a causal self-attention module (CSAM) and a global causal algorithm to identify latent causal links and reduce spurious correlations.

This paper proposes the Conditional Informer, a Transformer-based encoder-decoder for ship trajectory prediction that conditions vessel dynamics on environmental contexts via a Conditional Attention mechanism, achieving 15.4% accuracy improvement and robust fallback via Modality Masking.

ClockRoPE introduces random Fourier rotations to model temporal periodicity in sequential recommendation, theoretically grounded and validated via online A/B tests at a major video-sharing platform.

This survey presents a systematic taxonomy of memory mechanisms in large language models, classifying along axes of representation, update dynamics, and persistence, and formalizing the underlying mechanistic components.