transformer

This paper introduces ProForma-20Q, a benchmark for forecasting 78 financial statement line items 1–20 quarters ahead, and Forma, a transformer-based model that beats generalist approaches, especially at long horizons, with near-accounting-identity coherence.

This paper introduces LabelFusion-TS, which fuses a fine-tuned RoBERTa encoder, a prompted LLM, and time-series transformers over market data to classify Federal Reserve communication as hawkish, dovish, or neutral. The fused system achieves 70.2% weighted F1, outperforming a zero-shot LLM and showing early evidence that market time series help financial text classification.

This paper proposes ODE-based Transformer decoders using Runge-Kutta integration methods to improve iterative refinement for sign language translation, achieving better BLEU scores than the IPSLT baseline without increasing model size.

A demo of chessformer_lens shows that ablating a single attention head in a chess transformer causes it to stop recognizing Morphy's queen sacrifice, demonstrating the concentration of specific capabilities in individual heads.

This paper evaluates the reliability of BERT-based QA models (RoBERTa, ALBERT, DistilBERT) under Monte Carlo Dropout and input paraphrasing, finding RoBERTa more consistent and validating MCD as a reliability metric.

This paper proposes UniF-MoE, a unified framework for token-adaptive Mixture-of-Experts computation that first shares reusable computation across experts and then routes the remaining residual demand, improving performance while reducing activated computation, latency, and memory on DomainBed and GLUE benchmarks.

This paper proposes CRHT, a Continuous Regression Hybrid Transformer for vessel trajectory prediction using AIS data, featuring an online K-means cluster sampling strategy and a CNN-Transformer hybrid architecture to address geographic bias and improve short-term forecasting accuracy.

This paper presents a decision-centric causal optimization framework for large-scale targeting and recommendation, combining a causal Transformer, Bayesian bandit layer, and dual-based linear programming. It reports a statistically significant +7.20% lift in LinkedIn Feed marketing traffic via online A/B testing.

DoGMA is a central-dogma-guided foundation model for pan-cancer multi-omics analysis, using a Transformer-MoE architecture with directed attention to align DNA-RNA-protein flows and pretraining via masked hierarchical omics reconstruction. It shows strong performance across cancer representation learning, survival prediction, and metastasis prediction tasks.

This paper applies Marchenko-Pastur random matrix theory to pre-trained attention weights, separating each projection matrix into a random-like bulk and spectral outliers. Causal experiments show zeroing these outliers in Mistral-7B drives performance near random chance, revealing that spectral outliers encode dominant learned structure across 11 transformers.

This paper proposes using signed, fusion-aware Integrated Gradients for attributing predictions in feature-tokenized transformers like BiomeGPT, overcoming limitations of CLS attention weights and revealing disease-supporting versus protective microbial signals.

This paper introduces a misinformation detection framework using activation engineering, projecting last-token activations onto a learned 'falsehood direction' in LLM residual streams. It evaluates across Gemma, Llama, and Qwen models on fact-checking benchmarks, showing truthfulness is linearly separable in latent space.

This paper introduces bioMoR, a biology-guided Mixture-of-Recursions framework for genomic learning that integrates structured biological knowledge into recursive Transformer architectures, improving efficiency and accuracy across omics benchmarks.

The author shares insights from training a small model with DeepSeek's latent attention, observing layer-dependent latent usage and a test-time trick that reduces KV cache 4x without loss change.

MACRO is a framework that learns task-specific execution routes over frozen LLM layers using Markov chain-based routing, improving reasoning accuracy without modifying model weights. It outperforms prior routing approaches while reducing search time significantly.

This paper introduces Spectral Aliasing Pretext (SAP), a self-supervised learning method for fault diagnosis in rotating machinery. By deliberately undersampling vibration signals and training a Transformer to reconstruct the original spectrum, SAP learns discriminative frequency-domain representations that achieve strong classification performance with limited labeled data.

This paper proposes a spatiotemporal graph Transformer framework for traffic forecasting in edge computing, combining graph neural networks for spatial correlations and Transformer self-attention for long-range temporal dependencies. Experiments on real-world cellular data show it outperforms recurrent graph-based baselines like GCN-LSTM and GCN-GRU.

Discusses the bottlenecks of using Transformer for large-scale inference and reviews the evolution of inference optimization technologies for large models from 2023 to 2026, including KV cache quantization, speculative decoding, architectural innovation, and hardware-software co-design.

Qapla' is a project that demonstrates training a small transformer language model from scratch on an $8 ESP32-S3 microcontroller, showing that on-device training is feasible without a GPU or datacenter.

This paper studies a self-supervised task for generating single-cell gene expression vectors using an autoregressive transformer with a quantized VAE tokenizer. It reports scaling laws and a compute-optimal frontier for single-cell foundation models, with potential fine-tuning for perturbation prediction.