V2TATC: A Joint Voice-Trajectory Embedding Framework and Dataset for Air Traffic Controller Situational Awareness
Summary
This paper presents V2TATC, a joint voice–trajectory embedding framework for air traffic controller situational awareness, and introduces a novel dataset for cross-modal retrieval experiments in congested airspaces.
arXiv:2608.28981v1 Announce Type: new
Abstract: As air traffic volumes in the National Airspace System continue to expand, in particular in the low altitude airspaces, the need for scalable decision support tools used by air traffic controllers will also require more development. This article introduces Voice-to-Trajectory for Air Traffic Control, a joint voice communication-flight trajectory data embedding framework, that can be a component of situational awareness in congested airspaces, and assist the development of tools for ATC as they reason in real-time over Automatic Dependent Surveillance-Broadcast trajectories, or the intent expressed by pilots in natural language. We show that these data modalities are not independent and represent a common physical referent: an aircraft flying through the airspace. V2TATC maps a voice instruction and the trajectory of the addressed aircraft to nearby points in a single latent space that can be queried in both directions. It combines a self-supervised trajectory encoder, a frozen large-scale speech encoder, a contrastive joint embedding, and a bijective lifting via normalizing flows. We demonstrate V2TATC's effectiveness on the San Francisco Bay Area, for its concentration of major airports, and its mix of commercial and general aviation low altitude traffic. Lastly, we release a novel paired voice-trajectory dataset, and report experiments on cross-modal retrieval, ablations, and latent-space analysis.
View Cached Full Text
Cached at: 09/01/26, 01:03 PM
# V2TATC: A Joint Voice–Trajectory Embedding Framework and Dataset for Air Traffic Controller Situational Awareness
Source: [https://arxiv.org/html/2608.28981](https://arxiv.org/html/2608.28981)
Alexandre M\. BayenNote:Liao\-Cho Endowed Chair Professor, Department of Electrical Engineering and Computer Sciences, University of California, Berkeley\.Affiliation:University of California, Berkeley, Berkeley, CA, 94720, USA
###### Abstract
As air traffic volumes in the National Airspace System continue to expand, in particular in the low altitude airspaces, the need for scalable decision support tools used by air traffic controllers will also require more development\. This article introduces Voice\-to\-Trajectory for Air Traffic Control, a joint voice communication–flight trajectory data embedding framework, that can be a component of situational awareness in congested airspaces, and assist the development of tools for ATC as they reason in real\-time over Automatic Dependent Surveillance–Broadcast trajectories, or the intent expressed by pilots in natural language\. We show that these data modalities are not independent and represent a common physical referent: an aircraft flying through the airspace\. V2TATC maps a voice instruction and the trajectory of the addressed aircraft to nearby points in a single latent space that can be queried in both directions\. It combines a self\-supervised trajectory encoder, a frozen large\-scale speech encoder, a contrastive joint embedding, and a bijective lifting via normalizing flows\. We demonstrate V2TATC’s effectiveness on the San Francisco Bay Area, for its concentration of major airports, and its mix of commercial and general aviation low altitude traffic\. Lastly, we release a novel paired voice–trajectory dataset, and report experiments on cross\-modal retrieval, ablations, and latent\-space analysis\.
## Nomenclature
\(All quantities are dimensionless unless units are given\.\)
𝐗\\mathbf\{X\}=trajectory window,T×FT\\times Farray of surveillance samplesxtx\_\{t\}=feature vector at timesteptt, withxt∈ℝFx\_\{t\}\\in\\mathbb\{R\}^\{F\}ww=raw voice waveform sampled at16kHz16\\text\{\\,\}\\mathrm\{kHz\}ϕ,λ,θ\\phi,\\lambda,\\theta=geodetic latitude and longitude, true track angle, radv,h˙v,\\dot\{h\}=ground speed and vertical rate, m s\-1dv,dt,djd\_\{v\},d\_\{t\},d\_\{j\}=voice, trajectory, and joint embedding dimensions \(dv=1280d\_\{v\}=1280,dt=1792d\_\{t\}=1792,dj=1024d\_\{j\}=1024\)ℰv,ℰt\\mathcal\{E\}\_\{v\},\\mathcal\{E\}\_\{t\}=voice and trajectory encodersPv,PtP\_\{v\},P\_\{t\}=voice and trajectory projectorsfv,ftf\_\{v\},f\_\{t\}=voice and trajectory normalizing flows inℝdj\\mathbb\{R\}^\{d\_\{j\}\}𝐯,𝐫\\mathbf\{v\},\\mathbf\{r\}=voice and trajectory embeddings,𝐯∈ℝdv\\mathbf\{v\}\\in\\mathbb\{R\}^\{d\_\{v\}\},𝐫∈ℝdt\\mathbf\{r\}\\in\\mathbb\{R\}^\{d\_\{t\}\}𝐯^,𝐫^\\hat\{\\mathbf\{v\}\},\\hat\{\\mathbf\{r\}\}=projected voice and trajectory embeddings,𝐯^,𝐫^∈ℝdj\\hat\{\\mathbf\{v\}\},\\hat\{\\mathbf\{r\}\}\\in\\mathbb\{R\}^\{d\_\{j\}\}𝐯~,𝐫~\\tilde\{\\mathbf\{v\}\},\\tilde\{\\mathbf\{r\}\}=ℓ2\\ell\_\{2\}\-normalized projected voice and trajectory embeddings𝐣\\mathbf\{j\}=joint\-space representation,𝐣∈ℝdj\\mathbf\{j\}\\in\\mathbb\{R\}^\{d\_\{j\}\}ℳ\\mathcal\{M\}=set of masked timestep indicesℒ∙\\mathcal\{L\}\_\{\\bullet\}=loss term \(subscript denotes its role\)λ∙\\lambda\_\{\\bullet\}=weight term in a loss \(subscript denotes its role\)SijS\_\{ij\}=cosine similarity between voiceiiand trajectoryjjτ\\tau=contrastive temperatureKK=retrieval cutoff used in Recall@KK
## 1Introduction
Operational needs in the National Airspace SystemAs traffic congestion density in theNational Airspace System\(NAS\) continues to increase,decision support tools\(DST\) used by air traffic controllers must also become increasingly capable of supporting this growth in high\-stakes operations\[[5](https://arxiv.org/html/2608.28981#bib.bib1)\]\.Air Traffic Control\(ATC\) requires real\-time reasoning over multiple streams of multimodal information in sequenced safety\-critical scenarios\. Without always realizing it, a controller managing a congested terminal airspace continuously analyzes 3D trajectories from*Automatic Dependent Surveillance–Broadcast \(ADS\-B\)*, the intent expressed by pilots in natural language over radio, environmental conditions reported by*Meteorological Aerodrome Reports \(METAR\)*, digital infrastructure of theFederal Aviation Administration \(FAA\)\(navigation aids \(navaids\),VHF Omnidirectional Range \(VORs\), etc\.\), and many other auxiliary signals\[[6](https://arxiv.org/html/2608.28981#bib.bib2)\]\. This ATC data represents a common physical entity, an aircraft flying through the airspace, and it is precisely the implicit relationship between them that allows a human controller to maintain situational awareness and ensure safe separation\. Most existing DSTs, by contrast, process these streams in isolation\. For example, speech recognition pipelines transcribe radio audio into text and further modules recover speaker roles and turn boundaries from those transcripts\[[16](https://arxiv.org/html/2608.28981#bib.bib3),[38](https://arxiv.org/html/2608.28981#bib.bib18)\]\. Trajectory prediction systems operate on ADS\-B features alone, whether sourced from operational surveillance or from large\-scale networks such as OpenSky\[[34](https://arxiv.org/html/2608.28981#bib.bib34)\], and forecast future positions with recurrent or attention\-based models\[[37](https://arxiv.org/html/2608.28981#bib.bib19),[12](https://arxiv.org/html/2608.28981#bib.bib5)\]\. Weather is used through a separate display, leaving its integration to the controller\[[5](https://arxiv.org/html/2608.28981#bib.bib1)\]\. The semantic correspondence between a controller’s verbal instruction and the resulting maneuver, which is the very basis of safe and efficient operations, is left implicit, and processed by the human\.
This article summarizes results that demonstrate that this implicit relationship can be learned\. We presentVoice\-to\-Trajectory for Air Traffic Control\(V2TATC\), a framework in which voice transmissions and ADS\-B trajectories live in a single representation space, so that a voice transmission and the trajectory of the addressed aircraft are mapped to nearby points, and so that the mapping can be queried in both directions\. The idea is inspired by recent progress in multi\-modal joint embeddings, in particular the contrastive language–image alignment introduced byContrastive Language\-Image Pre\-training\(CLIP\)\[[29](https://arxiv.org/html/2608.28981#bib.bib8)\]and its successors in vision–language\[[18](https://arxiv.org/html/2608.28981#bib.bib9),[25](https://arxiv.org/html/2608.28981#bib.bib10)\], and by the maturity of large\-scale speech encoders such as Whisper\[[30](https://arxiv.org/html/2608.28981#bib.bib11)\]and Wav2Vec 2\.0\[[1](https://arxiv.org/html/2608.28981#bib.bib12)\]\. These models share a common component, namely a pre\-trained per\-modality encoder followed by a lightweight alignment head trained on paired data using a contrastive objective\. We adopt the same approach with two specificities that matter for ATC\. First, the alignment target is not an open\-ended natural\-language description\. It is a structured time series of several features broadcast at a roughly constant rate\. This calls for a specialized trajectory encoder that captures motion dynamics, which we obtain through aMasked Autoencoder\(MAE\) objective\[[14](https://arxiv.org/html/2608.28981#bib.bib16)\]\. Second, retrieval alone is not enough, because a DST may eventually need to generate a candidate trajectory from a voice instruction, or vice versa\. This calls for an invertible alignment, which we obtain by training two normalizing flows\[[9](https://arxiv.org/html/2608.28981#bib.bib6),[22](https://arxiv.org/html/2608.28981#bib.bib7)\]on top of the contrastive joint space\.
The joint embedding is intended as a building block for DSTs rather than as an end\-user application by itself\. A small set of operations on top of the joint space already maps to concrete controller needs\. Selecting an aircraft on the surveillance display and recovering the corresponding recent voice transmissions \(*trajectory→\\tovoice*retrieval\) supports rapid context recovery during handovers and high\-workload situations\. The symmetric operation, going from a clipped voice transmission to a candidate trajectory \(*voice→\\totrajectory*\), supports listening tools that help a supervisor verify which aircraft is being addressed\. Inconsistency between a voice instruction and the corresponding trajectory in the joint space provides a straightforward signal for anomaly detection, including missed acknowledgments and non\-compliant maneuvers\. Finally, the bijective lifting opens the door to generative decision support: short\-horizon trajectory forecasts conditioned on voice instructions, plausible\-instruction synthesis for simulator training, and conflict probes that explore counterfactual maneuvers\.
a\)b\)
Figure 1:Geographic scope of the data collection over the San Francisco Bay Area\. a\) Map boundaries corresponding to the latitude\-longitude bounding box of the bay, which is also the geographical window used to scrape ADS\-B information from the OpenSky Network \([opensky\-network\.org](https://opensky-network.org/)\)\. Red circles mark the four airports whose ATC tower frequencies are streamed through[LiveATC\.net](https://www.liveatc.net/)\. Blue markers indicate the weather reporting stations from which METAR observations are periodically collected via[aviationweather\.gov](https://aviationweather.gov/)\. b\) VFR \(Visual Flight Rules\) sectional chart of the same area, taken from[vfrmap\.com](https://vfrmap.com/), showing how densely concentrated the airspace is, with overlapping controlled airspace boundaries, airways and reporting points packed into a small region\.We instantiate V2TATC over the San Francisco Bay Area as a case study for the geographic area shown in Fig\.[1](https://arxiv.org/html/2608.28981#S1.F1)\. The Bay Area is a particularly relevant region of interest for four reasons\. \(i\) It hosts several major commercial airports, San Francisco International \(KSFO\), Oakland International \(KOAK\), and San Jose International \(KSJC\), as well as Moffett Field \(KNUQ\), all operating under the overlapping sectors of the Northern California TRACON\. \(ii\) Traffic density is among the highest in the NAS, with a steady stream of arrivals and departures across the day\. \(iii\) Commercial traffic and general aviation coexist on the same frequencies, so the data naturally captures both categories\. \(iv\) The same airspace is expected to host some of the first operational trials of urban air mobility aircraft \(e\.g\.,electric vertical take\-off and landing\(eVTOL\) aircraft\), which will further stress controller workload in the near future\[[20](https://arxiv.org/html/2608.28981#bib.bib13)\]\. This concentration of features makes it possible to collect a rich multimodal dataset while keeping the geographic boundaries small\. We describe how to collect synchronized voice and surveillance data over this airspace, how to clean and link them at scale, and how to train each component of the framework\. We then present results on cross\-modal retrieval, short\-horizon trajectory forecasting, voice\-conditioned trajectory generation, and the structure of the learned latent space\.
A side product of the present work is the paired voice–trajectory dataset itself\. To the best of our knowledge, no such dataset has been released publicly for the Bay Area, since existing ATC datasets are either voice\-only or trajectory\-only, and the few works that combine them do so on closed proprietary data or over smaller, uncongested regions\. Building the dataset required audio\-archive ingestion, real\-time ADS\-B polling, transcription, callsign extraction and temporal linking, all described in Section[4](https://arxiv.org/html/2608.28981#S4)\. We release a subset of our dataset111Available at[huggingface\.co/datasets/Lbrusset/SF\_bay\_voice2traj\_dataset](https://huggingface.co/datasets/Lbrusset/SF_bay_voice2traj_dataset)\.and the entire processing pipeline so that future work, whether on speech recognition, trajectory modeling, joint embeddings, or operational decision support tools, can build on it\. The contributions of the present article are threefold\. First, we formulate a learnable correspondence between voice and trajectory submanifolds and propose a framework that combines self\-supervised pre\-training, contrastive alignment, and bijective lifting through normalizing flows\. Second, we describe an end\-to\-end implementation of V2TATC using publicly available data sources and standard self\-supervised techniques, and we release both the assembled dataset and the code\. Third, we report quantitative results that show that the two modalities can be aligned with a contrastive objective in spite of the noisy nature of ATC audio, and that the alignment supports useful downstream operations such as short\-horizon forecasting and cross\-modal retrieval\.
Related Work
Several articles have explored the use of machine learning in ATC, focusing primarily on a single ATC modality\.Automatic Speech Recognition \(ASR\)for ATC has progressed steadily, from constrained\-grammar systems built on hidden Markov models\[[33](https://arxiv.org/html/2608.28981#bib.bib17)\]to transformer\-based models trained on aviation\-specific corpora such as ATCO2\[[39](https://arxiv.org/html/2608.28981#bib.bib4)\], on top of which downstream understanding tasks have been built\[[38](https://arxiv.org/html/2608.28981#bib.bib18)\]\. Trajectory modeling has likewise matured, from recurrent architectures\[[37](https://arxiv.org/html/2608.28981#bib.bib19)\]to attention\-based encoders operating on richer trajectory representations\[[12](https://arxiv.org/html/2608.28981#bib.bib5)\]\. Related works onlarge language models\(LLM\) for ATC have shown that short\-horizon trajectory forecasting is feasible with these transformer\-based architectures\[[7](https://arxiv.org/html/2608.28981#bib.bib14)\], and that agentic ATC can be performed from communication data in natural language alone\[[11](https://arxiv.org/html/2608.28981#bib.bib15)\]\. Readback error detection, which compares a transcribed pilot acknowledgment against the controller instruction\[[17](https://arxiv.org/html/2608.28981#bib.bib21)\], has a similar single\-task focus\. SIA\-FTP\[[13](https://arxiv.org/html/2608.28981#bib.bib20)\]uses spoken instructions as side information to improve flight\-trajectory prediction, in a setup that is end\-to\-end supervised on the prediction task\. V2TATC differs in that it learns a self\-supervised joint embedding and then evaluates several downstream tasks on top of it\. To our knowledge no prior work aligned raw ATC voice with surveillance trajectories at the embedding level\. This article is to our knowledge the first step in this direction, and is conceptually related to the joint\-embedding predictive architectures presented by LeCun\[[24](https://arxiv.org/html/2608.28981#bib.bib22)\]\.
## 2Problem Formulation
### 2\.1Situational Awareness in Congested Airspaces
Situational awareness, in the sense of Endsley\[[10](https://arxiv.org/html/2608.28981#bib.bib23)\], is the controller’s ability to perceive, comprehend, and project the state of the airspace\. In a busy terminal area, this state is high\-dimensional: tens of aircraft, each with a dynamic trace, an identity, and an active stream of verbal instructions\. Modern controller working positions display trajectory information in great detail through radar plots and ADS\-B integration\. The radio channel, by contrast, is still largely consumed as raw audio\. While ASR tools are beginning to be deployed, voice is rarely interpreted and aligned with the surveillance stream in standard operational settings\. The intuitive feature*"click on this aircraft and recover what is being said about it"*is not natively supported by existing DSTs, because the surveillance display and the radio frequency are not aligned in any explicit way\. Yet this operation is what controllers do mentally, dozens of times per hour\. The cognitive load it generates has been quantified in the human\-factors literature\[[15](https://arxiv.org/html/2608.28981#bib.bib24)\], and it is widely recognized as one of the structural limits of controller throughput\.
We formalize this operation as a mapping between two manifolds\. Let𝒳v\\mathcal\{X\}\_\{v\}be the space of voice transmissions and𝒳t\\mathcal\{X\}\_\{t\}be the space of trajectory windows\. Both spaces describe a portion of the same airspace, but with different observables\. The space𝒳v\\mathcal\{X\}\_\{v\}carries semantic information about the controller’s intent, while𝒳t\\mathcal\{X\}\_\{t\}carries the observed resulting trajectory\. To bring the two observables to a comparable semantic level, the trajectory\-side works on the differencesΔxt=xt\+1−xt\\Delta x\_\{t\}=x\_\{t\+1\}\-x\_\{t\}between two consecutive timesteps rather than on absolute positions\. With this choice, we expect that intent can determine a relative maneuver and, conversely, that a relative maneuver carries enough information to reveal the underlying intent\. The two observables are linked by the physical aircraft, but the link is not directly accessible from the raw signals\. Our goal is to learn a shared representation space𝒵\\mathcal\{Z\}together with two encoders such that a voice phrase and the trajectory of the addressed aircraft project to nearby points in𝒵\\mathcal\{Z\}, while unrelated pairs project to distant points\. With this property, a controller could find the trajectory associated with a given voice transmission by nearest\-neighbor lookup in𝒵\\mathcal\{Z\}, and vice versa\. In effect,𝒵\\mathcal\{Z\}serves as a learned cartography of the airspace\.
### 2\.2Two\-Tower Joint Embedding Approach
The proposed V2TATC approach follows the two\-tower design that has become standard in multi\-modal retrieval\[[29](https://arxiv.org/html/2608.28981#bib.bib8),[18](https://arxiv.org/html/2608.28981#bib.bib9)\]\. One tower processes trajectory data, the other processes voice data\. The two towers do not share parameters, and each is specialized for its own modality\. They communicate through a single shared space at their output\. Figure[2](https://arxiv.org/html/2608.28981#S2.F2)shows the conceptual layout\. The design is practical for several reasons\. Candidate embeddings can be pre\-computed offline, so online retrieval reduces to a single matrix\-vector multiplication\. Each tower also carries the right inductive bias for its own modality, with causal masking on the voice side and time\-aware attention on the trajectory side\. And because the towers are independent, either one can be swapped without retraining the other\.
Controller and pilotradio exchangeAircraft ADS\-Bsurveillance trackVoice data \(ℝB×L\\mathbb\{R\}^\{B\\times L\}\)Voice encoderℰv\\mathcal\{E\}\_\{v\}ProjectorPvP\_\{v\}Trajectory data \(ℝB×T×F\\mathbb\{R\}^\{B\\times T\\times F\}\)Trajectory encoderℰt\\mathcal\{E\}\_\{t\}ProjectorPtP\_\{t\}Joint space\(ℝB×dj\\mathbb\{R\}^\{B\\times d\_\{j\}\}\)∙\\bullet𝐯^\\hat\{\\mathbf\{v\}\}▲\\blacktriangle𝐫^\\hat\{\\mathbf\{r\}\}wwXX𝐯\\mathbf\{v\}𝐫\\mathbf\{r\}𝐯^\\hat\{\\mathbf\{v\}\}𝐫^\\hat\{\\mathbf\{r\}\}ℒNCE\\mathcal\{L\}\_\{\\text\{NCE\}\}Figure 2:Two\-tower joint embedding overview\. The left tower processes voice, the right tower processes trajectories, and both towers map their input to a common space in which a contrastive objective pulls matching pairs together and pushes mismatched pairs apart\.djd\_\{j\}is the dimension of the joint embedded space\.A contrastive objective is well suited here: it requires no ground\-truth labels and no predefined classes, only a binary answer to the question of whether two observations describe the same aircraft at the same time\. This binary signal is naturally available in our data, because every ADS\-B trajectory carries the aircraft identity,*International Civil Aviation Organization \(ICAO\)*24\-bit transponder code and callsign, and every transcribed voice phrase mentions a callsign\. Pairing is therefore deterministic at the data level\. The contrastive loss then turns this pairing into a metric in the joint space\. Contrastive alignment alone, however, is not invertible\. The projectorsPvP\_\{v\}andPtP\_\{t\}are forward\-only\. Given an embedding they produce a point in𝒵\\mathcal\{Z\}, but they cannot be run in reverse\. To support the use case*"given a point in𝒵\\mathcal\{Z\}, recover the corresponding voice or trajectory embedding"*, we add a second stage that mimics a bijection between each modality space and𝒵\\mathcal\{Z\}\. This bijection is implemented as two normalizing flows\[[9](https://arxiv.org/html/2608.28981#bib.bib6)\], one per tower\. Once trained, the flows transform contrastive retrieval into a bidirectional translation between modalities, so that a voice can be turned into a synthetic trajectory representation, and vice versa\.
### 2\.3Bijectivity Beyond Retrieval
Retrieval is the simplest application of a joint space, but it is not the most useful one operationally\. A controller\-facing tool may need to answer questions of the form “if this voice instruction is followed, what would the trajectory look like?” or “what would a controller most plausibly say about this maneuver?” Both questions require sampling in modality space, conditioned on a point in𝒵\\mathcal\{Z\}\. A contrastive projector cannot do this because its inverse is not defined\. We use the term*bijection*in a loose, geometric sense: not the strict set\-theoretic bijection between𝒳v\\mathcal\{X\}\_\{v\}and𝒳t\\mathcal\{X\}\_\{t\}\(which does not exist in practice, since a single trajectory can be described by many equivalent voice phrases and a single phrase can match several plausible trajectories\), but a learned correspondence between the two low\-dimensional submanifolds that voice and trajectory embeddings populate inside the joint space\. The term*bijectivity*also reflects our use of the RealNVP\[[9](https://arxiv.org/html/2608.28981#bib.bib6)\]network, whose built\-in inversion maps one probability density to another and back through the same set of weights, so that a single network can be queried as voice→\\totrajectory or trajectory→\\tovoice without retraining and without a separate inverse model\.
## 3Approach
This section gives the formal description of each block of V2TATC, and we keep the discussion conceptual\. The concrete data sources, models, and hyperparameters are deferred to Section[4](https://arxiv.org/html/2608.28981#S4)\.
### 3\.1Trajectory Tower
A trajectory window𝐗∈ℝT×F\\mathbf\{X\}\\in\\mathbb\{R\}^\{T\\times F\}is a fixed\-length sequence of ADS\-B samples observed at irregular times\. Each timestepxtx\_\{t\}contains positions, velocity, heading, and vertical rate\. The trajectory tower learns an encoderℰt:ℝT×F→ℝdt\\mathcal\{E\}\_\{t\}:\\mathbb\{R\}^\{T\\times F\}\\to\\mathbb\{R\}^\{d\_\{t\}\}that maps such a window to a single vector or a sequence of token vectors\. The encoder is trained in a self\-supervised manner using a masked reconstruction objective\. The MAE framework, introduced for vision by He et al\.\[[14](https://arxiv.org/html/2608.28981#bib.bib16)\]and rooted in the masked language modeling tradition\[[8](https://arxiv.org/html/2608.28981#bib.bib25)\], is well suited to trajectory data for two reasons\. First, the supervision signal is the data itself, so no labels are required\. This matters because trajectory datasets at scale come without semantic annotation\. Second, the reconstruction task can only be solved if the encoder produces a globally consistent representation of the visible context\. The resulting representation transfers well to downstream tasks\[[14](https://arxiv.org/html/2608.28981#bib.bib16)\]\. This is also why we deliberately keep the decoderggas small as possible: with little capacity of its own, the decoder cannot compensate for a poor encoder output, so it acts as a capacity bottleneck that pushes the burden of producing a globally consistent representation ontoℰt\\mathcal\{E\}\_\{t\}\. The decoder must nonetheless remain expressive enough to reconstruct the target signal, so that gradients flow back and effectively optimize the encoder parameters\.
Letℳ⊂\{1,…,T\}\\mathcal\{M\}\\subset\\\{1,\\dots,T\\\}be a random subset of timesteps, with\|ℳ\|=⌊mT⌋\|\\mathcal\{M\}\|=\\lfloor mT\\rfloorwherem∈\(0,1\)m\\in\(0,1\)is the masking ratio\. The masked timesteps are replaced by a learned mask token𝐞mask∈ℝd\\mathbf\{e\}\_\{\\text\{mask\}\}\\in\\mathbb\{R\}^\{d\}\. The encoder processes the full corrupted sequence and produces hidden statesH=\(h1,…,hT\)H=\(h\_\{1\},\\dots,h\_\{T\}\), and a lightweight decoderggreconstructs the target representationyty\_\{t\}on the masked positions\. The training loss is
ℒMAE=1\|ℳ\|⋅F∑t∈ℳ‖g\(ht\)−yt‖22\.\\mathcal\{L\}\_\{\\text\{MAE\}\}\\;=\\;\\frac\{1\}\{\|\\mathcal\{M\}\|\\cdot F\}\\sum\_\{t\\in\\mathcal\{M\}\}\\bigl\\\|g\(h\_\{t\}\)\-y\_\{t\}\\bigr\\\|\_\{2\}^\{2\}\.\(1\)The choice of targetyty\_\{t\}has a strong influence on what the encoder learns\. We use the normalized first\-order difference \(deltazz\-score\) of each feature:yt=\(zt−zt−1\)/σδy\_\{t\}=\(z\_\{t\}\-z\_\{t\-1\}\)/\\sigma\_\{\\delta\}wherezt=\(xt−μ\)/σz\_\{t\}=\(x\_\{t\}\-\\mu\)/\\sigma,μ,σ\\mu,\\sigmaare per\-feature mean and standard deviation, andσδ\\sigma\_\{\\delta\}is the standard deviation of consecutive differences\. This separation of scales prevents features at very different absolute ranges \(e\.g\., longitude in km vs\. vertical rate in m/s\) from dominating the loss\.
ADS\-B samples are not produced at a strictly regular rate\. Ground stations relay messages opportunistically, and missed reports lead to variableΔt\\Delta tbetween consecutive points\. To expose this irregular sampling to the encoder, we add a Time2Vec encoding\[[21](https://arxiv.org/html/2608.28981#bib.bib26)\]ofΔt\\Delta tto each token\. Time2Vec is a learned vector representation defined by
Time2Vec\(τ\)k=\{ω0τ\+ϕ0k=0,sin\(ωkτ\+ϕk\)1≤k≤dτ−1,\\text\{Time2Vec\}\(\\tau\)\_\{k\}\\;=\\;\\begin\{cases\}\\omega\_\{0\}\\tau\+\\phi\_\{0\}&k=0,\\\\ \\sin\(\\omega\_\{k\}\\tau\+\\phi\_\{k\}\)&1\\leq k\\leq d\_\{\\tau\}\-1,\\end\{cases\}\(2\)whereωk,ϕk\\omega\_\{k\},\\phi\_\{k\}are learnable parameters\. This encoding combines a linear component \(which captures monotonic time\) and Fourier components \(which capture periodic structure\), without requiring a hand\-designed positional scheme\.
After pre\-training, the encoder output is used as the trajectory embedding𝐫∈ℝdt\\mathbf\{r\}\\in\\mathbb\{R\}^\{d\_\{t\}\}\. We obtain a fixed\-size vector by flattening the encoder hidden states across allTTtimesteps\. This choice preserves the temporal structure compared to mean pooling and makes more information available to the projector\.
### 3\.2Voice Tower
A voice transmissionwwis a variable\-length waveform\. The voice tower uses a frozen large\-scale*Automatic Speech Recognition \(ASR\)*encoderℰv\\mathcal\{E\}\_\{v\}that has been pre\-trained on hundreds of thousands of hours of speech\[[30](https://arxiv.org/html/2608.28981#bib.bib11),[1](https://arxiv.org/html/2608.28981#bib.bib12)\]\. The encoder maps the waveform to a sequence of hidden states over audio frames,ℰv\(w\)∈ℝL′×dv\\mathcal\{E\}\_\{v\}\(w\)\\in\\mathbb\{R\}^\{L^\{\\prime\}\\times d\_\{v\}\}, whereL′L^\{\\prime\}depends on the waveform length, and we apply a temporal mean pool to obtain a single embedding:
𝐯=1L′∑ℓ=1L′ℰv\(w\)ℓ∈ℝdv\.\\mathbf\{v\}\\;=\\;\\frac\{1\}\{L^\{\\prime\}\}\\sum\_\{\\ell=1\}^\{L^\{\\prime\}\}\\mathcal\{E\}\_\{v\}\(w\)\_\{\\ell\}\\;\\in\\;\\mathbb\{R\}^\{d\_\{v\}\}\.\(3\)
We keepℰv\\mathcal\{E\}\_\{v\}frozen for three reasons\. First, the embedding it produces already encodes the semantic content of speech robustly across acoustic conditions, including noisy ones\[[30](https://arxiv.org/html/2608.28981#bib.bib11)\]\. Second, fine\-tuning a 1\.5B\-parameter speech encoder on ATC data risks catastrophic forgetting and demands much more training data than is currently available\. Third, freezingℰv\\mathcal\{E\}\_\{v\}keeps the number of trainable parameters of the framework small, which makes contrastive training data\-efficient\.
The choice of pooling deserves a comment\. Whisper produces a dense temporal sequence of frame\-level hidden states: at roughly100100frames per second, a typical15s15\\text\{\\,\}\\mathrm\{s\}ATC clip yields close to1,5001\{,\}500frame embeddings of dimensiondv=1280d\_\{v\}=1280\. At single\-precision floating\-point \(4B4\\text\{\\,\}\\mathrm\{B\}per scalar\), one clip alone takes about1,500×1280×4B≈7\.7MB1\{,\}500\\times 1280\\times$4\\text\{\\,\}\\mathrm\{B\}$\\approx$7\.7\\text\{\\,\}\\mathrm\{MB\}$\. Across the entire paired dataset of about83,00083\{,\}000voice samples, storing and processing the full frame\-level Whisper output would therefore require roughly650GB650\\text\{\\,\}\\mathrm\{GB\}of disk and RAM, which is impractical for the rest of the pipeline\. Reducing each clip to a single vector through mean pooling brings the per\-sample footprint down to1280×4B≈5kB1280\\times$4\\text\{\\,\}\\mathrm\{B\}$\\approx$5\\text\{\\,\}\\mathrm\{kB\}$and the full corpus to under500MB500\\text\{\\,\}\\mathrm\{MB\}, three orders of magnitude lighter\.
We selected mean pooling over alternatives such as max pooling because the arithmetic mean is a linear, information\-preserving aggregator\. Every frame contributes to the result, and no salient component of the embedding is discarded as it would be by an argmax\. Mean pooling has also been reported as a strong baseline for non\-autoregressive uses of pre\-trained speech encoders\[[3](https://arxiv.org/html/2608.28981#bib.bib27)\]\. A consequence of any temporal pooling is that the mapping from the raw waveform to the pooled embedding is not bijective, since the pooled vector cannot be inverted back to the original frame sequence, which creates a strict information bottleneck between the voice tower and the joint space\. The bijective stage of V2TATC operates downstream of this bottleneck, so it cannot recover the raw audio from a point in the joint space\. In practice, this can be partially circumvented at inference time using a look\-up table of the training data\. Given a query point in the joint space, the nearest training samples can be retrieved and their associated audio re\-used directly\. The pooled embedding therefore supports retrieval and cross\-modal generation in latent space, while access to a faithful waveform is delegated to the training data itself\.
### 3\.3Joint Embedding via Contrastive Alignment
The two\-tower outputs live in different spaces \(ℝdv\\mathbb\{R\}^\{d\_\{v\}\}andℝdt\\mathbb\{R\}^\{d\_\{t\}\}, withdv≠dtd\_\{v\}\\neq d\_\{t\}in general\)\. We project both into a common spaceℝdj\\mathbb\{R\}^\{d\_\{j\}\}through twoFeed\-Forward Neural Network\(FFNN\) projectors:
𝐯^=Pv\(𝐯\),𝐫^=Pt\(𝐫\),𝐯^,𝐫^∈ℝdj\.\\hat\{\\mathbf\{v\}\}=P\_\{v\}\(\\mathbf\{v\}\),\\qquad\\hat\{\\mathbf\{r\}\}=P\_\{t\}\(\\mathbf\{r\}\),\\qquad\\hat\{\\mathbf\{v\}\},\\hat\{\\mathbf\{r\}\}\\in\\mathbb\{R\}^\{d\_\{j\}\}\.\(4\)We thenℓ2\\ell\_\{2\}\-normalize both projections:𝐯~=𝐯^/‖𝐯^‖2\\tilde\{\\mathbf\{v\}\}=\\hat\{\\mathbf\{v\}\}/\\\|\\hat\{\\mathbf\{v\}\}\\\|\_\{2\}and𝐫~=𝐫^/‖𝐫^‖2\\tilde\{\\mathbf\{r\}\}=\\hat\{\\mathbf\{r\}\}/\\\|\\hat\{\\mathbf\{r\}\}\\\|\_\{2\}\. The normalization moves the embeddings to the unit hypersphere, which has the effect of bounding inner products to\[−1,1\]\[\-1,1\]and turning them into cosine similarities\. We then train\(Pv,Pt\)\(P\_\{v\},P\_\{t\}\)with the symmetric*Information Noise\-Contrastive Estimation*\(InfoNCE\) objective\[[35](https://arxiv.org/html/2608.28981#bib.bib28),[29](https://arxiv.org/html/2608.28981#bib.bib8)\]\. For a batch ofBBpaired observations, define the similarity matrix
Sij=𝐯~i⊤𝐫~j,i,j∈\{1,…,B\}\.S\_\{ij\}\\;=\\;\\tilde\{\\mathbf\{v\}\}\_\{i\}^\{\\top\}\\tilde\{\\mathbf\{r\}\}\_\{j\},\\qquad i,j\\in\\\{1,\\dots,B\\\}\.\(5\)The InfoNCE loss treats each pair\(i,i\)\(i,i\)as its own positive and all other in\-batch pairs as negatives, in both directions:
ℒNCE=−12B∑i=1B\[logeSii/τ∑j=1BeSij/τ\+logeSii/τ∑j=1BeSji/τ\],\\mathcal\{L\}\_\{\\text\{NCE\}\}\\;=\\;\-\\frac\{1\}\{2B\}\\sum\_\{i=1\}^\{B\}\\left\[\\log\\frac\{e^\{S\_\{ii\}/\\tau\}\}\{\\sum\_\{j=1\}^\{B\}e^\{S\_\{ij\}/\\tau\}\}\+\\log\\frac\{e^\{S\_\{ii\}/\\tau\}\}\{\\sum\_\{j=1\}^\{B\}e^\{S\_\{ji\}/\\tau\}\}\\right\],\(6\)whereτ\>0\\tau\>0is a learnable temperature\.
The temperatureτ\\taucontrols the sharpness of the softmax distribution over candidates\. A smallτ\\tauamplifies similarity differences and produces a sharper distribution, which favors precise discrimination between positive and hard negative pairs\. A largeτ\\tausoftens the distribution and encourages a more uniform assignment\. In CLIP,τ\\tauis typically learned in log\-space \(τ=exp\(θ\)\\tau=\\exp\(\\theta\)withθ\\thetatrainable\) and clamped to a positive interval to prevent collapse to zero\[[29](https://arxiv.org/html/2608.28981#bib.bib8)\]\. We adopt the same parameterization\. Theoretical analyses of InfoNCE link a smallτ\\tauto a tighter lower bound on mutual information between the two modalities\[[28](https://arxiv.org/html/2608.28981#bib.bib29)\], which provides a principled justification for the empirical observation thatτ\\tautends to drift downward during training\. Earlier multi\-modal alignment work often relied on triplet or margin losses, in which one anchor, one positive, and one negative are sampled per training step\. InfoNCE generalizes this by using all in\-batch examples as negatives simultaneously, which improves data efficiency and gradient quality at large batch sizes\[[4](https://arxiv.org/html/2608.28981#bib.bib30)\]\. The cost is that very large batches are required to expose the model to enough hard negatives, which is why batch size is treated as a first\-class hyperparameter in CLIP\-style training\.
### 3\.4Bijective Lifting via Normalizing Flows
After contrastive training, voice and trajectory embeddings that describe the same aircraft are close in the joint space\. However, the projectorsPvP\_\{v\}andPtP\_\{t\}are not invertible, because they are general FFNNs whose inverses are not defined in closed form\. To enable bidirectional translation between modalities, we train two normalizing flowsfv,ft:ℝdj→ℝdjf\_\{v\},f\_\{t\}:\\mathbb\{R\}^\{d\_\{j\}\}\\to\\mathbb\{R\}^\{d\_\{j\}\}that mimic invertible maps between each modality space \(padded toℝdj\\mathbb\{R\}^\{d\_\{j\}\}\) and the joint space\. A normalizing flow is a sequence of invertible parametric transformations with tractable Jacobian determinant\. The change\-of\-variables formula then yields an exact and differentiable expression for the log\-likelihood under the transformed distribution\[[31](https://arxiv.org/html/2608.28981#bib.bib31)\]\. The original*Real\-valued Non\-Volume Preserving*\(RealNVP\) construction\[[9](https://arxiv.org/html/2608.28981#bib.bib6)\]introduced the affine coupling layer, in which the input is split in two halves\(𝐱1,𝐱2\)\(\\mathbf\{x\}\_\{1\},\\mathbf\{x\}\_\{2\}\)along a binary mask𝐦∈\{0,1\}dj\\mathbf\{m\}\\in\\\{0,1\\\}^\{d\_\{j\}\}and transformed as
𝐲1=𝐱1,𝐲2=𝐱2⊙exp\(s\(𝐱1\)\)\+t\(𝐱1\),\\mathbf\{y\}\_\{1\}=\\mathbf\{x\}\_\{1\},\\qquad\\mathbf\{y\}\_\{2\}=\\mathbf\{x\}\_\{2\}\\odot\\exp\\bigl\(s\(\\mathbf\{x\}\_\{1\}\)\\bigr\)\+t\(\\mathbf\{x\}\_\{1\}\),\(7\)wheressandttare neural\-network conditioners and⊙\\odotis the element\-wise product\. The inverse is available in closed form:
𝐱2=\(𝐲2−t\(𝐱1\)\)⊙exp\(−s\(𝐱1\)\),𝐱1=𝐲1\.\\mathbf\{x\}\_\{2\}=\\bigl\(\\mathbf\{y\}\_\{2\}\-t\(\\mathbf\{x\}\_\{1\}\)\\bigr\)\\odot\\exp\\bigl\(\-s\(\\mathbf\{x\}\_\{1\}\)\\bigr\),\\qquad\\mathbf\{x\}\_\{1\}=\\mathbf\{y\}\_\{1\}\.\(8\)The Jacobian of the coupling is triangular, so its log\-determinant reduces tolog\|detJ\|=∑ks\(𝐱1\)k\\log\|\\det J\|=\\sum\_\{k\}s\(\\mathbf\{x\}\_\{1\}\)\_\{k\}\. Stacking coupling layers with alternating masks transforms all dimensions progressively\. ActNorm layers\[[22](https://arxiv.org/html/2608.28981#bib.bib7)\], introduced in Glow, are inserted between coupling blocks to normalize activations per dimension, which improves conditioning and training stability\. The structure of an affine coupling layer is illustrated in Fig\.[3](https://arxiv.org/html/2608.28981#S3.F3)\. More detail on the overall structure of RealNVP is given in Appendix[D\. Affine Coupling Layer in RealNVP](https://arxiv.org/html/2608.28981#Sx3.SSx4), Fig\.[21](https://arxiv.org/html/2608.28981#Sx3.F21)\.
Figure 3:Affine coupling layer of a RealNVP flow\[[9](https://arxiv.org/html/2608.28981#bib.bib6)\]\. Panel \(a\) shows the forward transformation, in which one half of the input is passed through unchanged and used to condition the scalessand translationttapplied to the other half\. Panel \(b\) shows the inverse transformation, obtained analytically by reversing the affine step on the second half while reusing the same conditioning input\. The block is invertible by construction and has a triangular Jacobian\.The voice space has dimensiondv=1280d\_\{v\}=1280, the trajectory space has dimensiondt=1792d\_\{t\}=1792, and the joint space lives inℝdj=ℝ1024\\mathbb\{R\}^\{d\_\{j\}\}=\\mathbb\{R\}^\{1024\}\. Because a normalizing flow preserves dimensionality, the bijection must operate at a single common width\. We set this working width toDmax=max\(dv,dt,dj\)=1792D\_\{\\text\{max\}\}=\\max\(d\_\{v\},d\_\{t\},d\_\{j\}\)=1792, and zero\-pad whichever vector is smaller \(typically the voice embedding and the joint vector\) up toDmaxD\_\{\\text\{max\}\}before passing it through the flow\. Rather than being discarded, the padded dimensions are retained in the reconstruction loss, and the flow learns to map them onto near\-zero values\. The resulting transformation is a distribution\-to\-distribution bijection in which the surplus dimensions are contracted toward00, which is precisely what lets a single equal\-width invertible network bridge two embedding spaces of different intrinsic dimensionality\.
### 3\.5Training objective
The two flows are trained jointly with a triple loss that combines forward, backward, and cross\-modal consistency:
ℒ=λf∑m∈\{v,t\}‖fm\(𝐞m\)−𝐣m‖22\+λb∑m∈\{v,t\}‖fm−1\(𝐣m\)−𝐞m‖22\+λc‖fv\(𝐞v\)−ft\(𝐞t\)‖22,\\mathcal\{L\}\\;=\\;\\lambda\_\{f\}\\sum\_\{m\\in\\\{v,t\\\}\}\\\!\\\!\\bigl\\\|f\_\{m\}\(\\mathbf\{e\}\_\{m\}\)\-\\mathbf\{j\}\_\{m\}\\bigr\\\|\_\{2\}^\{2\}\+\\lambda\_\{b\}\\sum\_\{m\\in\\\{v,t\\\}\}\\bigl\\\|f\_\{m\}^\{\-1\}\(\\mathbf\{j\}\_\{m\}\)\-\\mathbf\{e\}\_\{m\}\\bigr\\\|\_\{2\}^\{2\}\+\\lambda\_\{c\}\\bigl\\\|f\_\{v\}\(\\mathbf\{e\}\_\{v\}\)\-f\_\{t\}\(\\mathbf\{e\}\_\{t\}\)\\bigr\\\|\_\{2\}^\{2\},\(9\)where𝐣m\\mathbf\{j\}\_\{m\}is the joint\-space target produced by the frozen contrastive projector\. To train the higher\-capacity RealNVP without overfitting and to make it faithfully reproduce the frozen FFNN projector over its whole domain, we augment the training set with synthetic pairs\. A mesh of points sampled from the encoder\-output representation space is passed through the frozen FFNN, which yields a much larger set of \(sample, target\) pairs\. The cross term, by contrast, is still trained only on the real \(non\-synthetic\) paired samples, so as to preserve an optimal cross\-modal alignment\. The forward term forces each flow to match the contrastive projector\. The backward term forces the inverse to recover the original embedding, which is the key property for cross\-modal generation\. The cross term aligns the two flows so that paired voice and trajectory map to the same joint point\.
Note that𝐣m\\mathbf\{j\}\_\{m\}is a frozen target, so no gradient flows back into the upstream encoders or into the contrastive projectors that produced it\. V2TATC’s framework is therefore trained stagewise: the trajectory encoderℰt\\mathcal\{E\}\_\{t\}is pre\-trained first; the contrastive projectors\(Pv,Pt\)\(P\_\{v\},P\_\{t\}\)are trained next on top of frozen\(ℰv,ℰt\)\(\\mathcal\{E\}\_\{v\},\\mathcal\{E\}\_\{t\}\); and finally the two flows\(fv,ft\)\(f\_\{v\},f\_\{t\}\)are trained on top of frozen\(ℰv,ℰt,Pv,Pt\)\(\\mathcal\{E\}\_\{v\},\\mathcal\{E\}\_\{t\},P\_\{v\},P\_\{t\}\)\. This staged setup is a deliberate choice\. Jointly fine\-tuning all stages would compound the computational cost of each upstream model, and the limited size of the paired dataset makes such end\-to\-end training unstable in practice\. Stagewise training also makes it easier to swap any single component without invalidating the others\. Once trained, the chainfvf\_\{v\}thenft−1f\_\{t\}^\{\-1\}implements avoice→\\rightarrowtrajectorytranslation in latent space, and the symmetric chain implements the reverse direction\. The result is a unified architecture in which the four operations*voice→\\tojoint*,*joint→\\tovoice*,*trajectory→\\tojoint*,*joint→\\totrajectory*are all available through forward or inverse passes of the same two networks\.
Figure 4:Overall architecture and the three alignment variants\. Boxes numbered in the figure are the alternatives benchmarked for each block\.Voice tower \(left\):a frozen open\-source ASR foundation model, either \(1\) Whisper\[[30](https://arxiv.org/html/2608.28981#bib.bib11)\]or \(2\) Wav2Vec 2\.0\[[1](https://arxiv.org/html/2608.28981#bib.bib12)\]\. Whisper is retained \(Appendix[C\. Choice of the Voice Encoder](https://arxiv.org/html/2608.28981#Sx3.SSx3)\)\.Trajectory tower \(right\):a MAE built around one of four sequence encoders trained from scratch on ADS\-B deltasδ𝐗t\\delta\\mathbf\{X\}\_\{t\}, namely \(1\) a bidirectional LSTM, \(2\) a full xLSTM with matrix memory\[[2](https://arxiv.org/html/2608.28981#bib.bib32)\], \(3\) a scalar xLSTM \(sLSTM cells only\), and \(4\) a Transformer\[[36](https://arxiv.org/html/2608.28981#bib.bib33)\]\. The Transformer is retained \(Appendix[B\. Trajectory Encoder Backbones](https://arxiv.org/html/2608.28981#Sx3.SSx2)\)\.Joining stage \(center\):three strategies compared in Section[5\.4](https://arxiv.org/html/2608.28981#S5.SS4), namely \(1\) contrastive FFNN projectors followed by a RealNVP bijective stage, which is the baseline used throughout the article, \(2\) a single pair of RealNVP flows trained contrastively, doing both jobs at once, and \(3\) one RealNVP mapping the two modality embeddings directly into one another, with no joint space, which isolates the contribution of the contrastive stage\. Red vector spaces give the dimension of the space each sample lives in, and orange badges give the parameter count of each block\. The voice tower connects to the joining stage through a temporal mean pool over the frame\-level hidden states\.
## 4Implementation
This section describes a concrete instantiation of the framework over the San Francisco Bay Area\. Trajectory and audio streams are collected concurrently and in real\-time by a multi\-threaded orchestrator, each source being polled on its own schedule\. Recording the two modalities in parallel over the same time window is what makes the later temporal alignment possible, and the two subsections below detail each source in turn\. The notation introduced in Section[3](https://arxiv.org/html/2608.28981#S3)is kept unchanged\. An overview of all the architectural pieces that we tested and of the three alignment variants compared in Section[5\.4](https://arxiv.org/html/2608.28981#S5.SS4)is given in Fig\.[4](https://arxiv.org/html/2608.28981#S3.F4), and the rest of this section discusses each block one by one\.
### 4\.1Trajectory Data
ADS\-B is a surveillance system in which each aircraft determines its own state vector from onboard navigation sources and broadcasts it periodically, without interrogation from the ground\. Each message carries the ICAO 24\-bit transponder code, the callsign, a timestamp,*World Geodetic System 1984*\(WGS84\) coordinates\(ϕ,λ,h\)\(\\phi,\\lambda,h\), ground speedvv, true trackθ\\theta, and vertical rateh˙\\dot\{h\}\. We collect ADS\-B data from two public sources\. The first one is the OpenSky Network\[[34](https://arxiv.org/html/2608.28981#bib.bib34)\]\([opensky\-network\.org](https://opensky-network.org/)\), a research\-oriented feed that aggregates messages from a worldwide volunteer receiver network and offers a stable historical API\. The second one is[adsb\.lol](https://adsb.lol/), an unauthenticated open feed that provides higher polling frequency at the cost of less complete historical coverage\. For operational reasons we eventually retained only the OpenSky Network feed for the experiments reported here, but future work would benefit from incorporating the higher\-frequency adsb\.lol stream as well\.
OpenSky state vectors are polled at30s30\\text\{\\,\}\\mathrm\{s\}intervals, and adsb\.lol at roughly1s1\\text\{\\,\}\\mathrm\{s\}intervals when available\. The geographic boundaries of the surveillance and voice streams are summarized in Fig\.[1](https://arxiv.org/html/2608.28981#S1.F1)\. ADS\-B is collected over a rectangular bounding box around the Bay Area, and voice audio is captured at the four airports whose ATC frequencies are streamed by LiveATC\. Table[1](https://arxiv.org/html/2608.28981#S4.T1)summarizes the raw ADS\-B data collected over the Bay Area, which serves as the pre\-training material for the trajectory encoder\.
Table 1:Statistics of the raw ADS\-B trajectory dataset collected from the OpenSky Network over the San Francisco Bay Area\. The final sequences are used as the pre\-training corpus for the trajectory encoder\.Raw geodetic coordinates are not suitable as a neural\-network input\. Three issues need to be fixed\. First, longitude is non\-linear, since1°1\\text\{\\,\}\\mathrm\{\\SIUnitSymbolDegree\}of longitude does not correspond to the same physical distance at every latitude\. Second, the three spatial dimensions have incompatible units \(degrees vs\. meters\), which would dominate any per\-feature normalization\. Third, true track is a circular variable \(359∘→0∘359^\{\\circ\}\\to 0^\{\\circ\}wrap\-around\), which standardzz\-score normalization cannot handle\. We resolve these issues by converting each ADS\-B sample into a local tangent\-plane Cartesian frame centered on a reference point\(ϕ0,λ0\)=\(37∘N,123∘W\)\(\\phi\_\{0\},\\lambda\_\{0\}\)=\(37^\{\\circ\}\\,\\text\{N\},123^\{\\circ\}\\,\\text\{W\}\):
x\\displaystyle x=\(R\+h\)\(λ−λ0\)cosϕ,\\displaystyle=\(R\+h\)\(\\lambda\-\\lambda\_\{0\}\)\\cos\\phi,\(10\)y\\displaystyle y=\(R\+h\)\(ϕ−ϕ0\),\\displaystyle=\(R\+h\)\(\\phi\-\\phi\_\{0\}\),z\\displaystyle z=h\.\\displaystyle=h\.whereR=6371kmR=$6371\\text\{\\,\}\\mathrm\{km\}$is the mean Earth radius andhhis the geometric altitude of the aircraft above the reference ellipsoid\. Velocity components are projected onto the same axes throughvx=vcosθv\_\{x\}=v\\cos\\theta,vy=vsinθv\_\{y\}=v\\sin\\theta,vz=h˙v\_\{z\}=\\dot\{h\}\. The resulting feature vector hasF=6F=6dimensions per timestep, all expressed in meters or meters per second\. Each flight segment is then sliced into fixed\-length windows𝐗∈ℝT×F\\mathbf\{X\}\\in\\mathbb\{R\}^\{T\\times F\}ofT=14T=14consecutive samples\. Window beginnings and ends are detected automatically when \(i\) the callsign changes or \(ii\) the gap between consecutive samples exceeds two minutes\. A continuity buffer of 13 points is carried over between consecutive raw data files so that long flights are not truncated artificially at file boundaries\.
The trajectory encoderℰt\\mathcal\{E\}\_\{t\}is pre\-trained as a MAE following the principle of Section[3](https://arxiv.org/html/2608.28981#S3)\. The MAE was originally introduced for natural images\[[14](https://arxiv.org/html/2608.28981#bib.bib16)\], and the recipe extends naturally to other domains as long as each input can be tokenized into a sequence\. In our case, each ADS\-B timestep is one token, and the encoder processes a sequence ofT=14T=14tokens\. The decoder reconstructs the normalized deltazz\-score of each feature on the masked positions\. We organize pre\-training in two stages\. Stage 1 operates directly on spherical coordinates\(ϕ,λ,h,v,θ,h˙\)\(\\phi,\\lambda,h,v,\\theta,\\dot\{h\}\)with normalized first\-order differences as both encoder input and decoder target\. Stage 2 operates on the Cartesian features\(x,y,z,vx,vy,vz\)\(x,y,z,v\_\{x\},v\_\{y\},v\_\{z\}\)of Eq\. \([10](https://arxiv.org/html/2608.28981#S4.E10)\), with absolute normalized positions as input and Cartesian deltas as the reconstruction target\. This eliminates the circular discontinuity ofθ\\thetaand the unit mismatch between angular and linear features\.
We compared four backbone architectures forℰt\\mathcal\{E\}\_\{t\}, all shown in the trajectory tower of Fig\.[4](https://arxiv.org/html/2608.28981#S3.F4), with hidden dimensiond=128d=128and four layers: a Transformer\[[36](https://arxiv.org/html/2608.28981#bib.bib33)\], a bidirectional LSTM, a scalar xLSTM \(sLSTM cells only\), and a full xLSTM with matrix memory\[[2](https://arxiv.org/html/2608.28981#bib.bib32)\]\. The Transformer is retained as the encoder for the rest of the framework\. The detailed quantitative comparison of the four backbones is deferred to Appendix[B\. Trajectory Encoder Backbones](https://arxiv.org/html/2608.28981#Sx3.SSx2)\. The trajectory embedding𝐫\\mathbf\{r\}is obtained by flattening the encoder output across timesteps, givingdt=T⋅d=14⋅128=1792d\_\{t\}=T\\cdot d=14\\cdot 128=1792\. Figure[5](https://arxiv.org/html/2608.28981#S4.F5)shows the UMAP projection of the trajectory latent space produced by the Transformer MAE\. This is the representation fed to the contrastive projectorPtP\_\{t\}in the next stage\.
Figure 5:UMAP projection of the trajectory latent space produced by the Transformer MAE on the OpenSky San Francisco dataset\. The points organize by flight phase even though no phase label was used during pre\-training, which indicates that the encoder captures the deepest essence of each window\. Each cluster corresponds to an elementary behavior, namely a curved ground track or a change of altitude, and this alone is enough to tell general aviation apart from commercial traffic\. Commercial flights follow standard procedures and repeat the same maneuvers from one flight to the next, so they form the tight and clearly separated clusters annotated on the plot\. General aviation trajectories are far less standardized and collapse instead into a single continuous blob that connects all phases\. Part of the structure is geographic rather than kinematic, since each Bay Area airport produces its own family of approach and departure trajectories and therefore its own cluster for a given phase label\.
### 4\.2Voice Data
ATC radio audio is operated by[LiveATC\.net](https://www.liveatc.net/)\[[26](https://arxiv.org/html/2608.28981#bib.bib39)\], a community\-based network of volunteer receivers that broadcasts live ATC frequencies publicly and provides historical audio archives on request\. For this study, we obtained the audio archives corresponding to eight Bay Area frequencies, listed in Table[3](https://arxiv.org/html/2608.28981#S4.T3), spanning ground, tower, approach, and en\-route sectors of KSFO, KOAK, KSJC, and Moffett Field\. The archives cover12h12\\text\{\\,\}\\mathrm\{h\}per day over four consecutive days, from*11 to 14 February 2026*, and are delivered as a sequence of30min30\\text\{\\,\}\\mathrm\{min\}MP3 chunks per frequency\. ID3 metadata embedded in each chunk contains the exact UTC start and end times, which is essential for later alignment with the ADS\-B stream\. A small UTC\-anchored buffer is kept between consecutive chunks to avoid losing transmissions that span the chunk boundary\.
VRAM probetorch\.cuda\.synchronize\(\)free\_gb = mem\_get\_info\(0\)Model pool builderLoad model→\\tore\-measure free VRAMRepeat until free VRAM<<4 GB headroomfree VRAMModelQueueW1W\_\{1\}W2W\_\{2\}WNW\_\{N\}NNinstancesfile\_1\.mp3file\_2\.mp3file\_3\.mp3⋮\\vdotsThread 1acquire\(WkW\_\{k\}\)Whisper inferenceThread 2acquire\(WkW\_\{k\}\)Whisper inferenceThread 3acquire\(WkW\_\{k\}\)Whisper inferenceJSON \+ \.tmp→\\toatomic renameJSON \+ \.tmp→\\toatomic renameJSON \+ \.tmp→\\toatomic renameSRT merge\(after every file\)→\\topartial SRT on diskFigure 6:VRAM\-aware transcription pipeline\. A probe measures the free GPU memory, a builder fills a pool with as many Whisper instances as fit under a configurable headroom, and a thread pool of workers consumes the pool to transcribe MP3 chunks in parallel\. Each completed file is merged into a single SRT file with absolute UTC timestamps\.Each MP3 chunk is transcribed by Whisper large\-v3\[[30](https://arxiv.org/html/2608.28981#bib.bib11)\], a large\-scale ASR foundation model pre\-trained on 680,000 hours of multi\-domain speech\. Whisper and Wav2Vec 2\.0 are the two voice encoders shown in the voice tower of Fig\.[4](https://arxiv.org/html/2608.28981#S3.F4)\. The same encoder is reused for two purposes: \(i\) producing the text transcription used by the callsign\-matching logic of Section[4\.3](https://arxiv.org/html/2608.28981#S4.SS3)and \(ii\) producing the voice embedding𝐯\\mathbf\{v\}consumed by the contrastive stage\. Using a single high\-capacity encoder for both roles keeps the pipeline simple and ensures that the embedding used downstream is exactly the one from which the transcription is decoded, so a point in the voice latent space can in principle be turned back into a plausible utterance through the same network\. The choice of Whisper over Wav2Vec 2\.0\[[1](https://arxiv.org/html/2608.28981#bib.bib12)\], namely a community\-fine\-tuned ATC\-specific variant, is justified in Appendix[C\. Choice of the Voice Encoder](https://arxiv.org/html/2608.28981#Sx3.SSx3)on the basis of \(i\) the qualitative structure of the latent space, \(ii\) the quality of the transcriptions obtained when decoding from the embedding, and \(iii\) a sweep over projector sizes\.
*Voice Activity Detection \(VAD\)*is run before transcription using Silero VAD, with parameters tuned for the short, clipped style of ATC speech: detection threshold0\.30\.3\(lower than the default0\.50\.5\), minimum speech duration50ms50\\text\{\\,\}\\mathrm\{ms\}, and800ms800\\text\{\\,\}\\mathrm\{ms\}of pad on either side of each detected segment\. These settings recover short clearances that the default settings would discard, while keeping false positives manageable\.
In practice, transcription was by far the most time\-consuming stage of the entire study and the main bottleneck of the pipeline\. The difficulty is twofold\. Firstly, Whisper large\-v3 is slow to run over the full volume of archived audio, and secondly its GPU memory footprint is incompatible with naive parallel processing, so simply launching many instances at once quickly exhausts the available VRAM\. We thus adopt a VRAM\-aware model pool, sketched in Fig\.[6](https://arxiv.org/html/2608.28981#S4.F6)\. Whisper instances are loaded one by one and the process stops as soon as the remaining free memory drops below a configurable headroom \(typically4GB4\\text\{\\,\}\\mathrm\{GB\}\)\. The resulting pool ofNNinstances is consumed by a thread pool, with each worker grabbing a model, transcribing a single MP3, and returning the model to the pool\. Outputs are written atomically through a\.tmprename so that a partial run can be resumed without re\-transcribing files already on disk\. Inference uses half\-precision float on GPU and integer quantization on CPU as a fallback\.
A transcribed line of ATC speech such as “United two three four, descend and maintain five thousand” carries the callsign of the addressed aircraft\. We extract this information through a cascade of regular\-expression patterns that go from most to least specific:
Patterns tolerate arbitrary whitespace and hyphens between characters, because ASR systems frequently insert separators inside identifiers \(e\.g\. “U\-A\-L 2\-3\-4”\)\. Lookups across consecutive transcription lines are also performed, since a callsign can be split across two transmissions\. Telephony designators are normalized through a curated mapping, directly scraped from this[FAA website](https://www.faa.gov/air_traffic/publications/atpubs/cnt_html/chap3_section_3.html)\(e\.g\. “Air France”→\\toAFR\), and digit words are converted from English to numerals \(e\.g\. “two three four”→\\to234\)\. When several patterns match the same line, only the highest\-priority match is kept\. The distribution of match types in the final dataset is shown in Fig\.[7](https://arxiv.org/html/2608.28981#S4.F7)\.
\(a\)Sample counts per ATC frequency\.
\(b\)Distribution of callsign matching strategies\.
Figure 7:Composition of the paired voice–trajectory dataset\.
### 4\.3Linking Modalities
The overall linking pipeline takes the raw ADS\-B and audio streams and produces the final HDF5 shards\. Its inputs, intermediate processes, and outputs are summarized in Fig\.[8](https://arxiv.org/html/2608.28981#S4.F8)\. The pairing itself operates flight\-first\. For every ADS\-B segment identified by its ICAO 24 code, callsign, and time window\[tstart,tend\]\[t\_\{\\text\{start\}\},t\_\{\\text\{end\}\}\], we search the transcription file for all lines whose callsign matches and whose timestamp falls within\[tstart−Δpre,tend\+Δpost\]\[t\_\{\\text\{start\}\}\-\\Delta\_\{\\text\{pre\}\},t\_\{\\text\{end\}\}\+\\Delta\_\{\\text\{post\}\}\]withΔpre=Δpost=60s\\Delta\_\{\\text\{pre\}\}=\\Delta\_\{\\text\{post\}\}=$60\\text\{\\,\}\\mathrm\{s\}$\. This direction is significantly more robust than the opposite one \(searching for flights given a transcribed callsign\) because the temporal constraint eliminates false positives from other flights with similar callsigns operating elsewhere\.
ADS\-BFilteringAudioTranscriptionMultimodalLinkingJointAssemblyADS\-B CSV filesATC MP3 filesNetCDF4trajectoriesSRT files\(UTC timestamps\)Callsignmatches JSONHDF5 shards\(waveform \+ traj\)Input dataProcessingOutput filesFigure 8:Linking pipeline from raw sources to joint HDF5 shards\. The first row lists the inputs, the second row the processing stages, and the third row the intermediate or final output files\.Each matched pair produces a sample of the joint dataset\. The sample contains both modalities and their metadata, namely \(i\) a raw audio clip extracted from the MP3 at the matched timestamps, with a750ms750\\text\{\\,\}\\mathrm\{ms\}tail pad to capture the end of each transmission cleanly; \(ii\) a 14\-step trajectory window centered on the speech event \(nine points before, current, four after\); and \(iii\) a metadata record with the callsign, the matched timestamps, the match type, the source frequency, and the source MP3 filename\. The data is serialized into chunked HDF5 shards organized by tower, with audio stored as variable\-lengthint16arrays at16kHz16\\text\{\\,\}\\mathrm\{kHz\}and trajectories stored asfloat32arrays of shape\(N,T,7\)\(N,T,7\)where the seventh column carries the absolute timestamp\. The high\-level statistics of the resulting joint corpus are reported in Table[2](https://arxiv.org/html/2608.28981#S4.T2), the per\-frequency sample distribution in Table[3](https://arxiv.org/html/2608.28981#S4.T3), and the geographic footprint together with the marginal distributions of altitude, ground speed, and vertical rate in Fig\.[9](https://arxiv.org/html/2608.28981#S4.F9)\.
Table 2:Summary of the paired voice–trajectory dataset over the San Francisco Bay Area\.MetricValueControl tower frequencies \(KSFO / KOAK / KSJC / Moffett\)8Unique aircraft \(ICAO24 codes\)962Unique callsigns1,411Paired voice–trajectory samples52,285Trajectory window lengthTT\(timesteps\)14Median ADS\-B sampling intervalΔt\\Delta t\(s\)27Median phrase duration \(s\)6\.0Median transcription length \(words\)12Median altitude \(m\)899Median ground speed \(m s\-1\)96Table 3:Distribution of paired voice–trajectory samples across the eight monitored ATC frequencies\. The three NorCal en\-route frequencies together account for more than 72 % of the dataset\.ATC frequency / positionSamples% of totalksfo\_koak\_norcal\_dep\(NorCal Departure\)21,10240\.4ksfo\_norcal\_app2\_l\(NorCal Approach L\)12,95424\.8koak\_del\_gnd\_twr\(KOAK Delivery / Ground / Twr\)7,70114\.7ksjc\_twr2\(KSJC Tower\)5,87511\.2koak\_norcal\_app\(KOAK NorCal Approach\)3,7067\.1ksjc\_del\_gnd\(KSJC Delivery / Ground\)5801\.1ksfo\_del\_gnd\_alt\(KSFO Delivery / Ground alt\.\)2540\.5knuq\_gnd\_twr\(KNUQ Ground / Tower\)1130\.2Total52,285100\(a\)Aircraft position at sentence start, colored by ATC frequency\. One dot per flight during two weeks\.
\(b\)Distributions of altitude, ground speed, and vertical rate over all trajectory points\. The joint dataset contains a large number of data points with zero altitude, zero speed, and zero vertical rate, since most communications take place on the ground during taxiing\.
Figure 9:Statistical overview of the paired joint dataset over the San Francisco Bay Area\.A side effect of the linking stage is a drastic reduction of the effective dataset size\. The raw ADS\-B feed contains millions of trajectory points, but only the segments that can be matched to a transcribed callsign within the temporal window survive into the joint dataset\. Figure[10](https://arxiv.org/html/2608.28981#S4.F10)illustrates how the altitude distribution evolves as the data move along the pipeline, from the raw ADS\-B data to the cleaned trajectory dataset and finally to the paired voice–trajectory dataset\. The general shape of the distribution is preserved, but the dataset becomes increasingly biased toward operations that produce transcribable voice activity, namely commercial flights on instrumented sectors\. This bias is a known limitation of the current corpus and motivates the multi\-source extension discussed in Section[6](https://arxiv.org/html/2608.28981#S6)\.
Figure 10:Evolution of the altitude distribution as the dataset moves from the raw ADS\-B feed to the cleaned trajectory dataset and finally to the joint voice–trajectory dataset\. Each pairing stage discards data points that cannot be matched, which shrinks the effective dataset and biases it toward instrumented commercial traffic\.The joint architecture is trained in two stages, as illustrated in Fig\.[11](https://arxiv.org/html/2608.28981#S4.F11)\.
Voiceℰv\\mathcal\{E\}\_\{v\}Traj\.ℰt\\mathcal\{E\}\_\{t\}FFNNPvP\_\{v\}FFNNPtP\_\{t\}JointℒNCE\\mathcal\{L\}\_\{\\text\{NCE\}\}\(a\) Stage 1: contrastive trainingVoiceℰv\\mathcal\{E\}\_\{v\}Traj\.ℰt\\mathcal\{E\}\_\{t\}RealNVPfvf\_\{v\}RealNVPftf\_\{t\}Jointℒ\\mathcal\{L\}\(Eq\.[9](https://arxiv.org/html/2608.28981#S3.E9)\)\(b\) Stage 2: bijective mimickingFigure 11:Two\-stage training pipeline\. In stage \(a\) the two FFNN projectors are trained jointly by the InfoNCE objective\. The frozen joint\-space targets are then used in stage \(b\) to train two normalizing flows that mimic a bijective map between each modality space and the joint space\. For stage \(b\), synthetic data generated by direct inference are used in addition to real data to increase the representativeness of the sampling of the initial space\.#### Stage 1: Contrastive Training
The two FFNN projectors are trained jointly with the symmetric InfoNCE objective of Eq\. \([6](https://arxiv.org/html/2608.28981#S3.E6)\)\. The upstream encodersℰv,ℰt\\mathcal\{E\}\_\{v\},\\mathcal\{E\}\_\{t\}are frozen\. Each projector has a single hidden layer of width 4096 with a SELU activation and a dropout of0\.20\.2, followed by a linear output layer\. We use the AdamW optimizer with learning rate10−410^\{\-4\}, weight decay10−410^\{\-4\}, batch sizeB=512B=512, and gradient clipping at norm1\.01\.0\. The temperatureτ\\tauis parameterized asτ=exp\(θ\)\\tau=\\exp\(\\theta\)withθ\\thetalearnable and clamped toτ∈\[0\.04,100\]\\tau\\in\[0\.04,100\]to prevent collapsing or explosion\. Learning\-rate scheduling is performed byReduceLROnPlateauon the validation loss, with patience seven and factor0\.50\.5, and training stops when no improvement is seen for fifteen consecutive epochs\. The complete set of hyperparameters is summarized in Table[4](https://arxiv.org/html/2608.28981#S4.T4)\.
#### Stage 2: Bijective Lifting
The contrastive projectors are frozen and used as ground\-truth oracles\. The two normalizing flows are then trained with the loss in Eq\. \([9](https://arxiv.org/html/2608.28981#S3.E9)\), with weightsλf=λb=0\.5\\lambda\_\{f\}=\\lambda\_\{b\}=0\.5andλc=0\.1\\lambda\_\{c\}=0\.1\. Each flow contains eight affine coupling blocks with conditioner MLPs of hidden width\[256,256\]\[256,256\], interleaved with ActNorm layers\. Because the real paired data only cover a limited region of the embedding space, the training set is augmented with4×1044\\times 10^\{4\}synthetic embeddings per modality: half drawn from the per\-dimension bounding box of real data, half from Gaussian perturbations centered on random real samples withσd=0\.5×stdd\\sigma\_\{d\}=0\.5\\times\\text\{std\}\_\{d\}\. Synthetic embeddings receive their joint\-space target by passing through the frozen contrastive projector\. The optimizer is again AdamW with learning rate10−410^\{\-4\}, weight decay10−410^\{\-4\}, and batch size 256\.
Table 4:Hyperparameters of the contrastive training stage \(Stage 1\)\.Network capacity vs\. data scale
The two projectors contain on the order of10710^\{7\}trainable parameters each: the voice projector has approximately1280⋅4096\+4096⋅1024≈9\.5×1061280\\cdot 4096\+4096\\cdot 1024\\approx 9\.5\\times 10^\{6\}weights, and the trajectory projector approximately1792⋅4096\+4096⋅1024≈1\.1×1071792\\cdot 4096\+4096\\cdot 1024\\approx 1\.1\\times 10^\{7\}weights\. The two normalizing flows together carry a comparable number of parameters, dominated by the conditioner MLPs of the coupling layers\. The validation set used for early stopping contains on the order of5×1035\\times 10^\{3\}paired samples\. A sample here is not a scalar observation\. On the trajectory side alone it carries a14×714\\times 7window of seven recorded channels, and it is paired with a full voice clip, so the alignment has to satisfy on the order of10510^\{5\}scalar constraints rather than5×1035\\times 10^\{3\}\. That figure remains below the parameter count, and the projectors are therefore large relative to the data they see\. What keeps the optimization well behaved is not the ratio itself but the regularization around it, namely the dropout, the weight decay, and the early stopping that stop the projectors from memorizing the validation set\. The persistent train–validation gap observed in Section[5\.1](https://arxiv.org/html/2608.28981#S5.SS1)indicates that a further reduction of projector width would be a sensible follow\-up, which we did not pursue here because retrieval quality is already informative enough to support the downstream studies\.
## 5Results
We evaluate V2TATC on six tasks that probe different aspects of the joint space\.
### 5\.1Cross\-Modal Retrieval
We first measure cross\-modal retrieval\. Given a voice query in the validation set, we rank all candidate trajectories by cosine similarity in the joint space and record whether the true trajectory falls in the topKK\. The metric*Recall atKK*\(R@KK\) is the fraction of queries for which this is the case\. R@KKvalues are reported in Table[5](https://arxiv.org/html/2608.28981#S5.T5)and the corresponding training curves are shown in Fig\.[12](https://arxiv.org/html/2608.28981#S5.F12)\.
Table 5:Cross\-modal retrieval performance in the joint space \(voice query, trajectory candidate,N=5,229N=5\{,\}229validation candidates\)\. Recall values are percentages; InfoNCE loss and temperature are dimensionless\. The trained model improves over random retrieval by roughly three orders of magnitude on R@1\.\(a\)Recall@KKon the5,2295\{,\}229validation candidates over training epochs\. R@1, R@5 and R@10 increase steadily and saturate after about 200 epochs at final values23\.5%23\.5\\%,59\.7%59\.7\\%and72\.6%72\.6\\%, three orders of magnitude above the random retrieval baseline \(R@1≈0\.02%\\text\{R@1\}\\approx 0\.02\\%\)\.
\(b\)Symmetric InfoNCE loss during training\. The training loss \(solid blue\) reaches a minimum of approximately0\.450\.45, while the validation loss \(dashed red\) stabilizes near1\.591\.59, reflecting the limited diversity of the paired dataset rather than overfitting\. Both curves are averaged over the voice\-to\-trajectory and trajectory\-to\-voice directions\.
Figure 12:Contrastive training dynamics: retrieval performance \(a\) and InfoNCE loss \(b\) over training epochs\.The trained model achieves R@1 of0\.2350\.235and R@10 of0\.7260\.726over5,2295\{,\}229validation candidates\. The random baseline isK/Nval≈2×10−4K/N\_\{\\text\{val\}\}\\approx 2\\times 10^\{\-4\}forK=1K=1\. The model therefore outperforms random retrieval by roughly three orders of magnitude, as quantified explicitly in Table[5](https://arxiv.org/html/2608.28981#S5.T5)\. The fact that R@10 is roughly three times R@1 suggests that the model frequently confuses the correct trajectory with a few similar candidates \(typically aircraft of the same operator on the same sector\), but it almost always recovers the right region of the joint space\. Thus, this analysis illustrates that our bijection is indeed established between submanifolds of our two spaces and not directly between pairs of samples\.
The learned temperatureτ\\taudrifts down to its floor value of0\.040\.04during training \(Fig\.[13](https://arxiv.org/html/2608.28981#S5.F13)\), which is consistent with the InfoNCE literature, where a small temperature sharpens the softmax distribution and pushes harder on negatives\. The clamp interval\[0\.04,100\]\[0\.04,100\]used here matches the CLIP defaults\[[29](https://arxiv.org/html/2608.28981#bib.bib8)\], and the steady drift toward the lower bound is the expected behavior on a well\-aligned dataset\. However, the wayτ\\tauhits the floor in our setting is informative on its own\. Instead of decreasing smoothly along the validation loss, it drops abruptly within a few epochs and then stays pinned at the floor\. Recent analyses of InfoNCE\[[32](https://arxiv.org/html/2608.28981#bib.bib35)\]show that the assumptions behind its theoretical guarantees rarely hold in practice, and in particular that the latent factors of positive pairs vary to very unequal extents\. We read the abrupt drop as a signature of the dataset rather than of the architecture\. The dataset does not sample the embedding space uniformly enough, so a small number of clearly separable directions are enough to drive the loss down sharply, and the model has no incentive to keepτ\\tauabove the clamp\. The persistent train/validation gap \(training loss≈0\.45\\approx 0\.45vs\. validation loss≈1\.59\\approx 1\.59at the best epoch\) is consistent with this reading, and the limiting factor at this stage is the diversity of the paired dataset rather than the capacity of the projectors\.
\(a\)Evolution of the learnable contrastive temperatureτ\\tau\(parameterized in log\-space\)\. The temperature drops abruptly to its lower boundτmin=0\.04\\tau\_\{\\min\}=0\.04\(dotted line\) within the first few epochs and remains pinned there\. The abrupt collapse reads as a signature of the limited diversity of the data, which provides a small set of clearly separable directions sufficient to drive the loss sharply downward\.
\(b\)Distribution of cosine similarities between voice and trajectory embeddings projected into the joint space\. Positive \(matched\) pairs should tend toward11and negative \(mismatched\) pairs concentrate near00\. The small overlap corresponds mainly to aircraft of the same operator on the same sector, which produce structurally similar voice phrases and motion profiles\.
Figure 13:Sharpness of the contrastive alignment: temperature evolution \(a\) and cosine similarity distributions \(b\)\.
### 5\.2Trajectory Forecast
We next evaluate the accuracy of trajectory forecasting\. The task is to predict the nextH=4H=4ADS\-B steps givenT−H=10T\-H=10steps of context\. We compare two paths\. Path A uses the MAE encoder directly, with the last four timesteps masked and reconstructed by the reconstruction head trained during MAE pre\-training\. Path B uses the cross\-modal chain voice→\\rightarrowjoint→\\rightarrowtrajectory\. A voice phrase is mapped to the trajectory side throughfvf\_\{v\}thenft−1f\_\{t\}^\{\-1\}, and a small learned head \(the StepPredictor\) predicts the next four steps from the recovered representation\. Path A is therefore a pure surveillance\-based forecast and serves as an upper bound\. Path B measures how much of that capability is preserved when the only input is a voice phrase\.
Table 6:Short\-horizon trajectory forecast \(next four ADS\-B steps from ten\-step context\)\. MSE values are in normalized delta z\-score space \(dimensionless\)\.
Figure 14:Trajectory forecast results\.Top: average predicted profile against the ground truth over the full validation set, one sub\-panel per feature \(projected back intoΔ\\Deltalon,Δ\\Deltalat,Δ\\Deltaalt,Δv\\Delta v,Δθ\\Delta\\theta,Δh˙\\Delta\\dot\{h\}\)\. Path A \(direct MAE, solid\) tracks the ground\-truth mean closely, in line with its global MSE of0\.0710\.071\. Path B \(cross\-modal chain, dashed\) keeps the qualitative trend on position features but is offset on heading and vertical rate, and the zero\-shot variant \(dotted\) deviates on all of them\.Bottom: per\-feature and per\-horizon MSE in normalized deltazz\-score space, for Path A \(*left*\) and for the zero\-shot Path B \(*right*\)\. Heading \(Δθ\\Delta\\theta\) and vertical rate \(Δh˙\\Delta\\dot\{h\}\) are the hardest targets for both paths\. The error stays flat across the four horizons, which follows from the single\-shot head\.Results are reported in Table[6](https://arxiv.org/html/2608.28981#S5.T6)and illustrated in Fig\.[14](https://arxiv.org/html/2608.28981#S5.F14)\. Path A reaches a global*Mean Squared Error*\(MSE\) of0\.0710\.071in the normalized deltazz\-score space\. Path B reaches a global MSE of0\.7100\.710, an order of magnitude higher\. The gap is explained by the residual distribution shift between MAE latents and flow\-reconstructed latents\. The bijective stage matches paired points on average, but it does not equate the marginal distributions, so the downstream MAE reconstruction head, which was trained on real latents, is mildly out\-of\-distribution when fed flow\-reconstructed ones\. The naive zero\-shot path that skips the StepPredictor degrades to an MSE of2\.02\.0, which confirms the importance of even a small task\-specific head\.
To make these numbers concrete, we convert the per\-feature MSE to physical units using the Stage 1 per\-feature normalization constants\. The conversion isRMSEphysical=MSEz⋅σfeature\\text\{RMSE\}\_\{\\text\{physical\}\}=\\sqrt\{\\text\{MSE\}\_\{z\}\}\\cdot\\sigma\_\{\\text\{feature\}\}where, for the OpenSky San Francisco dataset,σlon=0\.273∘\\sigma\_\{\\text\{lon\}\}=0\.273^\{\\circ\},σlat=0\.292∘\\sigma\_\{\\text\{lat\}\}=0\.292^\{\\circ\},σalt=1867m\\sigma\_\{\\text\{alt\}\}=$1867\\text\{\\,\}\\mathrm\{m\}$, andσtrk=105\.6∘\\sigma\_\{\\text\{trk\}\}=105\.6^\{\\circ\}\. At the Bay Area latitudeϕ≈37∘\\phi\\approx 37^\{\\circ\}N, one degree of longitude corresponds to about89km89\\text\{\\,\}\\mathrm\{km\}and one degree of latitude to about111km111\\text\{\\,\}\\mathrm\{km\}\. For Path A this gives an RMS position error of roughly1\.5km1\.5\\text\{\\,\}\\mathrm\{km\}per predicted step, an RMS altitude error close to180m180\\text\{\\,\}\\mathrm\{m\}\(≈580\\approx 580ft\), and an RMS heading error near42∘42^\{\\circ\}\. For Path B with the StepPredictor head the same conversion gives roughly3\.1km3\.1\\text\{\\,\}\\mathrm\{km\},280m280\\text\{\\,\}\\mathrm\{m\}\(≈910\\approx 910ft\), and86∘86^\{\\circ\}\. For the zero\-shot variant the heading error reaches149∘149^\{\\circ\}and altitude≈2\.1km\\approx$2\.1\\text\{\\,\}\\mathrm\{km\}$, which is essentially uninformative\.
A per\-feature breakdown shows that the first three quantities \(longitude, latitude, altitude\) are by far the easiest targets, while the next three \(ground speed, true track, vertical rate\) are the hardest\. This split is natural, because positions are essentially integrated velocities, and since the model already observes the position trace, predicting the next delta is close to predicting the local velocity, which is itself close to an integration of what was just seen\. The velocity\-related quantities, by contrast, depend on acceleration, which is not directly observable from the input window\. A controller’s instruction such as “climb and maintain flight level three five zero” or “turn left heading two seven zero” can radically change either feature within a few seconds, and a surveillance\-only forecast simply has no way to anticipate that change\.
A second observation from Fig\.[14](https://arxiv.org/html/2608.28981#S5.F14)is that the error does not grow with the horizon, and the per\-step MSE stays roughly flat across the four predicted steps\. This is a direct benefit of the single\-shot prediction head, which outputs the four future steps in one forward pass instead of feeding its own output back as context\. Autoregressive forecasters typically see an error budget that compounds with each step\. The single\-shot head trades this compounding for a slightly worse per\-step accuracy att=1t=1, but it keeps the four\-step horizon stable\. Longer horizons can still be obtained by repeating the prediction over successive windows, in which case the compounding behavior would re\-appear and would have to be managed at the outer loop\.
### 5\.3Next Sentence Prediction
We also evaluate the reverse direction: given a trajectory window, retrieve the most likely voice phrase among the validation set\. This is performed by ranking voice candidates by cosine similarity toPt\(𝐫\)P\_\{t\}\(\\mathbf\{r\}\)in the joint space\. The retrieved phrase is then compared with the ground\-truth transcription\. The pattern of retrieval errors is informative\. Aircraft on similar flight phases \(e\.g\., climbing departures out of KSFO\) retrieve voice phrases issued by the same controller and following the same phraseology, even when the exact addressee differs\. This suggests that the joint space captures the coarse semantics of ATC instructions \(instruction type, sector identity, flight phase\) in addition to fine\-grained identity matching\. Per\-tower retrieval breakdowns confirm that towers with stereotyped exchanges \(ground, delivery\) yield higher R@1 than towers with longer, more diverse instructions \(NorCal departure\), and that the cluster structure observed in the joint space \(Section[5\.5](https://arxiv.org/html/2608.28981#S5.SS5)\) is consistent with this distinction\.
### 5\.4Ablation Study on the Joining Stage
We compare the three alignment architectures sketched in Fig\.[4](https://arxiv.org/html/2608.28981#S3.F4)\.*Arch\. 0*is the baseline used throughout the article: a contrastive stage built on top of two FFNN projectors is followed by a bijective mimicking stage, that is, two normalizing flows trained with the loss of Eq\. \([9](https://arxiv.org/html/2608.28981#S3.E9)\)\.*Arch\. 1*is a one\-stage variant in which a single pair of normalizing flows directly plays both roles, and the same flows are trained jointly with the InfoNCE objective and a reconstruction term, without the FFNN projectors in between\.*Arch\. 2*is a flow\-only variant in which a single RealNVP is trained with cycle\-consistency between the two modality embeddings, with no intermediate joint space and no contrastive supervision\. It tests how much of the alignment can be recovered by reconstruction alone\. The variance carried by each latent dimension of the joint space is reported in Table[7](https://arxiv.org/html/2608.28981#S5.T7)and the cumulative\-variance profiles are plotted in Fig\.[15](https://arxiv.org/html/2608.28981#S5.F15)\.
Table 7:Ablation study on the joining stage\. Cumulative variance carried by the firstKKprincipal directions of the joint space, dimensionless, for the three architectures of Fig\.[4](https://arxiv.org/html/2608.28981#S3.F4)\.\(a\)Cumulative variance carried by the sorted latent dimensions\.
\(b\)Per\-term loss decomposition during stage 2 training\.
Figure 15:Ablation of the joining stage, for the three architectures sketched in Fig\.[4](https://arxiv.org/html/2608.28981#S3.F4)\. \(a\) Dotted horizontal lines mark the50%50\\%and90%90\\%variance thresholds\. \(b\) Baseline pipeline only, with the total loss split into its three terms\.Arch\. 0 concentrates more variance in the first few principal directions of the joint space than the other two variants\. Its first five directions carry about42%42\\%of the total variance against27%27\\%for the flow\-only Arch\. 2, and its first twenty carry78%78\\%against62%62\\%\. The effective rank of the joint space follows the same ordering, at203203for Arch\. 0,564564for Arch\. 2, and806806for Arch\. 1\. This concentration is desirable, since a compact and coherent cross\-modal alignment is one that uses a small number of axes consistently across both modalities\. Contrastive supervision is what produces it, even when a separate flow is trained on top\. Removing that supervision or folding it into the flow spreads the same information over many more directions\. A second observation from the loss decomposition is that voice reconstruction converges roughly35×35\\timesfaster than trajectory reconstruction\. The asymmetry is structural\. The voice embedding produced by Whisper concentrates most of its variance along a few principal directions of a semantically smooth manifold, while the flattened MAE embedding has high\-dimensional correlations between timesteps that the flow must model simultaneously\.
### 5\.5Latent Space Analysis
To illustrate the structure of the learned space, we project voice and trajectory embeddings to two dimensions with*Uniform Manifold Approximation and Projection*\(UMAP\)\[[27](https://arxiv.org/html/2608.28981#bib.bib36)\]and color the points by control tower and by flight phase\. Figure[16](https://arxiv.org/html/2608.28981#S5.F16)shows that the two modalities form overlapping clusters that organize primarily by tower \(which corresponds to a coarse partition of the airspace\) and secondarily by flight phase \(ground vs\. en\-route\)\. The fact that voice and trajectory projections share the same coarse structure, without ever being supervised on these labels, confirms that the contrastive objective recovers physically meaningful axes of variation\.
Figure 16:UMAP projection of the joint space\. The left panel shows the voice projections and the right panel the trajectory projections\. The labels of the voice points are inferred from the voice signal itself, and the labels of the trajectory points are inferred from the kinematic features\. The two modalities share the same coarse partition of the airspace, and flight\-phase structure is visible inside each tower cluster\.A particularly striking observation is that ground frequencies \(KOAK delivery, KSJC delivery\) form tight, separated clusters from approach frequencies \(NorCal approach, NorCal departure\)\. This is consistent with what controllers report, namely that the phraseology and the typical aircraft state are very different on the two types of frequencies, and the joint space picks up the distinction without ever being told that towers exist as labels\. Within each cluster, flight phase produces a secondary axis \(ground taxi vs\. climb vs\. cruise\) visible in the voice projection too, since the words used by controllers correlate with the kinematic regime of the aircraft they address\. This double structure \(tower \+ phase\) is the empirical signature of a joint cartography of the airspace\. We note that the joint space remains effectively low\-dimensional even though we work inℝ1024\\mathbb\{R\}^\{1024\}, with the variance concentrated on a few principal directions \(Section[5\.4](https://arxiv.org/html/2608.28981#S5.SS4)\)\. This is the regime in which contrastive representation learning is known to be vulnerable to dimensional collapse\[[19](https://arxiv.org/html/2608.28981#bib.bib37)\], which is why the dropout and weight decay regularizers used during training, together with the bijective stage, are partly chosen to mitigate this risk\.
### 5\.6Continuity in the Trajectory Latent Space
Beyond the clustered structure analyzed in Section[5\.5](https://arxiv.org/html/2608.28981#S5.SS5), the trajectory latent space is also locally smooth\. As an aircraft moves through successive trajectory windows, its representation traces a continuous path inside each cluster, and small changes in motion produce small displacements in the latent space\. The trajectory only “jumps” from one cluster to another at flight\-phase transitions, which can be read as a change of regime\. Figure[17](https://arxiv.org/html/2608.28981#S5.F17)illustrates this behavior on a single inbound flight to KSFO\. The aircraft starts in the descent cluster and moves continuously inside it, then jumps to the final approach and landing cluster as the flight phase changes from descent to touchdown, and moves continuously again until the trajectory ends\. This piecewise\-continuous structure mirrors the way a controller reasons about a flight, as a sequence of continuous regimes \(climbing, cruising, descending, taxiing\) separated by discrete maneuver transitions\.
Figure 17:Trajectory of a single aircraft in the latent space of the trajectory tower\. The flight is continuous within each behavior cluster \(descent, landing\) and jumps when the flight phase changes\. Clusters in this projection encode broad maneuver classes such as landing, climbing, or turning\.
## 6Conclusion
Achievements
We presented a situational awareness framework that aligns ATC voice transmissions and aircraft trajectories in a single learned latent space\. V2TATC follows a two\-tower design with two complementary training stages: a contrastive alignment of pre\-trained per\-modality embeddings, followed by a bijective lifting through normalizing flows\. We have shown that this framework can be instantiated over a congested terminal area using public data only, that the resulting joint space supports cross\-modal retrieval well above random, and that it carries enough structure to drive simple downstream tasks such as short\-horizon trajectory forecasting from voice instructions\.
Several practical uses can be envisioned\. The most immediate one is a controller\-facing visualization in which a click on an aircraft retrieves the recent voice exchanges that concern it, and conversely\. A second use is anomaly detection, where voice instructions and observed trajectories that are too far apart in the joint space could signal a non\-compliant maneuver or a developing situational\-awareness mismatch\. A third direction, which we leave to future work, is to leverage the bijective mapping to generate synthetic voice–trajectory pairs for stress\-testing other decision support tools, or to drive a more capable language model conditioned on the surveillance picture\. V2TATC’s framework is generic\. It does not depend on the specific encoders we used, and could be extended to additional modalities such as flight\-plan data\.
The current limitations are also visible\. Path B \(voice\-conditioned trajectory forecast\) is still an order of magnitude less precise than specify Path A, mainly because of the residual distribution shift introduced by the bijective stage\. Three directions could close this gap\. Adversarial alignment between MAE latents and flow\-reconstructed latents would attack the distribution shift directly\. Fine\-tuning the StepPredictor end\-to\-end with the flow would instead let the head adapt to the latents it actually receives\. Collecting data over additional terminal areas would scale the contrastive dataset and reduce the residual overfitting observed here\. We expect that the combination of these three directions can bring Path B within a factor of two of Path A, which would make the framework operationally relevant\.
Future Work
We plan to keep developing the framework along three complementary axes\. First, we will focus on the voice\-to\-trajectory direction\. We want to design more expressive joining architectures that better capture the variability of the airspace\. In particular, we plan to apply classifier\-free guidance and diffusion models, so that voice instructions can act as guidance signals for a generative trajectory model\. Second, we will broaden the set of input modalities that the framework can ingest\. The current paired dataset is already synchronized with the METAR observations collected during the same four\-day window, so weather data can be added without additional alignment work\. Flight\-plan and METAR streams could then be combined with voice and trajectories in a single multimodal joint space\. Third, we will continue scraping new data and improve the post\-processing pipeline\. The goal is to grow the paired dataset to a size where the contrastive data samples the embedding space more uniformly, and where the joint dataset reflects the diversity of operations across multiple terminal areas rather than the San Francisco Bay Area alone\.
## Acknowledgments
We want to thank Trevor Darrell from the Berkeley Artificial Intelligence Research \(BAIR\) Lab for his early conversations on joint embeddings\. We also would like to thank Tom Davis, Jim Murphy and Vishwanath Bulusu from Crown Innovations LLC, John Robinson and Parimal Kopardekar of NASA Ames Research Center, and Dragos Margineantu from Boeing for their insightful discussions throughout the project\. We are grateful to LiveATC\.net for granting access to the ATC audio archives that made this study possible\. We also thank the OpenSky Network and aviationweather\.gov for openly sharing their surveillance and weather data, and more broadly for their sustained data\-collection initiatives, which are essential to research such as ours\.
## References
- \[1\]\(2020\)Wav2vec 2\.0: a framework for self\-supervised learning of speech representations\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2006.11477)Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p2.1),[Figure 4](https://arxiv.org/html/2608.28981#S3.F4),[Figure 4](https://arxiv.org/html/2608.28981#S3.F4.7),[§3\.2](https://arxiv.org/html/2608.28981#S3.SS2.p1.1),[§4\.2](https://arxiv.org/html/2608.28981#S4.SS2.p2.1),[C\. Choice of the Voice Encoder](https://arxiv.org/html/2608.28981#Sx3.SSx3.p1.1)\.
- \[2\]M\. Beck, K\. Pöppel, M\. Spanring, A\. Auer, O\. Prudnikova, M\. Kopp, G\. Klambauer, J\. Brandstetter, and S\. Hochreiter\(2024\)xLSTM: extended long short\-term memory\.arXiv preprint arXiv:2405\.04517\.External Links:[Link](https://arxiv.org/abs/2405.04517)Cited by:[Figure 4](https://arxiv.org/html/2608.28981#S3.F4),[Figure 4](https://arxiv.org/html/2608.28981#S3.F4.7),[§4\.1](https://arxiv.org/html/2608.28981#S4.SS1.p5.1),[B\. Trajectory Encoder Backbones](https://arxiv.org/html/2608.28981#Sx3.SSx2.p1.1)\.
- \[3\]S\. Chen, C\. Wang, Z\. Chen, Y\. Wu, S\. Liu, Z\. Chen, J\. Li, N\. Kanda, T\. Yoshioka, X\. Xiao,et al\.\(2022\)WavLM: large\-scale self\-supervised pre\-training for full stack speech processing\.IEEE Journal of Selected Topics in Signal Processing\.External Links:[Link](https://arxiv.org/abs/2110.13900)Cited by:[§3\.2](https://arxiv.org/html/2608.28981#S3.SS2.p4.1)\.
- \[4\]T\. Chen, S\. Kornblith, M\. Norouzi, and G\. Hinton\(2020\)A simple framework for contrastive learning of visual representations\.InInternational Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2002.05709)Cited by:[§3\.3](https://arxiv.org/html/2608.28981#S3.SS3.p2.1)\.
- \[5\]Y\. Chi, J\. Nie, L\. Zhong, Y\. Wang, and D\. Delahaye\(2023\)A review of situational awareness in air traffic control\.IEEE Access11,pp\. 134040–134057\.External Links:[Document](https://dx.doi.org/10.1109/ACCESS.2023.3336415)Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p1.1)\.
- \[6\]T\. Darrell, M\. Ghazanfari, J\. K\. Kam, A\. Bayen, A\. Tabrizian, and P\. Wei\(2026\)Towards automated air traffic safety assessment around non\-towered airports using large language models\.InAIAA AVIATION 2026 Forum,pp\. 4220\.External Links:[Link](https://arxiv.org/abs/2605.12332)Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p1.1)\.
- \[7\]T\. Darrell and J\. Kam\(2026\)Representing aircraft trajectories in large language models\.In2026 Regional Student Conferences,pp\. 115013\.Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p7.1)\.
- \[8\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InNAACL\-HLT,External Links:[Link](https://arxiv.org/abs/1810.04805)Cited by:[§3\.1](https://arxiv.org/html/2608.28981#S3.SS1.p1.1)\.
- \[9\]L\. Dinh, J\. Sohl\-Dickstein, and S\. Bengio\(2017\)Density estimation using Real NVP\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/1605.08803)Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.28981#S2.SS2.p2.1),[§2\.3](https://arxiv.org/html/2608.28981#S2.SS3.p1.1),[Figure 3](https://arxiv.org/html/2608.28981#S3.F3),[Figure 3](https://arxiv.org/html/2608.28981#S3.F3.4),[§3\.4](https://arxiv.org/html/2608.28981#S3.SS4.p1.1)\.
- \[10\]M\. R\. Endsley\(1995\)Toward a theory of situation awareness in dynamic systems\.Human Factors37\(1\),pp\. 32–64\.External Links:[Document](https://dx.doi.org/10.1518/001872095779049543)Cited by:[§2\.1](https://arxiv.org/html/2608.28981#S2.SS1.p1.1)\.
- \[11\]M\. Ghazanfari, M\. Casanova, J\. Kam, A\. Zongo, P\. Wei, T\. Darrell, and A\. Bayen\(2026\)Air traffic control using large language models: prompt engineering, architecture, and evaluation\.arXiv preprint arXiv:2608\.19299\.External Links:[Link](https://arxiv.org/abs/2608.19299)Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p7.1)\.
- \[12\]D\. Guo, E\. Q\. Wu, Y\. Wu, J\. Zhang, R\. Law, and Y\. Lin\(2023\)FlightBERT: binary encoding representation for flight trajectory prediction\.IEEE Transactions on Intelligent Transportation Systems24\(2\),pp\. 1828–1842\.External Links:[Document](https://dx.doi.org/10.1109/TITS.2022.3219923)Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p1.1),[§1](https://arxiv.org/html/2608.28981#S1.p7.1)\.
- \[13\]D\. Guo, J\. Zhang, and Y\. Lin\(2023\)SIA\-FTP: a spoken instruction aware flight trajectory prediction framework\.arXiv preprint arXiv:2305\.01661\.External Links:[Link](https://arxiv.org/abs/2305.01661)Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p7.1)\.
- \[14\]K\. He, X\. Chen, S\. Xie, Y\. Li, P\. Dollár, and R\. Girshick\(2022\)Masked autoencoders are scalable vision learners\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 16000–16009\.External Links:[Link](https://arxiv.org/abs/2111.06377)Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p2.1),[§3\.1](https://arxiv.org/html/2608.28981#S3.SS1.p1.1),[§4\.1](https://arxiv.org/html/2608.28981#S4.SS1.p4.1)\.
- \[15\]H\. Helmke, O\. Ohneiser, T\. Mühlhausen, and M\. Wies\(2016\)Reducing controller workload with automatic speech recognition\.InIEEE/AIAA 35th Digital Avionics Systems Conference \(DASC\),pp\. 1–10\.External Links:[Document](https://dx.doi.org/10.1109/DASC.2016.7778024)Cited by:[§2\.1](https://arxiv.org/html/2608.28981#S2.SS1.p1.1)\.
- \[16\]H\. Helmke and O\. Ohneiser \(Eds\.\)\(2024\)Automatic speech recognition and understanding in air traffic management\.MDPI,Basel, Switzerland\.External Links:[Document](https://dx.doi.org/10.3390/books978-3-7258-0315-6)Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p1.1)\.
- \[17\]H\. Helmke, K\. Ondřej, S\. Shetty, H\. Arilíusson, T\. S\. Simiganoschi, M\. Kleinert, O\. Ohneiser, H\. Ehr, J\. Zuluaga\-Gomez, and P\. Smrz\(2022\)Readback error detection by automatic speech recognition and understanding: results of HAAWAII project for Isavia’s enroute airspace\.InSESAR Innovation Days,Budapest, Hungary\.Note:URL:[https://www\.sesarju\.eu/sites/default/files/documents/sid/2022/paper\_3\.pdf](https://www.sesarju.eu/sites/default/files/documents/sid/2022/paper_3.pdf)\[retrieved 17 July 2026\]Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p7.1)\.
- \[18\]C\. Jia, Y\. Yang, Y\. Xia, Y\. Chen, Z\. Parekh, H\. Pham, Q\. V\. Le, Y\. Sung, Z\. Li, and T\. Duerig\(2021\)Scaling up visual and vision\-language representation learning with noisy text supervision\.InInternational Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2102.05918)Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.28981#S2.SS2.p1.1)\.
- \[19\]L\. Jing, P\. Vincent, Y\. LeCun, and Y\. Tian\(2021\)Understanding dimensional collapse in contrastive self\-supervised learning\.arXiv preprint arXiv:2110\.09348\.External Links:[Link](https://arxiv.org/abs/2110.09348)Cited by:[§5\.5](https://arxiv.org/html/2608.28981#S5.SS5.p2.1)\.
- \[20\]J\. K\. Kam, M\. Casanova, V\. Bulusu, A\. Bayen, and R\. Sengupta\(2025\)Operational air taxi flight routes in a metropolitan region\.InAIAA AVIATION FORUM AND ASCEND 2025,pp\. 3240\.Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p4.1)\.
- \[21\]S\. M\. Kazemi, R\. Goel, S\. Eghbali, J\. Ramanan, J\. Sahota, S\. Thakur, S\. Wu, C\. Smyth, P\. Poupart, and M\. Brubaker\(2019\)Time2Vec: learning a vector representation of time\.arXiv preprint arXiv:1907\.05321\.External Links:[Link](https://arxiv.org/abs/1907.05321)Cited by:[§3\.1](https://arxiv.org/html/2608.28981#S3.SS1.p3.1)\.
- \[22\]D\. P\. Kingma and P\. Dhariwal\(2018\)Glow: generative flow with invertible 1x1 convolutions\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/1807.03039)Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p2.1),[§3\.4](https://arxiv.org/html/2608.28981#S3.SS4.p1.3),[A\. Hyperparameters](https://arxiv.org/html/2608.28981#Sx3.SSx1.p2.1)\.
- \[23\]G\. Klambauer, T\. Unterthiner, A\. Mayr, and S\. Hochreiter\(2017\)Self\-normalizing neural networks\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/1706.02515)Cited by:[A\. Hyperparameters](https://arxiv.org/html/2608.28981#Sx3.SSx1.p1.1)\.
- \[24\]Y\. LeCun\(2022\)A path towards autonomous machine intelligence\.Note:Version 0\.9\.2, 2022\-06\-27External Links:[Link](https://openreview.net/pdf?id=BZ5a1r-kVsf)Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p7.1)\.
- \[25\]J\. Li, D\. Li, C\. Xiong, and S\. Hoi\(2022\)BLIP: bootstrapping language\-image pre\-training for unified vision\-language understanding and generation\.InInternational Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2201.12086)Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p2.1)\.
- \[26\]\(2026\)LiveATC\.net – live air traffic control audio streams\.External Links:[Link](https://www.liveatc.net/)Cited by:[§4\.2](https://arxiv.org/html/2608.28981#S4.SS2.p1.1)\.
- \[27\]L\. McInnes, J\. Healy, and J\. Melville\(2018\)UMAP: uniform manifold approximation and projection for dimension reduction\.arXiv preprint arXiv:1802\.03426\.External Links:[Link](https://arxiv.org/abs/1802.03426)Cited by:[§5\.5](https://arxiv.org/html/2608.28981#S5.SS5.p1.1)\.
- \[28\]B\. Poole, S\. Ozair, A\. van den Oord, A\. A\. Alemi, and G\. Tucker\(2019\)On variational bounds of mutual information\.InInternational Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/1905.06922)Cited by:[§3\.3](https://arxiv.org/html/2608.28981#S3.SS3.p2.1)\.
- \[29\]A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. Sutskever\(2021\)Learning transferable visual models from natural language supervision\.InInternational Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2103.00020)Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.28981#S2.SS2.p1.1),[§3\.3](https://arxiv.org/html/2608.28981#S3.SS3.p1.2),[§3\.3](https://arxiv.org/html/2608.28981#S3.SS3.p2.1),[§5\.1](https://arxiv.org/html/2608.28981#S5.SS1.p3.1)\.
- \[30\]A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. Sutskever\(2023\)Robust speech recognition via large\-scale weak supervision\.InInternational Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2212.04356)Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p2.1),[Figure 4](https://arxiv.org/html/2608.28981#S3.F4),[Figure 4](https://arxiv.org/html/2608.28981#S3.F4.7),[§3\.2](https://arxiv.org/html/2608.28981#S3.SS2.p1.1),[§3\.2](https://arxiv.org/html/2608.28981#S3.SS2.p2.1),[§4\.2](https://arxiv.org/html/2608.28981#S4.SS2.p2.1),[C\. Choice of the Voice Encoder](https://arxiv.org/html/2608.28981#Sx3.SSx3.p1.1)\.
- \[31\]D\. Rezende and S\. Mohamed\(2015\)Variational inference with normalizing flows\.InInternational Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/1505.05770)Cited by:[§3\.4](https://arxiv.org/html/2608.28981#S3.SS4.p1.1)\.
- \[32\]E\. Rusak, P\. Reizinger, A\. Juhos, O\. Bringmann, R\. S\. Zimmermann, and W\. Brendel\(2024\)InfoNCE: identifying the gap between theory and practice\.arXiv preprint arXiv:2407\.00143\.External Links:[Link](https://arxiv.org/abs/2407.00143)Cited by:[§5\.1](https://arxiv.org/html/2608.28981#S5.SS1.p3.1)\.
- \[33\]D\. Schaefer\(2001\)Context\-sensitive speech recognition in the air traffic control simulation\.Technical reportTechnical ReportEEC Note No\. 02/2001,EUROCONTROL Experimental Centre,Brétigny\-sur\-Orge, France\.Note:Doctoral thesis, Universität der Bundeswehr München\. URL:[https://www\.eurocontrol\.int/sites/default/files/library/004\_Context\-sensitive\_Speech\_Recognition\.pdf](https://www.eurocontrol.int/sites/default/files/library/004_Context-sensitive_Speech_Recognition.pdf)\[retrieved 17 July 2026\]Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p7.1)\.
- \[34\]M\. Schäfer, M\. Strohmeier, V\. Lenders, I\. Martinovic, and M\. Wilhelm\(2014\)Bringing up OpenSky: a large\-scale ADS\-B sensor network for research\.InProceedings of the 13th International Symposium on Information Processing in Sensor Networks,pp\. 83–94\.External Links:[Link](https://opensky-network.org/)Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p1.1),[§4\.1](https://arxiv.org/html/2608.28981#S4.SS1.p1.1)\.
- \[35\]A\. van den Oord, Y\. Li, and O\. Vinyals\(2018\)Representation learning with contrastive predictive coding\.arXiv preprint arXiv:1807\.03748\.External Links:[Link](https://arxiv.org/abs/1807.03748)Cited by:[§3\.3](https://arxiv.org/html/2608.28981#S3.SS3.p1.2)\.
- \[36\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/1706.03762)Cited by:[Figure 4](https://arxiv.org/html/2608.28981#S3.F4),[Figure 4](https://arxiv.org/html/2608.28981#S3.F4.7),[§4\.1](https://arxiv.org/html/2608.28981#S4.SS1.p5.1),[B\. Trajectory Encoder Backbones](https://arxiv.org/html/2608.28981#Sx3.SSx2.p1.1)\.
- \[37\]Z\. Zhao, W\. Zeng, Z\. Quan, M\. Chen, and Z\. Yang\(2019\)Aircraft trajectory prediction using deep long short\-term memory networks\.In19th COTA International Conference of Transportation Professionals \(CICTP\),pp\. 124–135\.External Links:[Document](https://dx.doi.org/10.1061/9780784482292.012)Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p1.1),[§1](https://arxiv.org/html/2608.28981#S1.p7.1)\.
- \[38\]J\. Zuluaga\-Gomez, S\. S\. Sarfjoo, A\. Prasad, I\. Nigmatulina, P\. Motlicek, K\. Ondrej, O\. Ohneiser, and H\. Helmke\(2022\)BERTraffic: BERT\-based joint speaker role and speaker change detection for air traffic control communications\.InIEEE Spoken Language Technology Workshop \(SLT\),pp\. 633–640\.External Links:[Link](https://arxiv.org/abs/2110.05781)Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p1.1),[§1](https://arxiv.org/html/2608.28981#S1.p7.1)\.
- \[39\]J\. Zuluaga\-Gomez, K\. Veselý, I\. Szöke, A\. Blatt, P\. Motlicek, M\. Kocour, M\. Rigault, K\. Choukri, A\. Prasad, S\. S\. Sarfjoo, I\. Nigmatulina, C\. Cevenini, P\. Kolčárek, A\. Tart, J\. Černocký, and D\. Klakow\(2022\)ATCO2 corpus: a large\-scale dataset for research on automatic speech recognition and natural language understanding of air traffic control communications\.arXiv preprint arXiv:2211\.04054\.External Links:[Link](https://arxiv.org/abs/2211.04054)Cited by:[§1](https://arxiv.org/html/2608.28981#S1.p7.1),[C\. Choice of the Voice Encoder](https://arxiv.org/html/2608.28981#Sx3.SSx3.p1.1)\.
## Appendix
The appendix collects supporting material that complements the main article and is useful for reproduction\.
### A\. Hyperparameters
The contrastive projectors are single\-hidden\-layer FFNNs of width40964096with SELU activation\[[23](https://arxiv.org/html/2608.28981#bib.bib38)\]and a dropout of0\.20\.2\. The Scaled Exponential Linear Unit was selected over more common activations such as ReLU because it is smooth, differentiable everywhere, and, when combined with the associated LeCun\-normal initialization, preserves the mean and variance of its input across layers, so activations remain approximately zero\-mean and unit\-variance throughout the projector\. This self\-normalization property reduces the need for explicit batch normalization inside the projector and stabilizes contrastive training when the batch size is limited\. The InfoNCE temperature is initialized atτ=0\.07\\tau=0\.07, parameterized in log\-space, and clamped to\[0\.04,100\]\[0\.04,100\]\. Training uses AdamW \(β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999\), gradient clipping at norm1\.01\.0, batch size 512, learning rate10−410^\{\-4\}withReduceLROnPlateauscheduling, and early stopping on the validation loss with a patience of fifteen epochs\.
The two normalizing flows are stacks of eight affine coupling layers with conditioner MLPs of hidden width\[256,256\]\[256,256\]\. ActNorm layers\[[22](https://arxiv.org/html/2608.28981#bib.bib7)\]are inserted between coupling blocks and initialized data\-dependently\. Smaller\-dimension embeddings are zero\-padded to the common working widthDmax=1792D\_\{\\text\{max\}\}=1792before entering the flow\. The bijective stage uses the same optimizer with batch size 256 and a maximum of 200 epochs\. The MAE encoder is a four\-layer Transformer with hidden widthd=128d=128, four attention heads, and feed\-forward dimension512512\. Training uses AdamW, peak learning rate10−310^\{\-3\}, linear warm\-up over five epochs followed by cosine decay, weight decay10−210^\{\-2\}\(excluding bias and normalization layers\), gradient clipping at1\.01\.0, batch size 1024, and mixed\-precision training\. The masking ratio ism=0\.25m=0\.25for Stage 1 and configurable for Stage 2\.
### B\. Trajectory Encoder Backbones
We compared four backbone architectures inside the MAE wrapper described in Section[3](https://arxiv.org/html/2608.28981#S3): a Transformer\[[36](https://arxiv.org/html/2608.28981#bib.bib33)\], a bidirectional LSTM, a scalar xLSTM \(sLSTM cells only\), and a full xLSTM with matrix memory\[[2](https://arxiv.org/html/2608.28981#bib.bib32)\]\. Validation losses on the OpenSky San Francisco dataset are reported in Table[8](https://arxiv.org/html/2608.28981#Sx3.T8)\. All four backbones converge to validation MSE values within4×10−44\\times 10^\{\-4\}of one another, which indicates that the MAE objective on 14\-timestep ADS\-B windows is largely architecture\-agnostic at this scale\.
Table 8:Validation MSE \(dimensionless, normalized\-delta space\) for four trajectory backbones trained as MAE on Bay Area ADS\-B data\. All values are within4×10−44\\times 10^\{\-4\}of one another; the Transformer is retained as the default encoder\.The UMAP projections of the encoder hidden states \(Fig\.[18](https://arxiv.org/html/2608.28981#Sx3.F18)\) confirm this quantitative similarity\. Beyond the near\-identical validation MSE reported in Table[8](https://arxiv.org/html/2608.28981#Sx3.T8), the four latent spaces exhibit the same qualitative structure: similar pairs of neighboring colors, a large central blob, and the same filament\-like extensions toward the periphery\. This is a strong signal on its own, as it indicates that the geometry of the latent space captures an intrinsic property of the airspace itself and does not depend on the specific inductive bias of the sequence model\. Given this near equivalence, we retain the Transformer because \(i\) it has the lowest validation MSE of the four candidates and \(ii\) its main blob is qualitatively the most spread out and the most cleanly organized by flight phase\. We note that BiLSTM is about four times cheaper to train per epoch \(Table[8](https://arxiv.org/html/2608.28981#Sx3.T8)\), which would be a relevant trade\-off in a larger\-scale experiment\. The standalone UMAP of the Transformer latent space is shown in the body of the article as Fig\.[5](https://arxiv.org/html/2608.28981#S4.F5)\.
The projections carry a few observations that are physically interpretable and that are consistent across all four backbones\. The large central blob is populated almost exclusively by general aviation trajectories\. It is continuous, connects all flight phases \(taxi, climb, cruise, descent, landing\) into a single manifold, and reflects the fact that general aviation traffic follows fewer standardized procedures than commercial traffic\. Commercial\-aviation trajectories, by contrast, form tighter and more clearly separated clusters, because commercial flights follow standard instrument procedures and standard arrival and departure routes that repeat from one flight to the next\. Local continuity between neighboring clusters is also visible and follows the temporal ordering of a flight: landing clusters sit next to descent clusters, and take\-off clusters sit next to climb clusters, so a real flight traces a nearly continuous path across the latent space as it transitions from one phase to the next\. Several disjoint clusters can be observed for a single flight phase; these correspond to the geographic separation between the main Bay Area airports \(KSFO, KOAK, KSJC, KNUQ\), each of which produces its own family of approach and departure trajectories despite sharing the same phase label\.
Figure 18:UMAP projections of the MAE trajectory latent space for the four candidate backbones \(Transformer, BiLSTM, sLSTM, xLSTM\)\. Colors indicate flight phase labels inferred directly from the raw kinematic features\.
### C\. Choice of the Voice Encoder
We considered two foundation ASR models as voice encoders: Whisper large\-v3\[[30](https://arxiv.org/html/2608.28981#bib.bib11)\]and Wav2Vec 2\.0 XLS\-R fine\-tuned on an ATC corpus\[[1](https://arxiv.org/html/2608.28981#bib.bib12),[39](https://arxiv.org/html/2608.28981#bib.bib4)\]\. The two models produce qualitatively different latent spaces\. Figure[19](https://arxiv.org/html/2608.28981#Sx3.F19)shows the UMAP projections of the voice embeddings, colored by control tower\. Three visual properties of these projections drive our choice in favor of Whisper\.
First, the Whisper latent space is markedly more*spread out*\. It occupies a wider portion of the UMAP plane and takes an elongated, filament\-like shape that spans several visibly connected regions\. The Wav2Vec 2\.0 latent space, by contrast, collapses into a more compact blob with a few isolated satellites\. A wider effective support is desirable for the downstream contrastive stage because the projectorPvP\_\{v\}has more room to separate distinct utterances without having to fight against a pre\-existing collapse\.
Second, the Whisper latent space is more*locally smooth*, so that neighboring points in UMAP correspond to acoustically neighboring utterances, and clusters connect through continuous transition regions rather than through empty gaps\. Smoothness matters directly for contrastive training, because a small perturbation of the input produces a small perturbation of the embedding, which yields well\-behaved gradients throughPvP\_\{v\}and stable in\-batch negatives\. The Wav2Vec 2\.0 projection, although it groups points by tower more cleanly on a coarse scale, is granular inside each cluster, where tight clusters are made of many small islands rather than a smooth density, so nearby points can correspond to unrelated utterances and the effective Lipschitz constant of the encoder is worse\.
Third, the tighter clustering of Wav2Vec 2\.0 is a direct consequence of the ATC fine\-tuning\. The model has specialized so strongly on the ATC domain that it has partially lost the broad acoustic prior it was pre\-trained with, a phenomenon known as catastrophic forgetting\. Because our joint dataset is small compared with the corpora on which foundation ASR models are trained, this loss is not compensated by our own data\. Whisper, kept frozen, retains the full acoustic prior of its680 000h680\\,000\\text\{\\,\}\\mathrm\{h\}of multi\-domain training, which turns out to be more valuable than the domain\-specific but narrower Wav2Vec 2\.0 representation\.
Taken together, the three properties suggest that the Whisper representation is the better substrate for a joint embedding: a wider support, smoother local geometry, and a stronger acoustic prior\. This visual analysis is consistent with the two other criteria discussed below \(transcription quality when decoding back from the embedding, and validation loss under the encoder–projector sweep of Table[9](https://arxiv.org/html/2608.28981#Sx3.T9)\), which independently favor Whisper\.
\(a\)Whisper large\-v3 embeddings\.
\(b\)Wav2Vec 2\.0 XLS\-R embeddings \(ATC fine\-tuned\)\.
Figure 19:UMAP projections of the voice embeddings, colored by control tower\. Whisper produces a smoother and more spread\-out latent space, while Wav2Vec 2\.0 produces tighter but less smooth clusters\.A second decisive observation concerns the quality of transcriptions decoded back from the embedding\. When the Whisper embedding is passed through a Whisper decoder, the recovered text is fluent and grammatically correct\. When the Wav2Vec 2\.0 embedding is decoded back to text, the output is not even a grammatically valid English sentence\. Since our downstream goal includes recovering plausible ATC phraseology from a point in the joint space, this observation alone is enough to retain Whisper\.
The two observations together support a general conclusion\. In principle, a model fine\-tuned on the target domain should outperform a generic foundation model\. In practice, ATC fine\-tuning is performed on relatively small corpora, and catastrophic forgetting of the broad acoustic knowledge embedded in the original model is hard to avoid\. Foundation models like Whisper, trained on huge and acoustically diverse corpora \(including noisy radio\-style speech and many accents\), turn out to be more robust on our data\. To quantify this choice and to pick the projector width, we ran a small grid over voice encoders and projector dimensions\. Results are summarized in Table[9](https://arxiv.org/html/2608.28981#Sx3.T9)and Fig\.[20](https://arxiv.org/html/2608.28981#Sx3.F20)\. Whisper outperforms Wav2Vec 2\.0 at every projector size, and the validation loss is essentially flat once the projector output dimension reaches10241024\. We therefore use a single\-hidden\-layer projector withdj=1024d\_\{j\}=1024as the working configuration, which keeps the projector at the smallest width that saturates the validation loss, as discussed in Section[4](https://arxiv.org/html/2608.28981#S4)\.
Table 9:Validation metrics for the voice\-encoder/projector\-size sweep\. Whisper consistently outperforms Wav2Vec 2\.0 across all projector sizes, and the largest projector\(4096→2048\)\(4096\\to 2048\)gives the best retrieval\.Figure 20:Validation InfoNCE loss during contrastive training for the eight configurations of the encoder–projector sweep \(two voice encoders×\\timesfour projector sizes\)\. Whisper large\-v3 \(solid lines, blue shades\) consistently achieves a lower validation loss than Wav2Vec 2\.0 \(dashed lines, red shades\) across all projector configurations\. Performance saturates once the projector output dimension reachesdj=1024d\_\{j\}=1024, which motivates our working configuration\. Darker shades correspond to larger projector hidden dimensions\.
### D\. Affine Coupling Layer in RealNVP
For completeness, Fig\.[21](https://arxiv.org/html/2608.28981#Sx3.F21)gives a self\-contained schematic of the forward and inverse passes of an affine coupling layer in a RealNVP flow\. The same idea is repeatedKKtimes with alternating masks to build the two normalizing flowsfvf\_\{v\}andftf\_\{t\}used in the bijective stage of the framework\.
Figure 21:Schematic of a RealNVP affine coupling layer\. The input is split in two halves\. One half is left unchanged and used as the conditioning input of two neural networks that produce the scalessand translationttof an affine transformation applied to the other half\. The inverse can be computed analytically in closed form\. Next, we combine these affine coupling layers and swap the mask by half so that the flow spreads across all dimensions\.
### E\. Code Repositories and Dataset
The dataset is released as sharded HDF5 files organized per control tower\. Each shard exposes four parallel datasets: variable\-lengthint16waveformsat16kHz16\\text\{\\,\}\\mathrm\{kHz\};float32trajectoriesof shape\(N,T,7\)\(N,T,7\)where the first six columns are longitude, latitude, geometric altitude, ground speed, true track, and vertical rate, and the seventh carries the absolute timestamp; per\-samplemetadata\_jsonstrings containing callsign, phrase timestamps, ICAO 24 code, match type, and source MP3 filename; and globalsample\_indices\. The code for collection, preprocessing, analysis, and training is split across five public repositories:
- •
- •
- •
- •
- •
A subset of the dataset is also released on HuggingFace for training and experimentation at the following link:[Lbrusset/SF\_bay\_voice2traj\_dataset](https://huggingface.co/datasets/Lbrusset/SF_bay_voice2traj_dataset)\. The subset contains approximately 83,223 paired audio–trajectory samples together with the metadata describing how each pair was linked \(callsign match type, timestamps, ICAO 24, source frequency, and source MP3 file\)\.Similar Articles
Long-Horizon Agent Trajectory Attribution: A Unified Benchmark and Fine-Grained Annotation Framework
This paper introduces a unified benchmark and fine-grained annotation framework for long-horizon agent trajectory attribution, enabling evaluation of primary attribution localization and attribution-chain recovery across diverse settings. It provides over 1,300 annotated trajectories from existing agent benchmarks and releases a reusable annotation skill for standardizing future trajectories.
Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoders
Presents a multimodal voice activity projection framework extending audio-only VAP to audio-visual inputs for turn-taking prediction in social robots, using pretrained backbones and low-rank adaptation. Achieves improvements on NoXi and Haru EDR corpora.
OpenVisTool: An Open Recipe for Synthesizing Instructive Visual Tool-Use Trajectories
This paper presents OpenVisTool, an open framework for synthesizing instructive visual tool-use trajectories, along with a dataset (OpenVisTool-42K) and benchmark. It shows that fine-tuning on causally grounded supervision improves visual tool-use performance across multiple model backbones.
Trajectory-aware Cross-view Geo-localization with Sequential Observations
Introduces SeqGeo-VL dataset and TrajLoc framework for trajectory-aware cross-view geo-localization using sequential observations (video clips or route descriptions), achieving substantial gains over state-of-the-art methods.
ENTRAP-VL: A Taxonomic Probe for Dual Contextual Entrainment in Vision-Language Models
Introduces ENTRAP-VL, a taxonomically structured dataset of 1,500 items to probe contextual entrainment in vision-language models, examining how textual and visual context independently influence model outputs.