Seven languages, 4,421 utterances: is emotion represented the same way in all of them? Mostly yes. The geometry lines up across languages sharing no writing system and little history. But it lines up with an accent, and the accent is measurable. https://oruk.ai/research/universal-with-an-accent…

Universal, with an accent | oruk

Source: https://oruk.ai/research/universal-with-an-accent In 2001, Klaus Scherer played German actors to listeners in nine countries and found that vocal emotion crosses every border, and pays a toll at each one. We ran that experiment again, inside speech models.

Below are 4,421 emotional recordings in seven languages — English, German, French, Persian, Urdu, Bangla, and Mandarin — inside a frozen speech encoder. Each panel is one language; every point is one utterance, colored by the emotion the speaker performed. The models were never trained on emotion labels, and nothing links the panels except the sound. The same four-colored arrangement appears in every one.

4,421 utterances in seven languages, one panel per language, one shared camera. Points are colored by acted emotion; each language is centered on its own mean. Drag to orbit, scrub the slider through the model’s layers, and use the chips to isolate one emotion. the setup

Seven corpora, four feelings, no labels

The recordings come from seven acted and semi-natural emotion corpora: CREMA-D (English), EmoDB (German), ESD (Mandarin), ShEMO (Persian), the URDU corpus, SUBESCO (Bangla), and CaFE (Québécois French). We kept the four emotions every corpus shares —anger,happiness,sadness,neutral— and ran every clip through the same four frozen encoders asThe shape of a voice: Whisper, WavLM, HuBERT, and a Conformer. One mean-pooled vector per utterance per layer. No fine-tuning, no emotion supervision.

The question is the one the cross-cultural literature has argued about for fifty years, transplanted into embedding space. Human listeners recognize vocal emotion above chance in essentially every culture tested, but do about five points better on voices from their own. Elfenbein calls this dialect theory: a universal language spoken everywhere with a local accent. If that structure is real and acoustic, it should be sitting inside these models.

the accent, measured

Train on one language, test on another

A linear probe trained and tested within one language names the acted emotion of a held-out speaker 81 percent of the time on average; chance is 25. Carry the same probe across a language border and it lands at 53. That is above chance in all 168 train–test pairs we measured (42 borders, four models) and well below home performance in almost every one of them. The human result, re-measured in a machine: recognizable everywhere, best at home.

The border is selective, too. Averaged over models, a probe crossing into a new language still catchessadness73 percent of the time andanger56, buthappinessonly 51. That is the ordering the human cross-cultural studies keep finding: sadness and anger travel; joy gets stopped at customs.neutralis the strange one. It crosses at just 31 percent, because what an unmarked voice sounds like turns out to be the most local convention in the set.

Emotion accuracy for probes trained on the row language and tested on the column language, at each model’s best layer. The toggle subtracts each language’s mean embedding before training and testing — one vector per language — and recovers between a fifth and a third of the transfer gap, depending on the model. The toggle is the interesting part.“Accent removal”here is one operation: compute each language’s average embedding and subtract it. No retraining, no labels from the target language. A deployed system could do it with a few minutes of unlabeled speech. Cross-language accuracy rises from 53 to 60 percent on average, and in Whisper the subtraction buys back a full third of the gap to home performance. So much of the accent turns out to be an offset. It moves where a language sits in the space, and the arrangement of emotions around that spot survives the trip.

the sheaf and the splay

One axis travels. The other stays home.

Psychologists compress emotion into two dimensions: arousal (calm to fired-up) and valence (negative to positive). Decades of acoustics say arousal is nearly biological — faster, louder, higher-pitched everywhere — while valence barely shows up in the raw signal. Here is that half-century of literature as one picture: for each language, the direction in embedding space that separates high arousal from low, and positive from negative.

Each arrow is one language’s emotion axis, projected into the shared 3-D basis. Solid arrows separate high arousal (anger, happiness) from low (sadness, neutral); dashed arrows separate positive (happiness) from negative (anger, sadness). In the audio-only models the arousal arrows bundle and the valence arrows scatter; in Whisper, both bundle. In the three encoders trained on audio alone — WavLM, HuBERT, the Conformer — the prediction lands. The average agreement between two languages’arousal directions runs from 0.33 to 0.52 (cosine similarity, where 1 is identical and 0 is unrelated); valence manages only 0.22 to 0.32. Probes agree. Separating anger from sadness, an arousal split at fixed negative valence, crosses borders at 86 percent on average; separating happiness from anger, a valence split between two loud emotions, lands at 78.

We expected that split everywhere. Whisper broke it: its valence directions align across languages almost exactly as well as its arousal directions (0.44 versus 0.46). Whisper is also the one encoder trained to transcribe, with an objective that ties sound to words in 96 languages. That is Wagner and colleagues’2023 finding run in reverse. They showed transformer valence gains come from implicit linguistics; here, the model with the linguistics is exactly the one whose valence crosses borders. The dimension people most want from an emotion system is the one the voice alone gives up last.

where universality lives

A middle-layer passport

Emotion that transfers across languages peaks in the middle of the network: layers 4 and 5 of 12 in the base models, layer 9 of 24 in the Conformer. Past the peak it decays, because the top of every encoder is busy converging on what was said. It is the same mid-stack region where, in our last post, phonetic categories crystallized. The early layers mix speaker and channel into raw acoustics, and the part of prosody that crosses borders lives in between. Meanwhile a language-ID probe never drops below 97 percent at any depth of any model. The network always knows which language it is hearing; universality and the accent coexist at every layer, just along different directions in the space.

The braid: every language’s four emotion centroids, traced layer by layer in one fixed 3-D frame. 28 strands enter scattered, and by the deep end each emotion’s seven strands ride the same track. Scrub depth to grow the braid; drag to orbit it. Probe accuracy by layer. Same-language accuracy and cross-language transfer both peak mid-stack; the accent-removal gain is largest there too. The thin gray curve is a language-identity probe — the accent never leaves. why we care

Reading the gradient

The human literature says vocal emotion is universal with an accent: above chance nearly everywhere, better at home, sadness and anger traveling furthest, joy and valence mostly staying put. Every one of those findings reappeared inside frozen speech encoders that never saw an emotion label. The gradient is in the sound, and the models absorbed it from audio alone. Nobody taught them dialect theory. Even the one deviation, Whisper’s traveling valence, points straight back at its own cause: text supervision.

For a company that scores how speech sounds, the readings are practical. Cross-language deployment without adaptation costs real accuracy, and the cost is measurable per language pair, not hand-waved. A surprising share of that cost is a per-language offset that unlabeled audio can remove. And any valence number that is not paired with a transcript-only control deserves suspicion, because in every language we tested, the voice alone is stingiest about exactly that dimension.

Methods, briefly

Data: 4,421 clips (4.1 hours) drawn from CREMA-D, EmoDB, ESD (Mandarin half), ShEMO, URDU, SUBESCO, and CaFE, resampled to 16 kHz mono, capped at 200 clips per language–emotion cell and stratified across speakers (259 speakers total). Encoders: Whisper-small (encoder half), WavLM-base+, HuBERT-base, and wav2vec2-Conformer-large, all frozen; one mean-pooled vector per utterance per layer over the clip’s valid frames.

Probes are L2-regularized logistic regressions on standardized features. Within-language accuracy averages five random held-out-speaker splits; cross-language accuracy trains on all of the row language and tests on all of the column language. Accent removal subtracts each language’s mean embedding, computed from its clips alone, before training and testing. Arousal and valence directions are centroid differences on standardized, language-centered features at each model’s best emotion layer. The 3-D clouds are PCA of the language-centered space, Procrustes-aligned across 13 depth stops; interactive payloads are int16-quantized coordinates streamed to a Canvas-2D renderer.

Sources and reproducibility

• Scherer, Banse & Wallbott (2001),“Emotion inferences from vocal expression correlate across languages and cultures,”J. Cross-Cultural Psychology— the nine-country experiment this post re-runs.
• Laukka & Elfenbein (2021),“Cross-cultural emotion recognition and in-group advantage in vocal expression,”Emotion Review— the meta-analytic case for“universal with an accent.”
• Wagner et al. (2023),“Dawn of the transformer era in speech emotion recognition,”IEEE TPAMI— why valence gains in modern models come from implicit linguistics.
• Corpora: CREMA-D (Cao et al., 2014), EmoDB (Burkhardt et al., 2005), ESD (Zhou et al., 2022), ShEMO (Mohamad Nezami et al., 2019), URDU (Latif et al., 2018), SUBESCO (Sultana et al., 2021), CaFE (Gournay et al., 2018). All publicly available; only the four shared emotion categories were used.
• Pipeline: manifest→frozen-encoder extraction→probes and axes→PCA + Procrustes→int16 web payloads, inscripts/universality/in the site repo. Seeds fixed; every number on this page regenerates from the scripts.