Ruby-ASR: Evidence-Preserving Supervision for Joint Orthographic and Lexical-Reading Recognition

# Ruby-ASR: Evidence-Preserving Supervision for Joint Orthographic and Lexical-Reading Recognition
Source: [https://arxiv.org/html/2609.27289](https://arxiv.org/html/2609.27289)
Yun LiuXuehao YangJun LiuChuanbo HuaXuanjun ChenLianbo LiuShiao ZhuZixiong Su

###### Abstract

Conventional Japanese automatic speech recognition \(ASR\) is supervised by an orthographic transcript, although the same written form can correspond to different lexical readings realized in speech\. Such utterances receive an identical target, so their reading distinction is absent from the supervision interface and cannot be recovered reliably by post\-hoc text\-only grapheme\-to\-phoneme conversion\. We presentRuby\-ASR, which refines the conventional target into a span\-bound orthographic–lexical\-reading sequence, e\.g\.,

今日\[きょう\]\. Unlike separate full\-sentence orthographic and phonological outputs, the ruby representation locally binds each written span to its realized reading and permits deterministic recovery of both views\. We instantiate the target under subtitle\-style and verbatim\-style transcription conventions using a Qwen3\-ASR backbone; a mora\-level CTC objective provides auxiliary monotonic reading supervision\. The experimental results across five Japanese benchmarks show that refining the recognition target can improve lexical\-reading recovery without sacrificing readable orthographic transcription\. We release the checkpoints and inference code\.

###### Index Terms:

Japanese ASR, lexical\-reading recognition, ruby annotation, orthography–phonology ambiguity

††address:Independent Researcher
hshi@ieee\.org,liuyun5620472@gmail\.com,sunskyxh@gmail\.comliujun@alu\.uestc\.edu\.cn,cbhua@kaist\.ac\.kr,zxsu@g\.ecc\.u\-tokyo\.ac\.jp## 1Introduction

Automatic speech recognition is conventionally formulated as mapping speech to an orthographic transcription\. For Japanese, the output is a readable mixture of kanji and kana and is commonly evaluated using character error rate \(CER\)\[[5](https://arxiv.org/html/2609.27289#bib.bib15),[6](https://arxiv.org/html/2609.27289#bib.bib16)\]\. However, it does not always identify how a lexical item was realized in speech\. Japanese orthography and lexical reading are not in one\-to\-one correspondence: different written forms can share a reading, while one written form can admit multiple readings\[[27](https://arxiv.org/html/2609.27289#bib.bib17),[29](https://arxiv.org/html/2609.27289#bib.bib18),[19](https://arxiv.org/html/2609.27289#bib.bib21),[10](https://arxiv.org/html/2609.27289#bib.bib3)\]\. If two utterances have different readings but the same orthographic transcript, orthographic ASR assigns them an identical target and does not require this distinction at the output\. We call this*target\-level lexical\-reading collapse*\. A downstream grapheme\-to\-phoneme \(G2P\) model cannot resolve the problem because it observes only the recognized text and infers a likely reading from linguistic context\[[12](https://arxiv.org/html/2609.27289#bib.bib12),[17](https://arxiv.org/html/2609.27289#bib.bib22),[7](https://arxiv.org/html/2609.27289#bib.bib23)\]\. Likewise, text\-only kanji\-reading benchmarks\[[10](https://arxiv.org/html/2609.27289#bib.bib3),[13](https://arxiv.org/html/2609.27289#bib.bib24)\]cannot test whether a reading follows the input speech\. Conversely, kana\- or phoneme\-only ASR preserves pronunciation but loses the orthographic distinctions and readability provided by kanji\.

Joint phoneme–grapheme ASR has addressed this tradeoff using shared encoders and separate utterance\-level decoders\[[9](https://arxiv.org/html/2609.27289#bib.bib25),[24](https://arxiv.org/html/2609.27289#bib.bib30),[14](https://arxiv.org/html/2609.27289#bib.bib26),[8](https://arxiv.org/html/2609.27289#bib.bib27)\]\. More closely, Omachi et al\. serialize each grapheme or multi\-grapheme span with its aligned phonemic annotation in a single sequence\[[16](https://arxiv.org/html/2609.27289#bib.bib28)\]\. Transcript\-prompted Whisper predicts Japanese phonemic and prosodic annotations from audio conditioned on a ground\-truth transcript\[[4](https://arxiv.org/html/2609.27289#bib.bib29)\], but is an annotation model rather than speech\-only ASR\. The remaining challenge is to scale speech\-conditioned lexical\-reading supervision while retaining a directly usable orthographic transcript\.

We therefore proposeRuby\-ASR, which replaces the conventional target with a structured ruby\-form sequence such as “

今日\[きょう\]\.” This refines the output equivalence classes induced by orthographic supervision: utterances with the same surface form but different realized readings receive different labels\. Its local span–reading binding permits deterministic recovery of both conventional orthographic and lexical\-reading transcripts\. We train two variants with the same architecture and objective:*Ruby\-ASR\-sub*follows a concise subtitle\-style convention, whereas*Ruby\-ASR\-ver*follows a more literal verbatim\-style convention\. Ruby generation is the primary inference path, while a shared mora\-level CTC branch provides auxiliary monotonic supervision and can optionally produce a decoder\-free reading hypothesis\.

Our contributions are threefold\. First, we formulate lexical\-reading loss as a target\-level collapse under orthographic supervision and introduce an oracle\-orthography G2P diagnostic that isolates the empirical text\-only reconstruction gap from orthographic ASR errors\. Second, we scale span\-bound ruby supervision to a large LLM\-based Japanese ASR system using an evidence\-constrained target\-construction pipeline\. Third, evaluation on five Japanese benchmarks shows improved direct lexical\-reading recognition with competitive orthographic accuracy; the checkpoints and inference code are publicly released111[https://github\.com/hshi\-speech/Ruby\-ASR\-1\.7B](https://github.com/hshi-speech/Ruby-ASR-1.7B)222[https://huggingface\.co/hshispeech/Ruby\-ASR\-1\.7B](https://huggingface.co/hshispeech/Ruby-ASR-1.7B)\.

## 2Beyond Orthographic Supervision

### 2\.1Target\-level lexical\-reading collapse

Let𝐗\\mathbf\{X\}denote an utterance,𝐎\\mathbf\{O\}its orthographic representation, and𝐑\\mathbf\{R\}its lexical reading realized in speech\. Conventional Japanese ASR learns

p⁡\(𝐎∣𝐗\)\.p\(\\mathbf\{O\}\\mid\\mathbf\{X\}\)\.\(1\)For lexical reading, orthography is not always sufficient:

H⁡\(𝐑∣𝐎\)\>0,I⁡\(𝐑;𝐗∣𝐎\)\>0\.H\(\\mathbf\{R\}\\mid\\mathbf\{O\}\)\>0,\\hskip 20\.00003ptI\(\\mathbf\{R\};\\mathbf\{X\}\\mid\\mathbf\{O\}\)\>0\.\(2\)For𝐎a=𝐎b\\mathbf\{O\}\_\{a\}=\\mathbf\{O\}\_\{b\}but𝐑a≠𝐑b\\mathbf\{R\}\_\{a\}\\neq\\mathbf\{R\}\_\{b\}, orthographic supervision maps𝐗a\\mathbf\{X\}\_\{a\}and𝐗b\\mathbf\{X\}\_\{b\}to the same target, so the reading distinction is absent from the learning interface\. Equivalently, it induces

𝐗a∼O𝐗b⇔𝐎a=𝐎b,\\mathbf\{X\}\_\{a\}\\sim\_\{\\mathrm\{O\}\}\\mathbf\{X\}\_\{b\}\\iff\\mathbf\{O\}\_\{a\}=\\mathbf\{O\}\_\{b\},\(3\)whereas Ruby supervision refines the relation to

𝐗a∼ruby𝐗b⇔\(𝐎a,𝐑a\)=\(𝐎b,𝐑b\),\\mathbf\{X\}\_\{a\}\\sim\_\{\\mathrm\{ruby\}\}\\mathbf\{X\}\_\{b\}\\iff\(\\mathbf\{O\}\_\{a\},\\mathbf\{R\}\_\{a\}\)=\(\\mathbf\{O\}\_\{b\},\\mathbf\{R\}\_\{b\}\),\(4\)thereby separating examples collapsed by orthographic supervision\.

### 2\.2Supervision interfaces and prior joint targets

Table[1](https://arxiv.org/html/2609.27289#S2.T1)compares the relevant supervision interfaces\. Orthographic ASR preserves readability but omits the realized reading; ASR followed by G2P infers it only after speech has been reduced to text\. Kana\-only ASR directly recognizes reading but loses orthographic identity\. Prior multi\-target systems retain both views using separate utterance\-level sequence heads\[[9](https://arxiv.org/html/2609.27289#bib.bib25),[14](https://arxiv.org/html/2609.27289#bib.bib26),[8](https://arxiv.org/html/2609.27289#bib.bib27)\], but do not explicitly bind their corresponding spans\. Omachi et al\. instead combine locally aligned grapheme, phoneme, and optional POS subsequences in one autoregressive target\[[16](https://arxiv.org/html/2609.27289#bib.bib28)\]\. Ruby\-ASR scales this closely related interface to LLM\-based Japanese ASR as

今日\[きょう\]学校\[がっこう\]に行\[い\]く, with deterministic orthographic and reading projections\. In contrast, transcript\-prompted annotation requires ground\-truth orthography at inference time\[[4](https://arxiv.org/html/2609.27289#bib.bib29)\]\.

### 2\.3Text\-only reconstruction gap

For recognized orthography, post\-hoc G2P predicts

𝐑^ASR\+G2P=G2P⁡\(𝐎^\)\.\\widehat\{\\mathbf\{R\}\}\_\{\\mathrm\{ASR\+G2P\}\}=\\mathrm\{G2P\}\(\\widehat\{\\mathbf\{O\}\}\)\.\(5\)This conflates orthographic recognition errors with text\-only reading reconstruction\. We therefore also evaluate

𝐑^oracle​\-​G2P=G2P⁡\(𝐎∗\),\\widehat\{\\mathbf\{R\}\}\_\{\\mathrm\{oracle\\text\{\-\}G2P\}\}=\\mathrm\{G2P\}\(\\mathbf\{O\}^\{\*\}\),\(6\)where𝐎∗\\mathbf\{O\}^\{\*\}is ground\-truth orthography, and define

Etext=CER⁡\(G2P⁡\(𝐎∗\),𝐑∗\)E\_\{\\mathrm\{text\}\}=\\mathrm\{CER\}\\\!\\left\(\\mathrm\{G2P\}\(\\mathbf\{O\}^\{\*\}\),\\mathbf\{R\}^\{\*\}\\right\)\(7\)as the*empirical text\-only reconstruction gap*of the selected G2P system, not a theoretical irreducible bound\. To avoid circularity, the primary analysis uses only reading references that are manually verified or prepared independently of that G2P system\.

Table 1:Comparison of orthography–reading supervision interfaces\. “Orth\.” and “Reading” denote supervised outputs; “Span\-bound” denotes explicit correspondence between eachOjO\_\{j\}andRjR\_\{j\}; and “Speech forRR” denotes speech\-conditioned reading prediction rather than post\-ASR G2P\. Separate sequence targets lack explicit span correspondence, whereas Ruby\-ASR uses an interleaved span\-bound target\.

## 3Ruby\-ASR

### 3\.1Speech–language backbone

Ruby\-ASR is initialized from Qwen3\-ASR\[[25](https://arxiv.org/html/2609.27289#bib.bib2)\]\. Given speech𝐗\\mathbf\{X\}, the acoustic encoder and modality projector produce

𝐇a=ℰaud​\(𝐗\),𝐙a=𝒫⁡\(𝐇a\)\.\\mathbf\{H\}^\{\\mathrm\{a\}\}=\\mathcal\{E\}\_\{\\mathrm\{aud\}\}\(\\mathbf\{X\}\),\\hskip 20\.00003pt\\mathbf\{Z\}^\{\\mathrm\{a\}\}=\\mathcal\{P\}\(\\mathbf\{H\}^\{\\mathrm\{a\}\}\)\.\(8\)An autoregressive language\-model decoder predicts

p⁡\(𝐘∣𝐗\)=∏n=1Np⁡\(yn∣𝐘<n,𝐙a\)\.p\(\\mathbf\{Y\}\\mid\\mathbf\{X\}\)=\\prod\_\{n=1\}^\{N\}p\(y\_\{n\}\\mid\\mathbf\{Y\}\_\{<n\},\\mathbf\{Z\}^\{\\mathrm\{a\}\}\)\.\(9\)Under conventional supervision,𝐘=𝐎\\mathbf\{Y\}=\\mathbf\{O\}and the realized reading is not part of the output target\.

### 3\.2Span\-bound Ruby target

LetOjO\_\{j\}denote an annotated orthographic span,RjR\_\{j\}its realized reading, andUjU\_\{j\}unannotated material whose reading is already explicit, primarily kana\. We serialize the Ruby target as

𝐘ruby=U0O1\[R1\]U1⋯OJ\[RJ\]UJ\.\\mathbf\{Y\}^\{\\mathrm\{ruby\}\}=U\_\{0\}O\_\{1\}\[R\_\{1\}\]U\_\{1\}\\cdots O\_\{J\}\[R\_\{J\}\]U\_\{J\}\.\(10\)Readings are normalized to hiragana, and brackets are reserved structural tokens\. Annotation is performed at lexical\-span rather than character level, because compound readings are not necessarily compositional\. For example,

今日\[きょう\]forms one pair, while okurigana remains outside the annotation, as in

行\[い\]く\. A complete target may therefore be

明日\[あした\]は学校\[がっこう\]に行\[い\]く\. This extends the interleaved transcription–annotation formulation of prior work\[[16](https://arxiv.org/html/2609.27289#bib.bib28)\]to Japanese lexical\-span Ruby\.

Both output views can be recovered deterministically:

𝐎^\\displaystyle\\widehat\{\\mathbf\{O\}\}=𝒫orth\(𝐘^ruby\)=U0O1U1⋯OJUJ,\\displaystyle=\\mathcal\{P\}\_\{\\mathrm\{orth\}\}\(\\widehat\{\\mathbf\{Y\}\}^\{\\mathrm\{ruby\}\}\)=U\_\{0\}O\_\{1\}U\_\{1\}\\cdots O\_\{J\}U\_\{J\},\(11\)𝐑^\\displaystyle\\widehat\{\\mathbf\{R\}\}=𝒫read\(𝐘^ruby\)=ρ\(U0\)R1ρ\(U1\)⋯RJρ\(UJ\),\\displaystyle=\\mathcal\{P\}\_\{\\mathrm\{read\}\}\(\\widehat\{\\mathbf\{Y\}\}^\{\\mathrm\{ruby\}\}\)=\\rho\(U\_\{0\}\)R\_\{1\}\\rho\(U\_\{1\}\)\\cdots R\_\{J\}\\rho\(U\_\{J\}\),whereρ⁡\(⋅\)\\rho\(\\cdot\)normalizes unannotated material and removes non\-spoken symbols\. Thus, one generated sequence provides locally aligned orthographic and lexical\-reading transcriptions\. The Ruby sequence is trained autoregressively:

ℒruby=−∑n=1Nlogp\(ynruby∣𝐘<nruby,𝐙a\)\.\\mathcal\{L\}\_\{\\mathrm\{ruby\}\}=\-\\sum\_\{n=1\}^\{N\}\\log p\\\!\\left\(y\_\{n\}^\{\\mathrm\{ruby\}\}\\mid\\mathbf\{Y\}\_\{<n\}^\{\\mathrm\{ruby\}\},\\mathbf\{Z\}^\{\\mathrm\{a\}\}\\right\)\.\(12\)This makes the realized reading an explicit speech\-conditioned target\. However, becauseRjR\_\{j\}followsOjO\_\{j\}, the decoder may also exploit preceding orthographic context; Ruby supervision preserves and binds reading information, but does not by itself prove causal reliance on speech\.

![Refer to caption](https://arxiv.org/html/2609.27289v1/ruby_asr.png)Figure 1:Motivation and overview\. Orthographic supervision assigns an identical target to acoustically distinct readings\. Ruby\-ASR instead binds each orthographic span to the reading realized in speech\.
### 3\.3Auxiliary mora\-level regularization

To provide complementary monotonic reading supervision, the shared encoder is followed by additional Transformer layers\[[22](https://arxiv.org/html/2609.27289#bib.bib4),[30](https://arxiv.org/html/2609.27289#bib.bib5),[23](https://arxiv.org/html/2609.27289#bib.bib6),[21](https://arxiv.org/html/2609.27289#bib.bib7)\]and a mora\-level CTC head:

𝐇m=𝒯mora​\(𝐇a\),ℒCTC=−log⁡pCTC​\(𝐌∣𝐇m\),\\mathbf\{H\}^\{\\mathrm\{m\}\}=\\mathcal\{T\}\_\{\\mathrm\{mora\}\}\(\\mathbf\{H\}^\{\\mathrm\{a\}\}\),\\hskip 20\.00003pt\\mathcal\{L\}\_\{\\mathrm\{CTC\}\}=\-\\log p\_\{\\mathrm\{CTC\}\}\(\\mathbf\{M\}\\mid\\mathbf\{H\}^\{\\mathrm\{m\}\}\),\(13\)where𝐌\\mathbf\{M\}is a mora\-level reading sequence\. Joint training uses

ℒ=\(1−λ\)​ℒruby\+λ​ℒCTC\.\\mathcal\{L\}=\(1\-\\lambda\)\\mathcal\{L\}\_\{\\mathrm\{ruby\}\}\+\\lambda\\mathcal\{L\}\_\{\\mathrm\{CTC\}\}\.\(14\)Ruby generation remains the main inference path\. The CTC branch can also produce an optional decoder\-free kana hypothesis, but is treated primarily as auxiliary regularization in this work\.

### 3\.4Automatic target construction

Most source corpora contain orthographic transcripts but no speech\-grounded lexical readings\. Pronunciation\-annotated text corpora\[[19](https://arxiv.org/html/2609.27289#bib.bib21)\]and furigana\-annotated speech resources\[[20](https://arxiv.org/html/2609.27289#bib.bib19),[15](https://arxiv.org/html/2609.27289#bib.bib20)\]provide valuable coverage, but do not supply matched annotations for the full large\-scale ASR training pool\. We therefore use evidence\-constrained closed\-set annotation\. Each transcript is normalized and morphologically segmented to identify kanji\-bearing lexical units and separate surface okurigana\. Multiple open\-source ASR systems filter unreliable audio–text pairs, and an ensemble of three CTC models aligns reading evidence to the retained spans\. For eachOjO\_\{j\}, OpenJTalk supplies a candidate set𝒞⁡\(Oj\)\\mathcal\{C\}\(O\_\{j\}\); an open\-source language model may resolve contextual ambiguity only by selectingRj∈𝒞⁡\(Oj\)R\_\{j\}\\in\\mathcal\{C\}\(O\_\{j\}\), rather than freely generating a label\. Nonempty, canonical\-kana candidates consistent with the aligned utterance reading are serialized asOj​\[Rj\]O\_\{j\}\[R\_\{j\}\]; unresolved or structurally invalid samples are excluded from Ruby supervision\.

## 4Experimental Setup

### 4\.1Training data and transcription conventions

We fine\-tune Qwen3\-ASR using Eq\. \([14](https://arxiv.org/html/2609.27289#S3.E14)\) on a source pool of approximately 93\.1 million audio–text segments \(171\.5 thousand hours\) of read, spontaneous, expressive, and synthetic Japanese speech\. It combines internally curated data with ReazonSpeech\[[28](https://arxiv.org/html/2609.27289#bib.bib8)\], Common Voice 26\[[2](https://arxiv.org/html/2609.27289#bib.bib9)\], and selected JSUT subsets\[[26](https://arxiv.org/html/2609.27289#bib.bib10)\]\. No evaluation audio from B5K, CSJ, Book, CV8, or TEDx is used for training\. Synthetic speech increases coverage of medium\- and high\-difficulty lexical readings\. We construct separate training sets for two transcription conventions using the same architecture and objective\.*Ruby\-ASR\-sub*follows a concise subtitle\-style convention similar to Whisper and ReazonSpeech\.*Ruby\-ASR\-ver*follows Qwen3\-ASR’s more literal convention and uses a substantially larger verbatim\-style subset\. We fine\-tune all parameters withλ=0\.3\\lambda=0\.3\. The CTC branch predicts 278 mora\-level classes through a two\-layer, eight\-head Transformer adapter\. Training uses length\-grouped dynamic batches with a 600\-s padded\-audio budget and bfloat16 AdamW with a peak learning rate of2×10−52\\times 10^\{\-5\}, cosine decay, and 1% linear warm\-up\. Starting from an earlier fine\-tuned checkpoint with a reinitialized optimizer, we train for one epoch \(185,186 steps\), evaluate every 1,000 steps, and select the checkpoint with the lowest development loss\.

### 4\.2Benchmarks and baselines

We evaluate on JSUT\-BASIC5000 \(B5K\), the Corpus of Spontaneous Japanese \(CSJ\)\[[11](https://arxiv.org/html/2609.27289#bib.bib11)\], JSUT\-Book \(Book\), Common Voice 8 \(CV8\), and TEDx\[[1](https://arxiv.org/html/2609.27289#bib.bib13)\]\. Baselines are Whisper\-Large\-v3 \(Whisp\.\)\[[18](https://arxiv.org/html/2609.27289#bib.bib1)\], Kotoba\-v2 \(Kotoba\)\[[3](https://arxiv.org/html/2609.27289#bib.bib14)\], NeMo\- and k2\-based ReazonSpeech systems \(Rz\-NeMo and Rz\-k2\), Qwen3\-ASR \(Qwen\)\[[25](https://arxiv.org/html/2609.27289#bib.bib2)\], and Kana\-Whisper \(K\-Whisp\.\)\[[10](https://arxiv.org/html/2609.27289#bib.bib3)\]\.

### 4\.3Metrics

References and hypotheses are consistently normalized for numerals, long vowels, sokuon, punctuation, ruby marks, and kana variants\.Raw CERevaluates script\-preserving orthographic transcription, whereasSA\-CERcompares kana regions by reading and kanji regions by orthography, reducing penalties from kana–kanji variation\.Kana CERevaluates lexical readings using Ruby\-ASR’s direct reading projection or a shared G2P for orthographic baselines\. Oracle G2P applies the same system to ground\-truth orthography\. To avoid circularity, primary oracle results are limited to B5K and CSJ, whose reading references were independently verified\. For correctly recognized orthographic spans, we define

𝒥corr=j:O^j=Oj∗,C\-KanaCER=∑j∈𝒥corrdedit​\(R^j,Rj∗\)∑j∈𝒥corr\|Rj∗\|\.\\displaystyle\\mathcal\{J\}\_\{\\mathrm\{corr\}\}=\{j:\\widehat\{O\}\_\{j\}=O\_\{j\}^\{\*\}\},\\ \\mathrm\{C\\text\{\-\}KanaCER\}=\\frac\{\\sum\_\{j\\in\\mathcal\{J\}\_\{\\mathrm\{corr\}\}\}d\_\{\\mathrm\{edit\}\}\(\\widehat\{R\}\_\{j\},R\_\{j\}^\{\*\}\)\}\{\\sum\_\{j\\in\\mathcal\{J\}\_\{\\mathrm\{corr\}\}\}\|R\_\{j\}^\{\*\}\|\}\.\(15\)We report it for all eligible spans and the multi\-reading subset\|𝒞⁡\(Oj∗\)\|\>1\|\\mathcal\{C\}\(O\_\{j\}^\{\*\}\)\|\>1, together with span\-level exact\-reading accuracy \(R^j=Rj∗\\widehat\{R\}\_\{j\}=R\_\{j\}^\{\*\}\)\. Rare and unseen subsets are defined by the training frequencyf⁡\(Oj∗,Rj∗\)f\(O\_\{j\}^\{\*\},R\_\{j\}^\{\*\}\)of each exact orthography–reading pair:1≤f<101\\leq f<10andf=0f=0, respectively; evaluation data are excluded from these counts\.

Table 2:Raw CER and script\-aware CER \(SA\-CER, %\) on five benchmarks\. W\.Avg\. is character\-count weighted\. Lower is better\.

## 5Results

### 5\.1Orthographic and script\-aware recognition

Table[2](https://arxiv.org/html/2609.27289#S4.T2)shows no tradeoff in readable transcription: Ruby\-ASR\-sub achieves the lowest weighted Raw CER \(8\.51%\), and Ruby\-ASR\-ver the lowest weighted SA\-CER \(6\.59%\), compared with 10\.78% and 8\.59% for Qwen3\-ASR\. On JSUT\-Book, Ruby\-ASR\-sub decreases from 23\.21% Raw CER to 8\.80% SA\-CER, indicating that much of the raw error reflects script realization rather than recognized\-reading mismatch\.

Table 3:Kana CER \(%\) for lexical\-reading transcription\. Ruby\-ASR and Kana\-Whisper are scored from direct reading outputs; the remaining orthographic systems use a shared G2P\.∗marks manually verified reading references\.
### 5\.2Lexical\-reading recognition

Ruby\-ASR\-ver and Ruby\-ASR\-sub obtain weighted Kana CERs of 3\.75% and 4\.01%, compared with 5\.74% for Qwen3\-ASR, 5\.64% for Rz\-k2, and 4\.66% for Kana\-Whisper\. Ruby\-ASR\-ver outperforms the standalone mora\-CTC path on every benchmark, reducing weighted error from 7\.31% to 3\.75%, consistent with treating CTC as auxiliary monotonic supervision rather than a replacement for contextualized Ruby generation\. Although direct Ruby prediction performs best in aggregate, Kana\-Whisper remains best on B5K and CSJ, and Whisper\+G2P on JSUT\-Book\. These results do not establish whether the AR decoder uses speech evidence beyond preceding orthographic context; answering that question requires a controlled intervention outside this paper’s scope\.

Table 4:Oracle\-orthography G2P diagnostic on independently verified B5K and CSJ references\. Values are Kana CER \(%\)\. Oracle and Qwen hypotheses use the same G2P and normalization; Ruby\-ASR uses its direct reading projection\.
### 5\.3Oracle\-orthography G2P diagnostic

Applying the same G2P to ground\-truth orthography separates orthographic recognition errors from text\-only reading errors\. Oracle G2P in Table[4](https://arxiv.org/html/2609.27289#S5.T4)retains 1\.69% CER on B5K and 3\.47% on CSJ, confirming a nonzero text\-only reconstruction gap; relative to Qwen\+G2P, it reduces error by 1\.38 and 6\.43 points, respectively, showing the additional effect of orthographic ASR errors\. Ruby\-ASR\-ver and Ruby\-ASR\-sub outperform oracle G2P on B5K by 0\.61 and 0\.37 points but trail it on CSJ by 1\.94 and 1\.90 points, making the benefit domain dependent\. This comparison is diagnostic rather than causal because the systems are unmatched and the Ruby decoder also observes preceding orthographic context\.

Table 5:Results on G2P\-hard utterances, defined by nonzero reading edit distance for oracle\-orthography G2P\. B5K contains 1,124/5,000 \(22\.5%\) and CSJ 2,579/8,460 \(30\.5%\) such utterances\. Exact is utterance\-level exact\-reading accuracy \(%\)\.
### 5\.4G2P\-hard reading analysis

Table[5](https://arxiv.org/html/2609.27289#S5.T5)localizes gains to readings that the selected G2P cannot reconstruct from correct orthography\. On B5K, Ruby\-ASR\-ver reduces Kana CER from 6\.07% for oracle G2P and 4\.79% for Rz\-k2\+G2P to 1\.59%, with 57\.9% exact\-reading accuracy\. On CSJ, oracle G2P has slightly lower CER than Ruby\-ASR \(7\.65% vs\. 8\.02%\), but the Ruby models more often recover the complete utterance reading \(16\.8–17\.1% vs\. at most 5\.4% for ASR–G2P\)\. Direct prediction therefore benefits B5K most clearly, whereas spontaneous CSJ retains a speech\-recognition bottleneck\. Because this subset depends on one G2P implementation, it is a targeted diagnostic rather than an intrinsic partition of reading difficulty\.

Table 6:Conditional reading results on correctly recognized orthographic spans\. Each entry is C\-KanaCER / exact\-reading accuracy \(%\)\. Ranges cover Ruby\-ASR\-ver and Ruby\-ASR\-sub\. The all\-span counts are approximately 22\.5k for B5K and 19\.3k for CSJ; multi\-reading spans comprise about 49%\.
### 5\.5Conditional and ambiguity\-sensitive reading accuracy

Because overall Kana CER is dominated by text\-predictable readings, we use Eq\. \([15](https://arxiv.org/html/2609.27289#S4.E15)\) to evaluate correctly recognized orthographic spans\. Table[6](https://arxiv.org/html/2609.27289#S5.T6)shows that both Ruby variants exactly recover about 97% of readings when the orthographic span is correct\. Multi\-reading spans, roughly half of the eligible spans, retain similar accuracy, suggesting that dictionary\-level ambiguity is not the dominant residual error\. Degradation on rare and unseen pairs instead identifies lexical coverage as the main remaining challenge, especially on spontaneous CSJ\. Because its rare and unseen subsets contain only 245 and 116 spans, these results are diagnostic\. Conditioning on correct orthography prevents surface\-recognition errors from being attributed to reading selection\.

Table 7:Boundary overflow on the filtered ReazonSpeech test subset \(n=3,431n=3\{,\}431\)\. Start and End are normalized insertion rates \(%\) before and after the annotated subtitle span; Total is their sum\. Utt\. and≥\\geq3 count utterances with any and at least three overflow characters, respectively; Max is the maximum overflow length\. Lower is better\.
### 5\.6Subtitle\-boundary diagnostic

Ruby\-ASR\-sub targets concise subtitle\-style transcriptions restricted to the current segment\. After numeral normalization and punctuation removal, we align each hypothesis and reference at the character level\. Insertions before the first or after the last reference character define*start*and*end overflow*, respectively, and usually reflect neighboring speech captured near a segment boundary\. Thus, boundary overflow measures segment\-level output control rather than hallucination or recognition accuracy\. Ruby\-ASR\-sub achieves the second\-lowest total overflow \(0\.57%\) and affects fewer utterances than Whisper, Rz\-NeMo, and Rz\-k2\. Kotoba\-v2 performs best, consistent with its ReazonSpeech fine\-tuning and closer match to the reference conventions\. Despite one 21\-character outlier, Ruby\-ASR\-sub produces substantially fewer utterances with at least three overflow characters than the other non\-Kotoba systems\. Both ReazonSpeech baselines show more start overflow, suggesting greater sensitivity to preceding\-segment speech\. Overall, subtitle\-style training improves boundary control\. We treat this only as a boundary diagnostic because the references are insufficiently reliable for content\-level scoring\.

## 6Conclusion

We presented Ruby\-ASR, an evidence\-preserving refinement of Japanese ASR supervision\. Conventional orthographic targets can collapse different realized readings into one label, leaving post\-hoc G2P to infer the reading from text alone; span\-bound ruby targets instead retain orthographic identity and lexical reading in one structured sequence\. Across five benchmarks, Ruby\-ASR improves direct lexical\-reading recognition while maintaining strong orthographic and script\-aware transcription; a separate diagnostic shows that Ruby\-ASR\-sub generally restricts output to the annotated subtitle span\. Oracle\-orthography, G2P\-hard, and conditional\-reading analyses show that direct prediction is particularly effective on B5K cases where the selected G2P fails, while rare and unseen orthography–reading pairs remain a major limitation\. The released models provide structured Ruby generation and optional decoder\-free mora output for Japanese speech and reading\-aware research\.

## References

- \[1\]S\. Ando and H\. Fujihara\(2021\)Construction of a large\-scale japanese asr corpus on tv recordings\.InProc\. ICASSP,pp\.6948–6952\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP39728.2021.9413425)Cited by:[§4\.2](https://arxiv.org/html/2609.27289#S4.SS2.p1.1)\.
- \[2\]R\. Ardila, M\. Branson, K\. Davis, M\. Henretty, M\. Kohler, J\. Meyer, R\. Morais, L\. Saunders, F\. M\. Tyers, and G\. Weber\(2020\)Common voice: a massively\-multilingual speech corpus\.InProc\. LREC,pp\.4218–4222\.Cited by:[§4\.1](https://arxiv.org/html/2609.27289#S4.SS1.p1.1)\.
- \[3\]S\. Gandhi, P\. von Platen, and A\. M\. Rush\(2023\)Distil\-whisper: robust knowledge distillation via large\-scale pseudo labelling\.arXiv preprint arXiv:2311\.00430\.Cited by:[§4\.2](https://arxiv.org/html/2609.27289#S4.SS2.p1.1)\.
- \[4\]R\. Hu, X\. Lin, J\. Liu, S\. Huang, and Z\. Zhan\(2025\)Transcript\-prompted Whisper with dictionary\-enhanced decoding for Japanese speech annotation\.InProc\. Interspeech,pp\.2540–2544\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2025-76),[Link](https://www.isca-archive.org/interspeech_2025/hu25_interspeech.html)Cited by:[§1](https://arxiv.org/html/2609.27289#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.27289#S2.SS2.p1.2)\.
- \[5\]S\. Karita, Y\. Kubo, M\. A\. U\. Bacchiani, and L\. Jones\(2021\)A comparative study on neural architectures and training methods for japanese speech recognition\.InProc\. Interspeech,pp\.2092–2096\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2021-775)Cited by:[§1](https://arxiv.org/html/2609.27289#S1.p1.1)\.
- \[6\]S\. Karita, R\. Sproat, and H\. Ishikawa\(2023\)Lenient evaluation of japanese speech recognition: modeling naturally occurring spelling inconsistency\.InProc\. CAWL,pp\.61–70\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.cawl-1.8)Cited by:[§1](https://arxiv.org/html/2609.27289#S1.p1.1)\.
- \[7\]T\. Koriyama\(2026\)Benchmarking large language models for grapheme\-to\-phoneme conversion: a Japanese case study\.arXiv preprint arXiv:2606\.22009\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2606.22009),[Link](https://arxiv.org/abs/2606.22009)Cited by:[§1](https://arxiv.org/html/2609.27289#S1.p1.1)\.
- \[8\]Krishna D N\(2021\)A dual\-decoder conformer for multilingual speech recognition\.arXiv preprint arXiv:2109\.03277\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2109.03277),[Link](https://arxiv.org/abs/2109.03277)Cited by:[§1](https://arxiv.org/html/2609.27289#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.27289#S2.SS2.p1.1)\.
- \[9\]Y\. Kubo and M\. Bacchiani\(2020\)Joint phoneme\-grapheme model for end\-to\-end speech recognition\.InProc\. IEEE Int\. Conf\. Acoust\., Speech Signal Process\. \(ICASSP\),pp\.6119–6123\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP40776.2020.9054557),[Link](https://doi.org/10.1109/ICASSP40776.2020.9054557)Cited by:[§1](https://arxiv.org/html/2609.27289#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.27289#S2.SS2.p1.1)\.
- \[10\]L\. Liu, S\. Zhu, K\. Washizaki, R\. Yoneyama, H\. Jeon, M\. Zhao, Y\. Fujita, H\. Shi, N\. Yoshida, Y\. Gao, R\. Koshkin, Y\. Hono, and Y\. Sudo\(2026\)Sarashina2\.2\-TTS: tackling kanji polyphony in japanese speech generation via data scaling and targeted data synthesis\.arXiv preprint arXiv:2606\.25369\.Cited by:[§1](https://arxiv.org/html/2609.27289#S1.p1.1),[§4\.2](https://arxiv.org/html/2609.27289#S4.SS2.p1.1)\.
- \[11\]K\. Maekawa\(2003\)Corpus of spontaneous japanese: its design and evaluation\.InProc\. SSPR,Cited by:[§4\.2](https://arxiv.org/html/2609.27289#S4.SS2.p1.1)\.
- \[12\]Y\. Matogawa, Y\. Sakai, T\. Watanabe, and C\. Taguchi\(2024\)Japanese rule\-based grapheme\-to\-phoneme conversion system and multilingual named entity dataset with international phonetic alphabet\.InProc\. SIGMORPHON,pp\.77–86\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.sigmorphon-1.9)Cited by:[§1](https://arxiv.org/html/2609.27289#S1.p1.1)\.
- \[13\]R\. Mibayashi, H\. Takamura, and H\. Yanaka\(2026\)YOMI\-Bench: a benchmark for evaluating Kanji reading and phonological understanding of LLMs for Japanese\.arXiv preprint arXiv:2607\.00664\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2607.00664),[Link](https://arxiv.org/abs/2607.00664)Cited by:[§1](https://arxiv.org/html/2609.27289#S1.p1.1)\.
- \[14\]S\. A\. Nadig, V\. Ramasubramanian, and S\. Rao\(2020\)Multi\-target hybrid CTC\-attentional decoder for joint phoneme\-grapheme recognition\.InProc\. IEEE Int\. Conf\. Signal Process\. Commun\. \(SPCOM\),pp\.1–5\.External Links:[Document](https://dx.doi.org/10.1109/SPCOM50965.2020.9179603),[Link](https://doi.org/10.1109/SPCOM50965.2020.9179603)Cited by:[§1](https://arxiv.org/html/2609.27289#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.27289#S2.SS2.p1.1)\.
- \[15\]National Diet Library\(2025\)Aozora Bunko furigana\-annotated speech corpus\.Note:Online datasetVersion 2, accessed Aug\. 27, 2026External Links:[Link](https://github.com/ndl-lab/hurigana-speech-corpus-aozora)Cited by:[§3\.4](https://arxiv.org/html/2609.27289#S3.SS4.p1.1)\.
- \[16\]M\. Omachi, Y\. Fujita, S\. Watanabe, and M\. Wiesner\(2021\)End\-to\-end ASR to jointly predict transcriptions and linguistic annotations\.InProc\. 2021 Conf\. North Amer\. Chapter Assoc\. Comput\. Linguistics: Human Lang\. Technol\. \(NAACL\-HLT\),pp\.1861–1871\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.149),[Link](https://aclanthology.org/2021.naacl-main.149/)Cited by:[§1](https://arxiv.org/html/2609.27289#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.27289#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2609.27289#S3.SS2.p1.5)\.
- \[17\]S\. Passaglia\(2023\)Yomikata: heteronym disambiguation for Japanese\.Note:SoftwareVersion 0\.0\.4, accessed Aug\. 27, 2026External Links:[Link](https://github.com/passaglia/yomikata)Cited by:[§1](https://arxiv.org/html/2609.27289#S1.p1.1)\.
- \[18\]A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. Sutskever\(2023\)Robust speech recognition via large\-scale weak supervision\.InProc\. ICML,pp\.28492–28518\.Cited by:[§4\.2](https://arxiv.org/html/2609.27289#S4.SS2.p1.1)\.
- \[19\]F\. Sato, N\. Yoshinaga, and M\. Kitsuregawa\(2022\)Building large\-scale Japanese pronunciation\-annotated corpora for reading heteronymous logograms\.InProc\. 13th Lang\. Resources Eval\. Conf\. \(LREC\),pp\.7113–7121\.External Links:[Link](https://aclanthology.org/2022.lrec-1.770/)Cited by:[§1](https://arxiv.org/html/2609.27289#S1.p1.1),[§3\.4](https://arxiv.org/html/2609.27289#S3.SS4.p1.1)\.
- \[20\]F\. Sato, N\. Yoshinaga, M\. Toyoda, and M\. Kitsuregawa\(2025\)Reconstruction of the Aozora Bunko furigana\-annotated speech corpus using prompts and multiple speech recognition candidates\.InProc\. 31st Annu\. Meeting Assoc\. Natural Lang\. Process\. \(NLP 2025\),pp\.3092–3096\.Note:\(in Japanese\)External Links:[Link](https://www.anlp.jp/proceedings/annual_meeting/2025/pdf_dir/B8-5.pdf)Cited by:[§3\.4](https://arxiv.org/html/2609.27289#S3.SS4.p1.1)\.
- \[21\]H\. Shi, Y\. Fujita, R\. Koshkin, M\. Zhao, Y\. Gao, L\. Liu, and Y\. Sudo\(2026\)Distilling LLM semantic priors into encoder\-only multi\-talker ASR with talker\-count routing\.arXiv preprint arXiv:2603\.10587\.Cited by:[§3\.3](https://arxiv.org/html/2609.27289#S3.SS3.p1.1)\.
- \[22\]H\. Shi, Y\. Fujita, T\. Mizumoto, L\. Liu, A\. Kojima, and Y\. Sudo\(2025\)Serialized output prompting for large language model\-based multi\-talker speech recognition\.InProc\. ASRU,pp\.1–8\.External Links:[Document](https://dx.doi.org/10.1109/ASRU65441.2025.11434801)Cited by:[§3\.3](https://arxiv.org/html/2609.27289#S3.SS3.p1.1)\.
- \[23\]H\. Shi, Y\. Gao, Z\. Ni, and T\. Kawahara\(2024\)Serialized speech information guidance with overlapped encoding separation for multi\-speaker automatic speech recognition\.InProc\. SLT,pp\.193–199\.External Links:[Document](https://dx.doi.org/10.1109/SLT61566.2024.10832330)Cited by:[§3\.3](https://arxiv.org/html/2609.27289#S3.SS3.p1.1)\.
- \[24\]H\. Shi, Y\. Gao, X\. Lu, and T\. Kawahara\(2026\)Beyond Acoustic Prefixes: Persistent Grounding in Serialized Acoustic Memory for LLM\-Based Multi\-Talker Speech Recognition\.External Links:2603\.27205,[Link](https://arxiv.org/abs/2603.27205)Cited by:[§1](https://arxiv.org/html/2609.27289#S1.p2.1)\.
- \[25\]X\. Shi, X\. Wang, Z\. Guo, Y\. Wang, P\. Zhang, X\. Zhang, Z\. Guo, H\. Hao, Y\. Xi, B\. Yang, J\. Xu, J\. Zhou, and J\. Lin\(2026\)Qwen3\-ASR technical report\.arXiv preprint arXiv:2601\.21337\.Cited by:[§3\.1](https://arxiv.org/html/2609.27289#S3.SS1.p1.1),[§4\.2](https://arxiv.org/html/2609.27289#S4.SS2.p1.1)\.
- \[26\]R\. Sonobe, S\. Takamichi, and H\. Saruwatari\(2017\)JSUT corpus: free large\-scale japanese speech corpus for end\-to\-end speech synthesis\.arXiv preprint arXiv:1711\.00354\.Cited by:[§4\.1](https://arxiv.org/html/2609.27289#S4.SS1.p1.1)\.
- \[27\]Y\. Sudo, K\. Hata, and K\. Nakadai\(2023\)Retraining\-free customized asr for enharmonic words based on a named\-entity\-aware model and phoneme similarity estimation\.InProc\. Interspeech,pp\.491–495\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2023-1320)Cited by:[§1](https://arxiv.org/html/2609.27289#S1.p1.1)\.
- \[28\]Y\. Yin, D\. Mori, and S\. Fujimoto\(2023\)ReazonSpeech: a free and massive corpus for japanese ASR\.InProc\. NLP,pp\.1134–1139\.Cited by:[§4\.1](https://arxiv.org/html/2609.27289#S4.SS1.p1.1)\.
- \[29\]W\. Zhang\(2023\)Pronunciation ambiguities in japanese kanji\.InProc\. CAWL,pp\.50–60\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.cawl-1.7)Cited by:[§1](https://arxiv.org/html/2609.27289#S1.p1.1)\.
- \[30\]J\. Zhao, H\. Shi, C\. Cui, T\. Wang, H\. Liu, Z\. Ni, L\. Ye, and L\. Wang\(2025\)Adapting Whisper for code\-switching through encoding refining and language\-aware decoding\.InProc\. ICASSP,pp\.1–5\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP49660.2025.10889634)Cited by:[§3\.3](https://arxiv.org/html/2609.27289#S3.SS3.p1.1)\.