GitHub 上有一份把「语音语言模型（SpeechLM）」研究脉络梳理得很清楚的资源库：Awesome-SpeechLM-Survey。 它把分类框架、代表模型、训练数据集到评测基准一站式整理成“知识地图”，查资料、补背景、找对标都很省时间。 GitHub：https://github.com/dreamtheater123/Awesome-SpeechLM-Survey… 你能在里面看到： - 50+ 语音语言模型全景盘点：GPT-4o、Moshi、Mini-Omni 等都有条理化整理 - 语音 tokenizer 技术谱系：语义 / 声学 / 混合方案一网打尽 - 20+ 训练数据集汇总：从 LibriSpeech 到最新指令数据覆盖完整 - 10+ 评测基准对比：包含自研 VoxEval 等关键基准 想系统吃透语音 AI 的技术路线与研究进展，这份仓库很适合用来快速建立认知框架并深入追踪。

dreamtheater123/Awesome-SpeechLM-Survey

Source: https://github.com/dreamtheater123/Awesome-SpeechLM-Survey

Awesome-SpeechLM-Survey

🎉🎉🎉Our survey paper “Recent Advances in Speech Language Models: A Survey” has been accepted to ACL 2025 main conference!

This is the Github repository for paper: Recent Advances in Speech Language Models: A Survey. In this paper, we survey the field of Speech Language Models (SpeechLMs), which are capable of performing end-to-end speech interactions with humans and serve as autoregressive foundation models.

News

• We have released VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models on arXiv!🎉 Also check it out on Github.

Introduction

Why SpeechLMs? SpeechLMs are used for end-to-end speech-based interactions. Traditional ASR + LLM + TTS setups suffer from information loss and cumulative errors during conversion. SpeechLMs directly model speech data, capturing both semantic and paralinguistic information for richer interactions!

Taxonomy

We introduce a novel taxonomy for SpeechLMs, categorizing them based on their architecture and training recipes.

Existing SpeechLMs

SpeechLM Tokenizers

Semantic Tokenizers

Acoustic Tokenizers

Mixed Tokenizers

Popular Training Datasets

Evaluation Benchmarks

Citation

@article{cui2024recent,
  title={Recent advances in speech language models: A survey},
  author={Cui, Wenqian and Yu, Dianzhi and Jiao, Xiaoqi and Meng, Ziqiao and Zhang, Guangyan and Wang, Qichao and Guo, Yiwen and King, Irwin},
  journal={arXiv preprint arXiv:2410.03751},
  year={2024}
}