这个开源工具要是现在不收藏，将来肯定得后悔——视频自动配音翻译，一口气支持 33 种语言，还能直接对视频内容提问。

在 GitHub 上发现一个宝藏工具，叫 Violin，完全开源，做的事情说出来有点离谱：你把视频丢进去，它自动识别语音、翻译、合成目标语言的配音，再无缝混回视频里，时间轴完全对齐，还顺手给你输出 SRT 字幕。整个流程一条龙，不用你手动碰任何东西。

底层跑的是 Whisper Large v3 做语音识别，DeepSeek V4 Pro 负责翻译，Cartesia Sonic 3 合成配音，ffmpeg 前后处理，整个 pipeline 设计得很干净。

功能上有几个点我觉得挺有意思：

支持 33 种目标语言，其中 16 种配有精选母语配音，用的是 Cartesia Sonic 3 加 ElevenLabs，听感不是那种机器腔。

视频内 Q&A，配音完之后你可以对视频任意时刻提问，它基于附近字幕和采样帧给你答案，这个设计有点超出预期。

自然语言选声音，你描述想要什么风格的声音，LLM 自动从语音库里帮你挑，不用一个个试。

六种风格预设：标准、儿童、学术、休闲、讲故事、新闻，每种预设连语速和情绪都调好了，直接用。

可插拔架构，转录、翻译、TTS 各阶段都能换，Together、OpenAI、ElevenLabs 随便组合，一个 YAML 文件配置搞定，不想动代码的人也能玩。

GitHub： https://github.com/shang-zhu/violin… 在线体验： https://violin-ai.com

shang-zhu/violin

Source: https://github.com/shang-zhu/violin

🎻 Violin

Open-source Video Translation Skill.

🌐 Live demo · 📝 Blog post · 📜 MIT License

Upload a video. Violin transcribes the speech, translates it, synthesizes a native-sounding voice-over in the target language, and remuxes it back into the video — fully aligned, with optional SRT subtitles.

Available as a CLI, a FastAPI web app, and a Claude Code skill.

✨ Features

• 33 target languages with handpicked native-speaker voices for the 16 most-used ones (Cartesia Sonic 3 + ElevenLabs)
• In-video Q&A — ask questions about any moment in the dubbed video; answers use nearby subtitles plus sampled frames
• Natural-language voice picker — describe the voice you want, an LLM picks from the catalog
• 6 style profiles (experimental) — standard / kids / academic / casual / storyteller / news
• Pluggable stack — Together / OpenAI / ElevenLabs interchangeable for every stage, one YAML

🚀 Quick start

Try it without installing anything

The live demo runs at https://www.violin-ai.com — drop a short clip in, get a dubbed video out in a few minutes.

Run locally

Requires Python 3.10+ and ffmpeg on PATH.

curl -LsSf https://astral.sh/uv/install.sh | sh   # install uv if you don't have it
uv tool install violin                            # recommended — faster, isolated
# or: pip install violin                          # if you'd rather install into your current Python env

export TOGETHER_API_KEY=...                       # get one at https://api.together.ai (add to ~/.zshrc to persist)

Three ways to use it:

1. CLI — translate one file:

violin lecture.mp4 lecture_zh.mp4 --language Chinese

2. Web app — full REST API + browser UI:

violin-api
# → http://127.0.0.1:8000           (browser UI)
# → http://127.0.0.1:8000/docs      (interactive API docs)

3. Claude Code skill — invoke from any Claude Code session:

violin --install-skill          # one-time: copies the skill into ~/.claude/skills/
claude
> please use the violin skill to translate path/to/video.mp4 into Chinese

git clone https://github.com/shang-zhu/violin.git
cd violin
uv sync
cp .env.example .env             # then fill in TOGETHER_API_KEY
uv run main.py lecture.mp4 lecture_zh.mp4 --language Chinese

uv tool uninstall violin     # if you've installed the PyPI version
uv tool install --editable .

📝 To Do List

• [-] support voice cloning.
• [-] lip sync generation.

🎬 How Violin works

Video
  │
  ├─ ffmpeg ─────────────────────► Extract audio (16 kHz WAV)
  │
  ├─ Whisper Large v3 ────────────► Word-level timestamps → sentence segments
  │
  ├─ LLM (DeepSeek V4 Pro by default) ──► Translate each segment, respecting style profile
  │
  ├─ TTS (Cartesia Sonic 3 by default) ─► Synthesize dubbed audio per segment
  │
  └─ ffmpeg ─────────────────────► Speed-align video to dubbed audio,
                                    concat with freeze-frame fallback,
                                    single-pass AAC encode the audio track,
                                    write output mp4 + optional SRT

⚙️ Configuration

Override any default by writing your own YAML and passing it with --config my.yaml — only the keys you want to change need to appear; values deep-merge with the built-in defaults.

Switch providers

# config/default.yaml — pick the stack you want
models:
  transcription:
    provider: together                  # together | openai
    model: openai/whisper-large-v3      # together → openai/whisper-large-v3 | openai → whisper-1
  translation:
    provider: together                  # together | openai
    model: deepseek-ai/DeepSeek-V4-Pro  # together → deepseek-ai/DeepSeek-V4-Pro | openai → gpt-5.5
  tts:
    provider: together                  # together | elevenlabs | openai
    model: cartesia/sonic-3             # together → cartesia/sonic-3 | elevenlabs → eleven_v3 | openai → tts-1-hd

Production overrides

A starter config/prod.yaml is included for public deployments. It adds upload limits, serializes jobs, and caps ffmpeg concurrency. The included Dockerfile + docker-compose.yml + Caddyfile are how the live demo is hosted — docker compose up -d --build after filling .env is enough to put a copy of Violin behind auto-HTTPS on any Docker host.

Environment variables

You only need keys for the providers you actually pick. Pure-OpenAI deployments (all stages on openai) work too — OPENAI_API_KEY alone is enough. Same idea for ElevenLabs.

🎭 Style profiles

Six built-in profiles tune both the translation LLM prompt and the TTS delivery. Use --style <name> on the CLI or pass style in API requests.

Add your own by editing prompts/styles.yaml.

See all available styles: violin --style list.

💻 CLI usage

Examples use the PyPI-installed violin command. If you’re running from a git checkout, substitute uv run main.py for violin (and uv run run_api.py for violin-api).

# Basic
violin lecture.mp4 lecture_es.mp4 --language Spanish

# Pick a style
violin talk.mp4 talk_zh.mp4 --language Chinese --style kids

# Pick a specific voice
violin lecture.mp4 lecture_fr.mp4 --language French --voice "french narrator man"

# Skip SRT
violin lecture.mp4 lecture_ja.mp4 --language Japanese --no-subtitles

# Full replacement (no original audio underneath)
violin lecture.mp4 lecture_ko.mp4 --language Korean --no-voiceover

# Custom config (e.g. switch to OpenAI/ElevenLabs)
violin lecture.mp4 lecture_it.mp4 --language Italian --config config/other_api.yaml

CLI flags

🛰️ Web app & REST API

violin-api                              # default dev mode
violin-api --host 0.0.0.0 --port 8080   # bind everywhere
violin-api --config config/prod.yaml    # production overrides (requires a git checkout for config/prod.yaml)

Core flow: POST /jobs to start, GET /jobs/{id} to poll, GET /jobs/{id}/video and /srt to download, POST /jobs/{id}/chat for in-video Q&A. Full list with request/response schemas at /docs.

Example

# Submit
JOB=$(curl -s -X POST http://localhost:8000/jobs \
  -F "[email protected]" \
  -F "language=Spanish" \
  -F "style=academic" | jq -r .id)

# Poll
curl -s http://localhost:8000/jobs/$JOB | jq '{status, progress}'

# Download
curl -OJ http://localhost:8000/jobs/$JOB/video
curl -OJ http://localhost:8000/jobs/$JOB/srt

Job data lives under jobs/{id}/. Set api.job_ttl_hours to auto-delete jobs older than N hours (default 0 = disabled; config/prod.yaml uses 24h for the public demo).

🌍 Supported languages

Violin supports 33 target languages. The 16 below ship with handpicked native-speaker voices for each provider; the rest fall back to the English voice catalog (which is multilingual under both Cartesia Sonic 3 and ElevenLabs eleven_v3).

Ordered by native-speaker population.

The 17 fallback languages (using the English voice catalog), also ordered by native speakers: Vietnamese, Tamil, Indonesian, Malay, Ukrainian, Romanian, Thai, Greek, Hungarian, Catalan, Czech, Bulgarian, Danish, Slovak, Croatian, Finnish, Norwegian.

🤝 Contributing

PRs welcome. Got questions or hit a bug? Email [email protected] or open an issue.

⚠️ Disclaimer

This is a personal open-source project, not a Together AI product. Users are responsible for ensuring they have the right to download and translate any content they process. Designed for Creative Commons, public domain, your own recordings, and other content you have permission to use.

📜 License

MIT — use it freely, including commercially.

🙏 Acknowledgements

Built on top of Together AI, Whisper, Cartesia Sonic 3, ElevenLabs, FastAPI, and ffmpeg.