I recently discovered an open-source tool called Claude-real-video that directly addresses a fundamental limitation in current LLM-based video analysis.

Until now, the entire industry has been obsessed with “natively baking video capabilities into large models” — essentially requiring every company to build a phone with a screen from scratch. That means either waiting for big players to drip-feed new features, or burning cash to train smaller models yourself. Ordinary developers simply can’t compete.

This tool takes a different approach: it’s a universal external module. No matter which LLM you’re using, plug it in and it can directly parse video content. It’s like adding a display to a feature phone that could only handle voice calls.

And it’s not just simple frame sampling every few seconds.

What’s even more impressive is that the community discussions around this project are full of practical insights — not the empty star-collecting kind. People talk about real-world limitations, suggest alternatives, and share actual use cases. Developers building video agents can pull this down and try it right away.

HUANGCHIHHUNGLeo/claude-real-video

Source: https://github.com/HUANGCHIHHUNGLeo/claude-real-video

claude-real-video

Let Claude — or any LLM — actually watch a video.

Most AI tools don’t really see a video. Paste a YouTube link into ChatGPT and it reads the transcript, not the picture. Claude won’t take a video file at all. Even Gemini, which can read video natively, has to send it up to Google and samples frames at a fixed interval (1 fps by default), so fast cuts slip past.

claude-real-video does it differently, and locally: point it at a URL or a file, and it pulls the frames that actually matter (every scene change, not a fixed quota), throws away the near-duplicates, transcribes the audio, and hands you a clean folder any LLM can read — on your own machine, nothing uploaded.

crv "https://www.youtube.com/watch?v=..."
# → crv-out/frames/*.jpg  +  crv-out/transcript.txt  +  crv-out/MANIFEST.txt

Then drop the frames + MANIFEST.txt into Claude / ChatGPT / Gemini and ask away.

Why not just sample frames?

Most “let an LLM watch a video” scripts (and Gemini’s own pipeline) grab frames at a fixed interval — e.g. one per second. That over-samples a static screencast and under-samples a fast-cut reel. claude-real-video is smarter:

You feed the model fewer, more meaningful frames — cheaper context, better understanding.

Install

pip install claude-real-video              # core (frames + dedup)
pip install "claude-real-video[whisper]"   # + audio transcription

System requirement: ffmpeg

ffmpeg / ffprobe are used for frame extraction and audio, and aren’t pip-installable. Install them once:

Verify it’s on your PATH:

ffmpeg -version

Transcription uses the whisper CLI (installed by the [whisper] extra, or pip install openai-whisper). Whisper also relies on ffmpeg.

Works on macOS, Windows, and Linux — Python 3.10+.

Usage

# A YouTube / Instagram / TikTok / ... link
crv "https://www.instagram.com/reel/XXXX/"

# A local file, English transcript, output to ./out
crv lecture.mp4 -o out --lang en

# Frames only, no transcription
crv clip.mp4 --no-transcribe

# A login-gated video (your own / authorised use): pass a Netscape cookie file
crv "https://..." --cookies cookies.txt

python -m claude_real_video ... works as an alias for crv too.

Options

Use it from Python

from claude_real_video import process

r = process("https://youtu.be/...", "out", lang="en")
print(r.frame_count, r.transcript_path)

How it works

1. Fetch — yt-dlp for URLs (optional cookies), or copy a local file.
2. Extract — one chronological ffmpeg select pass grabs every scene change plus a density floor (at least one frame every --fps-floor seconds), so fast cuts and slow screencasts are both covered.
3. Dedup — real pixel difference (downscaled RGB, not a perceptual hash — hashes go blind on flat colours and equal-luma hue changes) against a sliding window of the last --dedup-window kept frames, so an A-B-A cutaway doesn’t re-send a shot the model has already seen. --report writes report.html showing every keep/drop decision with its diff %, for tuning.
4. Text — if the video already has subtitles (a sidecar .srt/.vtt next to a local file, or an embedded subtitle track), those are used as the transcript — faster and more accurate than re-transcribing. Only when there are no subtitles does it fall back to Whisper on the audio (skipped cleanly if there’s no audio).
5. Audio (optional, --keep-audio) — save the full original soundtrack (audio.m4a: music + speech + effects, copied losslessly when possible). The transcript only has the words; the audio file lets a model that can listen (Gemini, GPT-4o, …) actually hear the music and tone.
6. Manifest — MANIFEST.txt summarises everything for the model.

So the model can see (key frames), read (transcript) and — with --keep-audio — hear (full soundtrack) the video. The transcript is plain text any model can read; the tool doesn’t burn subtitles into the video — burning is a presentation choice, not something needed to make a video AI-readable.

Notes

• Only download content you have the right to. The --cookies option is for your own, authorised access — don’t ship credentials in a repo.
• Re-running overwrites the output directory.

License

MIT