claude-video: The /watch Skill That Lets Claude Actually Watch Any Video

Claude can read a webpage, run a script, browse a repo. What it cannot do, out of the box, is watch a video. Paste a YouTube link and it either guesses from the title or pulls a transcript that misses 90% of what is on screen. claude-video closes that gap with a single command, /watch, that downloads the video, extracts frames, transcribes the audio, and hands everything to Claude so it finally sees the picture.

1. What It Is

claude-video (github.com/bradautomates/claude-video) is an open-source Agent Skill. It has 14,204 GitHub stars (as of 2026-08-06, subject to real-time change), 1,363 forks, primary language Python, MIT license, created on 2026-04-24, with its latest push on 2026-07-01. This week it ranks #20 on the GitHub weekly trending list with a gain of 3,359 stars. The official one-liner: "Give Claude the ability to watch any video" - /watch downloads a video, extracts frames, transcribes it, and hands it all to Claude.

It is not a standalone app. It ships as a Skill: a plugin in Claude Code, a global skill in Codex, Cursor, Copilot, Gemini CLI, and 50+ other Agent Skills hosts. Under the hood it leans on yt-dlp (download + native captions), ffmpeg (frame extraction), and optional Whisper (transcription when captions are absent). Author Brad Bonanno packages the whole pipeline into a self-contained skills/watch/ folder that every installer copies as a unit.

2. The Pain Point It Solves

One sentence: Claude natively cannot watch video.

Give Claude a YouTube link and it has no way to "watch." Best case, the video has captions and Claude pulls a transcript - but a transcript only captures what is spoken. Everything on screen - the UI, the code, the charts, the slides, the demo, the presenter's actions - is gone. The README puts it bluntly: a transcript is "missing 90% of what's on screen."

That blocks several real workflows:

• Analyzing someone else's content. You want to break down a viral video's opening hook, pacing, cuts. Reading the transcript cannot tell you "at second 3 the frame hard-cuts to a product close-up."
• Diagnosing a bug from a screen recording. A teammate sends a recording saying "it crashes here." You cannot ask the AI to watch the recording and find the frame where the issue appears.
• Summarizing a long video fast. Even at 2x speed it costs time, and an AI summarizing from captions alone misses every on-screen demonstration.
• Stripping hype from an update video. A "game-changer" launch has ten minutes of intro and overselling. You want the few things that actually changed, and captions cannot tell you "this frame is the real feature demo."

/watch solves this by splitting the video into two modalities Claude can consume - frames (images) + transcript (text) - and letting Claude read every frame like a picture before it answers.

3. How /watch Works

The README breaks /watch into six steps. It is a pipeline: download -> extract frames -> transcribe -> hand to Claude.

1. You paste a video and a question. A URL (anything yt-dlp supports: YouTube, Loom, TikTok, X, Instagram, and a few hundred more) or a local path (.mp4, .mov, .mkv, .webm).
2. yt-dlp checks captions first. In transcript mode, captioned URLs return without downloading video. It only downloads when audio is needed, and only what the run needs.
3. ffmpeg extracts frames at the chosen detail. efficient decodes keyframes only (near-instant); balanced/token-burner prefer scene-change frames and fall back to a duration-aware uniform sampler. JPEGs are 512px wide by default, clamped to 1998px tall for Claude Read compatibility.
4. The transcript comes from one of two places. First try: yt-dlp pulls native captions (manual or auto-generated) - free, instant, accurate-ish. Fallback: extract a mono 16 kHz 64 kbps mp3 clip (~480 kB/min) and send it to Whisper - Groq's whisper-large-v3 (preferred, cheaper and faster) or OpenAI's whisper-1.
5. Frames + transcript are handed to Claude. The script prints frame paths with t=MM:SS markers and a timestamped transcript. Claude Reads each frame in parallel - JPEGs render directly as images in its context.
6. Claude answers grounded in what it actually saw and heard. Not "based on the description" or "according to the title," but the way someone who watched the video would. The temp directory is cleaned up afterward unless you plan follow-up questions.

The critical design choice is the frame budget. Every frame is an image, and image tokens add up fast, so the script runs auto-fps logic to stop you blowing your context budget on a sparse scan of a 30-minute video:

A second token-saver is frame deduplication. A screen recording that holds one slide for 90 seconds produces a dozen near-identical frames, each billed as a separate image. Dedup runs by default (--no-dedup turns it off): each frame is scaled to a 16×16 grayscale thumbnail, the mean absolute difference is computed against the last kept frame, and anything at or below threshold 2.0 is dropped. Comparing against the last kept frame (not the previous one) catches slow fades. The frame cap applies after dedup, so the budget is spent on distinct frames.

4. Up and Running in Three Minutes

Claude Code (recommended - auto-updates via marketplace):

/plugin marketplace add bradautomates/claude-video
/plugin install watch@claude-video

Codex / Cursor / Copilot / Gemini CLI and 50+ hosts:

npx skills add bradautomates/claude-video -g

-g installs globally (~/.codex/skills, ~/.cursor/skills, etc.); drop it to scope per-project.

First run: On the first /watch, scripts/setup.py --check verifies ffmpeg/yt-dlp on your PATH and whether a Whisper key is set. If missing, it walks you through it - macOS auto-runs brew install, Linux/Windows print the exact apt/dnf/winget/pip commands. The check is a sub-100ms lookup, silent on subsequent runs.

Use it:

/watch https://youtu.be/dQw4w9WgXcQ what happens at the 30 second mark?
/watch https://www.tiktok.com/@user/video/123 summarize this
/watch ~/Movies/screen-recording.mp4 when does the UI break?

Focus a section (denser frames, lower token cost):

/watch https://youtu.be/abc --start 2:15 --end 2:45
/watch video.mp4 --start 50 --end 60
/watch "$URL" --start 1:12:00            # from 1h12m to end

Do you need an API key? Most public videos have native captions and run free. The Whisper fallback only triggers when a video genuinely has no caption track (local files, some TikToks/Vimeos, the occasional caption-less YouTube upload). Then grab a GROQ_API_KEY (preferred) from the Groq Console or an OPENAI_API_KEY from OpenAI and drop it in ~/.config/watch/.env. Add --no-whisper to skip transcription entirely. Use placeholders like sk-xxx in docs; never commit real keys.

Common knobs:

The four detail modes, measured by the README against a real 49:08 YouTube screen recording (1280×720, English auto-captions):

efficient is the speed tier (it only reconstructs keyframes, ~40× faster than the scene modes); token-burner only diverges from balanced past the cap - this clip had 116 cuts, so balanced sampled 100 and token-burner kept them all.

5. Who It Is For, and the Gotchas

Who it is for:

• Content analysts. Breaking down viral hooks, ad creative, competitor launches, podcast intros - anywhere the how matters as much as the what.
• Developers debugging from recordings. A teammate sends a "it breaks here" screen recording; Claude watches it, finds the frame, and often pinpoints the cause.
• Note-takers from long video. Run /watch summarize this to a note across a course or channel and turn hours of video into a searchable note set.
• Hype filters. what's actually new - skip the hype strips a launch down to the few things that matter.

Gotchas:

1. Long-video token cost. The biggest one. Frames are images; image tokens add up fast. balanced caps at 100 frames, and past ~10 minutes coverage thins out - the script prints a "sparse scan" warning. Either re-run focused with --start/--end, or --detail token-burner to lift the cap (tokens climb faster). --resolution 1024 roughly 4×'s per-frame tokens; only use it when you need to read on-screen text.
2. ffmpeg / yt-dlp are hard dependencies. No install, no run. macOS auto-installs via brew on first run; Linux/Windows print commands you have to run yourself. On claude.ai web you must enable "Code execution and file creation" under Capabilities first, because the skill shells out to ffmpeg.
3. Transcription accuracy. Native captions are "free, instant, accurate-ish" - auto-generated ones can be wrong. Whisper is more accurate but costs money and time. Videos with no caption track at all (some TikToks, occasional Vimeos, local files) can only go through Whisper; without a key, fall back to --no-whisper for frames only.
4. Whisper key setup. ~/.config/watch/.env is scaffolded at 0600 with commented placeholders for GROQ_API_KEY (preferred) and OPENAI_API_KEY. Use placeholders like sk-xxx in docs; never commit real keys.
5. Dedup edge cases. The threshold (2.0) is deliberately low and measures brightness, not structure, so a one-line code diff or a terminal scrolling one row survives. But two frames with large structural differences but similar brightness could in theory be dropped - if you notice missing detail, --no-dedup keeps everything.

6. Compared to Alternatives

Only verifiable claims, no invented competitors.

/watch is not replacing yt-dlp or Whisper. It is the orchestration layer that adds ffmpeg frame extraction and a token economics (frame budget + dedup + detail modes) on top, then hands Claude a bundle of images and text with timestamps it can consume directly. The differentiation is giving Claude both modalities at once so its answer is grounded in the picture and the audio, not a caption-only guess.

Verdict

claude-video targets an underrated gap. Everyone talks about multimodal models, but the everyday job of letting a coding agent like Claude Code actually "watch" a video has had no systematic answer. 14,204 stars and a #20 weekly rank suggest the point landed - developers really do need an AI that watches a recording to debug, watches a launch to extract updates, watches a competitor to break down structure.

The engineering is deliberately restrained. It does not reinvent downloading or transcription; it bets entirely on the mature yt-dlp + ffmpeg + Whisper stack and concerns itself only with the token economics (frame budget, dedup, detail modes) and the output format Claude can Read. That "don't reinvent the wheel, just fill the gap" stance is why it stays lightweight under MIT with pure Python stdlib.

There are barriers: you manage token cost on long videos, you install ffmpeg/yt-dlp, and caption-less videos need a Whisper key. But once the pipeline runs, what you save is the time you would have spent scrubbing through video yourself.

References

• claude-video GitHub repo: https://github.com/bradautomates/claude-video
• README (the /watch mechanism, install, usage, detail-mode measurements, frame budget table): https://github.com/bradautomates/claude-video/blob/main/README.md
• Agent Skills (the npx skills CLI and supported hosts): https://agentskills.io
• Groq Console (Whisper key): https://console.groq.com/keys
• Star/fork/language/license/created/push data per GitHub API (verified 2026-08-06, 14,204 stars, subject to real-time change)
• Weekly trending rank and gain (#20, +3,359) per GitHub weekly trending (2026-08-06)