@Pluvio9yte: The problem of constantly getting CAPTCHAs when using an Agent to scrape Douyin comments has finally been completely solved by a skill. This skill can scrape all comments in a Douyin comment section without triggering CAPTCHAs, and is about 2.5 times faster than approaches like browser automation that require manually clicking to expand replies. What it can do 1. Identify pain points, uncover needs, and see what people are complaining ab…

The problem of constant CAPTCHA prompts when using an Agent to scrape Douyin comments has finally been completely solved by a skill. This skill can scrape all comments in a Douyin video’s comment section without triggering CAPTCHAs, and is about 2.5 times faster than approaches such as browser automation that click to expand replies one by one. What it can do👇 1. Identify pain points and needs, see what people are complaining about, and uncover business opportunities 2. If you’re making videos, see what people are asking in the comments on competing videos, then draw on those questions for your new videos Open source🔗:https://github.com/Pluviobyte/douyin-comments… — # Pluviobyte/douyin-comments Source: https://github.com/Pluviobyte/douyin-comments # douyin-comments An Agent Skill for scraping the complete comment section of a single Douyin video. It collects top-level comments, all nested replies, and video details, with an optional 720p video download. The final output is a Full Comments.md file sorted by likes, plus raw JSON. Requests are sent from your own logged-in Chrome page, and signatures are calculated by the page’s built-in SDK. No paid APIs such as TikHub, no Playwright, and no reverse engineering of a_bogus are needed. ## Test results | Video | Collected | Comment count displayed on video | Requests / Time | CAPTCHA | | — | — | — | — | — | | 7-minute AI tools video | 83 top-level + 112 replies = 195 | 195 | 35 / 138 seconds | None | | 46-minute tutorial | 77 top-level + 58 replies = 135 | 135 | 63 / 252 seconds | None | The results for the first video were compared comment by comment against those collected by manually clicking “Expand replies” one at a time, and matched exactly. The manual approach took about 6 minutes. ## How it works Douyin’s web comment API requires risk-control parameters such as a_bogus, msToken, and uifid. If you call the nested-reply endpoint with only device_platform and aid, a text-selection CAPTCHA immediately appears and the request remains pending. This happens because the request is missing the more than 30 common parameters included in the page’s normal requests. So this skill does not assemble parameters itself. Instead, it copies the page’s actual requests: 1. page_hook.js: Intercepts the first comment request sent by the page itself and saves its URL and request headers as a template. 2. page_fetch.js (primary approach): For each subsequent request, replaces only three parameters—aweme_id, cursor, and comment_id—and copies everything else. Requests are sent through XHR within the page, and the page’s SDK adds the signature. 3. page_ui_expand.js (fallback approach): Clicks “Expand N replies” one by one in the comment section, letting the page send the requests itself while the script only intercepts the responses. This approach is slow but does not trigger risk controls. 4. recv.py: A local receiver service. The page POSTs each page of responses to 127.0.0.1 to save them to disk. 5. build_report.py: Merges and deduplicates the data, checks each comment thread to verify that all replies have been collected, then generates Full Comments.md. Requests are spaced 2–4 seconds apart, with an additional pause of 8–15 seconds every 15 requests. ## Dependencies - Claude Code, connected to the Claude in Chrome browser extension (run /chrome in Claude Code) - Chrome logged in to Douyin - Python 3 (standard library only), curl ## Installation sh git clone https://github.com/Pluviobyte/douyin-comments.git ~/.claude/skills/douyin-comments You can also place it under your project’s .claude/skills/ directory. If the target directory already exists, inspect its contents first; do not overwrite it directly. ## Usage Give Claude Code a Douyin video link and ask it to scrape the comments: text Scrape the comments on this video https://www.douyin.com/video/7xxxxxxxxxxxxxxxxxx Three link types are supported: douyin.com/video/, profile links containing modal_id=, and v.douyin.com short links. By default, output is written to .internal/Douyin Comments// in the current project: raw/ # Templates, raw responses for each page, receiver logs comments_top.json # Top-level comments (deduplicated) comments_replies.json # Nested replies (deduplicated) Full Comments.md # Sorted by likes, with replies indented beneath their parent comments Reference Video.mp4 # Optional See SKILL.md for the full workflow. ## Limitations - Does not solve CAPTCHAs for you. If a CAPTCHA appears, the skill pauses so you can complete it manually in Chrome, then continues. - Handles only a single video. It does not collect all of a creator’s videos in bulk, nor does it support Xiaohongshu or Bilibili. - Deleted, collapsed, or author-only comments are not returned by the API, so the total collected may be slightly lower than the comment count displayed on the video. - Videos with over a thousand comments: After paginating through roughly 500 top-level comments, the API starts returning duplicates. Collapsed comments are also unavailable. In testing, 823 / 1067 comments were collected. - Not yet tested: Pagination when a single thread has more than 20 replies. - For personal research and content reference only. Please follow Douyin’s user agreement, limit request frequency, and do not use this for bulk collection or commercial distribution. ## Acknowledgments The overall approach drew on MediaCrawler (https://github.com/NanmiCoder/MediaCrawler): the common parameter set, the uifid request header, collecting top-level comments before fetching replies thread by thread, and spacing out requests. This repository does not use its code.