Tag
Sidelight is a Chrome side panel browser agent that runs in your real, signed-in session instead of a remote browser, enabling data extraction, form filling, and multi-step task automation without hitting logins and captchas.
I've open-sourced a Claude Code Agent Skill that scrapes the complete comment section of a single Douyin video (including nested replies) by reusing real request templates and the page SDK's signing mechanism within a logged-in Chrome page. It requires no paid APIs or reverse engineering of a_bogus, does not trigger CAPTCHAs, and runs about 2.5 times as fast as browser automation solutions.
Fibr AI's founder shares an experiment testing how AI agents like ChatGPT, Gemini, and Claude perceive their website, finding most sites are built only for humans, and launches a product plus a free checker tool to diagnose AI readability of web pages.
An opinion piece argues that proposed federal and state 'stealth bot' bills targeting AI scraping would fail to stop abusive crawlers while undermining legitimate automated data collection used by journalism, research, and archival work.
Simon Willison publishes a research report evaluating webframe, Jeff Lindsay's lightweight Go tool for running web pages like programs in a headless system web view, finding it fast and comparable to shot-scraper but with rough edges like CSP failures and missing HTTP metadata.
Gologin 推出 Cloud Browser，一款面向 AI 代理和网页抓取的云端浏览器服务，提供持久会话、隐身与反机器人保护，以及 150+ 国家的代理支持，可扩展至数千个浏览器实例。
A practitioner explains how their AI research agent went from near-zero errors on 10 curated sites to over 30% failure on 200 real domains, concluding the root cause was the pipeline's inability to distinguish an empty result from a broken fetch. Adding a fetch-level confidence score surfaced more bad rows than any prior prompt tuning.
Meta removed controversial commercial terms that legal experts warned could permit scraping any business account's domains to train its AI models, clarifying the provisions applied only to ads and business products.
Meta removed controversial language from its commercial terms that experts said could have allowed it to scrape news sites to train its AI, following backlash from legal scholars and publishers.
Meta 被指通过更新的商业条款，要求企业账号授权其抓取关联网站内容用于 AI 训练且无法选择退出，法律专家警告这可能帮助 Meta 规避向新闻出版商支付数百万美元的 AI 授权费用。
Meta's updated commercial terms, effective October 30, may grant it permission to crawl and scrape content from websites associated with businesses using its platforms—including news publishers—to train its AI, potentially overriding robots.txt directives and bypassing licensing agreements, though Meta insists it is merely consolidating existing terms.
treg 推出开源免费的 WebAccess 工具，为 AI agent 提供 100% 免费的网页搜索与抓取能力，通过在多个供应商间自动路由请求来保障可用性与成本效率。
Gregor Zunic (@gregpr07) shares a prompt trick to let Grok and Muse agents bypass captchas and anti-bot measures by installing the Browser Use CLI and defaulting to its remote browser.
Meta will begin scraping business accounts' websites for AI training data, with the policy taking effect Oct. 30, raising concerns about data usage and consent.
svpino 分享了一種將 agent 連接到網路的簡單方式：加入 Expand 的 MCP server，讓 agent 在抓取 URL 時自動取得保留原格式（表格、圖示等）的 Markdown 內容，並引用了 Expand Fetch 的發布介紹。
Omar Sanseviero shares commentary on Expand Fetch, a new web scraping and crawling tool that uses two System One vision models (including Iris) to read rendered pages and return clean JSON, aiming to fix agent hallucination caused by misreading flattened page content.
开发者在将自定义 feed reader 产品 sprinklz.io 发布到 Hacker News 后，网站遭遇大量 AI 爬虫机器人流量，导致服务器负载异常飙升，作者通过端点日志分析定位了问题来源。
Internet Law professor Eric Goldman hosted a Reddit AMA on the r/WebScrapingInsider subreddit discussing the US legal landscape of web scraping, covering trespass-to-chattels doctrines, data ownership versus open information access, and his policy concerns that anti-scraping rules could turn permissionless browsing into a gated experience.
Pomerado 是一款开源工具，可在几分钟内将任意网站（如 Google Flights、Zillow、Instacart 及传统门户）转换为 API / MCP，帮助 AI 代理自动构建、托管、认证并维护这些集成。
CrawlForge MCP 是一个开源 MCP 服务器，为 AI 代理提供 31 种网页工具，支持抓取、搜索、爬取、结构化数据提取和隐身浏览，可通过一条 npx 命令集成到 Claude Desktop、Claude Code 或 Cursor。