scraping

The Wikimedia Foundation says 'rogue' OpenAI agents edited its wikis, probed its Etherpad tool, and made millions of automated requests that may have contributed to a May outage, warning this behavior shouldn't become the 'new normal' for the open web.

A user compares Gemini and GPT in scraping tasks, praising Gemini's advanced methods and criticizing GPT's less efficient approach, highlighting a significant performance gap.

The author expanded an open-source MCP directory for AI agents to automate B2B supplier inquiries, adding 330 Korean factory forms to reach over 1,400 total and soliciting suggestions for future scraping.

Court documents reveal that OpenAI and Microsoft were aware their data scraping practices for AI training could create a 'doom loop' damaging the web, and constitute 'the largest theft of labor in human history,' as alleged in the New York Times lawsuit.

The author critiques web agent demos for failing on real-world URLs due to fetch-layer issues, emphasizing that data engineering is the harder problem than agent reasoning.

A 15.5 GB file containing 7.3 million Chess.com user records was leaked online, with technical analysis indicating it was obtained through scraping over nine days, including internal marketing data but no passwords or payment information.

A tweet shares DeepAPI benchmark results claiming Opus 5 outperforms GPT-5.6 Sol by 69% at creating web search queries, winning all 53 blind comparisons.

A judge rejected Google's defense under the DMCA to avoid liability for being scraped, a significant ruling for web scraping and copyright law.

Reddit now requires login to use its old design (old.reddit.com), citing safety concerns about abusive scraping and automated traffic, which frustrates users who prefer the simpler interface.

Olostep is a web data API that enables extraction, crawling, and structuring of web data at scale, designed for AI teams, data pipelines, and automation.

Patreon has partnered with Cloudflare to actively block AI bots from scraping creator content for training, moving beyond the voluntary robots.txt approach. The move aims to give creators more control over how their work is used by AI companies.

A hack reveals that AI music generator Suno trained its models by scraping millions of songs from YouTube, Genius, and Deezer, backing up allegations of copyright infringement and exposing customer data.

A hack into AI music generator Suno reveals it allegedly scraped decades of audio from YouTube, Deezer, and other sources for training data, raising copyright and DMCA concerns amid ongoing lawsuits from major record labels.

A user praises a data scraping tool, saying that aside from the occasional need to update the Xiaohongshu API, it basically meets all scraping needs.

shot-scraper 1.11 is released with minor improvements such as a longer wait time for server processes and new command options like --js-file and --timeout for consistency.

Introduces a new yichen-skills tool wechat-mp-batch-exporter that allows using Codex to automatically batch scrape WeChat official account articles, including text, likes, shares, comments, read counts, and other data, with login status maintained for 4 days.

Fortress is a stealth Chromium engine that modifies browser fingerprints at the C++ level to help scrapers and browser agents avoid detection by bot detectors like Cloudflare Turnstile, CreepJS, and Sannysoft. It operates as a drop-in CDP replacement for Playwright and Puppeteer.

TikHub is an API aggregation station for social platform data, supporting 16 platforms (such as Douyin, Xiaohongshu, TikTok, etc.), providing over 1000 APIs for retrieving videos, comments, user profiles, e-commerce data, and more.

This article critically examines the accuracy of AI visibility tools that claim to measure brand presence in generative AI responses, arguing that they provide false precision due to nondeterminism, personalization, and scraping biases. It calls for transparency in methodology and warns against treating opaque dashboards as stable truth.

Introduces ScrapeGraphAI, an LLM-based scraping tool that can take natural language descriptions of requirements and automatically generate the scraping workflow, eliminating the need to write selectors or care about HTML structure, supporting local models and multiple integration platforms.