Tag
UniBrowse introduces a unified data pipeline for multimodal BrowseComp tasks, generating training data covering three information-flow patterns and achieving state-of-the-art performance on five benchmarks, surpassing GPT-5 and Gemini models.
A practical guide for building structured agent trajectory datasets for training tool-using agents, emphasizing the importance of designing trajectories with six key parts and treating them as data assets rather than logs.
Crawl4AI is an LLM-optimized open-source crawler that converts web content into clean, structured Markdown. It supports intelligent content filtering, LLM-driven extraction, browser automation, and is ideal for RAG and AI Agent scenarios.
Crawl4AI is an open-source web crawler tool that converts web content into clean Markdown format, designed for LLM's RAG, Agents, and data pipelines. Zero API Key, fast output of structured data.
Photoroom details their data strategy for training PRX, including assembling diverse datasets, re-captioning with a VLM, and using Mosaic Data Shards for efficient training.
The author discusses the architecture for scaling AI memory ingestion from thousands to millions of documents, emphasizing orchestration and parallelism over raw compute power, and using Prefect for workflow management.
OpenZL is a compression library that generates specialized compressors for specific data formats, achieving high compression ratios at high speeds suitable for datacenter workloads like AI processing.
The author proposes replacing traditional chunks with IdeaBlocks (Question-Answer + governance fields) to improve RAG knowledge units. The Blockify tool has been open-sourced, which can reduce corpus size by 40x, tokens by 3x, and increase relevance by 2.3x.
Robotics teams are rebuilding the data stack from scratch to overcome the 'data layer tax' that slows down iteration and scaling in robot learning, as existing infrastructure doesn't handle multi-rate and multimodal data.
ProfiLLM introduces an agentic LLM pipeline that generates utility-aligned user profiles from platform-scale behavioral logs for industrial ride-hailing dispatch, achieving significant improvements in outcome prediction and GMV in production at DiDi.
Artie is a self-serve product for real-time data replication to data warehouses, offering sub-minute latency, exactly-once delivery, and easy deployment without managing Kafka or DMS infrastructure.
TinyFish Bigset is an open-source multi-agent system that turns natural language prompts into structured datasets from the live web, with schema inference, autonomous research agents, and scheduled refresh. It runs self-hosted via Docker and is built on TinyFish's search infrastructure.
DataChain is a Python library that adds a context layer to unstructured files in S3, GCS, and Azure, turning them into versionable, queryable typed datasets with support for parallel processing, incremental updates, and agent workflow integration.
SmartDirector is a framework that enhances video generation by using multiple keyframes to improve narrative structure and temporal pacing, operating in a two-stage process of low-resolution generation and high-resolution refinement.
Norway's National Library is building a sovereign Norwegian LLM using 2 PB of Huawei OceanStor Dorado flash storage for its AI training data pipeline, addressing the need for a local language model.
Snowflake now supports job-based batch inference powered by Ray, enabling distributed GPU execution for scaling model inference over millions of unstructured datapoints with a single API call.
The article discusses the importance of quality control for reinforcement learning data, outlining the shortcomings of current data vendors and the evaluation criteria used by frontier AI labs for RL data.
Anyscale is hosting a hands-on virtual lab session teaching developers how to build and scale data pipelines with Ray, covering video data curation, distributed GPU inference, and CPU/GPU streaming pipelines.
Prefect is an open-source workflow orchestration framework for building and automating data pipelines in Python.