Tag
The author shares frustration with duct-taping data prep tools for AI agents and proposes a simpler approach: upload raw files, describe the desired output in plain English, and receive cleaned, structured data ready for agents.
This book provides a practical guide to using BigQuery for data exploration and preparation, covering cleaning, transforming, and analyzing data for business insights.
DataPrep-Bench is a unified benchmark evaluating LLMs' capabilities in training data construction and quality evaluation across six domains, including a skill-guided agent (Data-Construction-Skill) and a distribution-based evaluator (DAS) that achieves strong cross-model correlation.
OmniParse is a local platform that ingests and parses unstructured data (documents, images, video, audio, web) into structured JSON optimized for LLM applications like RAG and fine-tuning.
The author shares that they find AI agents useful for repetitive data prep work, specifically using Pandada to clean and standardize raw files, which reduces manual effort and mistakes.
DataFlow is an open-source tool with visual, low-code pipelines to generate, clean, and prepare high-quality LLM training datasets from raw data. It includes a technical report on arXiv.
DataFlow is an LLM-driven framework for automated data preparation and workflow engineering, featuring nearly 200 reusable operators and six domain-general pipelines that improve LLM performance across tasks like math, code, and Text-to-SQL.