Tag
ProfiLLM introduces an agentic LLM pipeline that generates utility-aligned user profiles from platform-scale behavioral logs for industrial ride-hailing dispatch, achieving significant improvements in outcome prediction and GMV in production at DiDi.
Artie is a self-serve product for real-time data replication to data warehouses, offering sub-minute latency, exactly-once delivery, and easy deployment without managing Kafka or DMS infrastructure.
TinyFish Bigset is an open-source multi-agent system that turns natural language prompts into structured datasets from the live web, with schema inference, autonomous research agents, and scheduled refresh. It runs self-hosted via Docker and is built on TinyFish's search infrastructure.
DataChain is a Python library that adds a context layer to unstructured files in S3, GCS, and Azure, turning them into versionable, queryable typed datasets with support for parallel processing, incremental updates, and agent workflow integration.
SmartDirector is a framework that enhances video generation by using multiple keyframes to improve narrative structure and temporal pacing, operating in a two-stage process of low-resolution generation and high-resolution refinement.
Norway's National Library is building a sovereign Norwegian LLM using 2 PB of Huawei OceanStor Dorado flash storage for its AI training data pipeline, addressing the need for a local language model.
Snowflake now supports job-based batch inference powered by Ray, enabling distributed GPU execution for scaling model inference over millions of unstructured datapoints with a single API call.
The article discusses the importance of quality control for reinforcement learning data, outlining the shortcomings of current data vendors and the evaluation criteria used by frontier AI labs for RL data.
Anyscale is hosting a hands-on virtual lab session teaching developers how to build and scale data pipelines with Ray, covering video data curation, distributed GPU inference, and CPU/GPU streaming pipelines.
freehire 是一个开源 IT 职位聚合工具，直接从 47 万家公司的 ATS 系统抓取 330 万+实时职位，去重后提供分面全文搜索、HTTP API 和 CLI，MIT 许可、可自托管。
Prefect is an open-source workflow orchestration framework for building and automating data pipelines in Python.