Tag
Show HN: xarray-sql lets you query Xarray datasets with SQL, pivoting them into tables so you can run queries on engines like DuckDB, Polars, PostgreSQL, BigQuery, and Arrow Flight SQL, with lazy round-tripping back to Xarray.
An introduction to The Data Engineering Handbook (~45k Stars on GitHub), an open-source project led by former Meta/Airbnb data engineer Zach Wilson. It is a one-stop learning roadmap library for data engineering, covering hands-on projects, interview prep, curated book lists, communities, email newsletters, and more.
DuckDB 2.0.0 LTS "Cinnamon Teal" is announced as the biggest feature release yet, shipping a polished Quack client-server protocol, asynchronous I/O, a first-class VARIANT type, agent mode, triggers, and an extensible SQL parser, built from nearly 20,000 commits since DuckDB 1.5.
swyx argues that data engineers building data-intensive applications are in far higher demand than typical "RAG guy" AI engineers, and is recruiting for a mission-focused role via Latent Space, with relocation and visa support offered.
作者用真实工作场景的 ETL 任务（shell 脚本重构与 Python 编排器开发）对比评估 Nemotron 3.5 Bolt 与 Qwen Flash Next，并由 Gemini 作为裁判判定胜负，结果各赢一局。
MotherDuck 的技术博客详细解析 DuckDB 2.0 alpha 版本为何更快，重点介绍了通过独立下载线程池实现的异步 I/O，使 S3 上的 Parquet 查询性能显著提升，并分享了实际基准测试数据。
G2i.ai CEO Gabbe Greenberg announced the company is hiring 50 staff software engineers this month to work with frontier AI labs on evals and training datasets, paying $200–$300/hr remotely. G2i positions itself as an engineering layer for AI training, building RL environments and curated evals for top labs.
Duckherder is a new DuckDB extension that enables distributed query execution across multiple worker nodes using Arrow Flight, aiming to enhance data engineering workflows with parallel processing.
This paper presents an end-to-end framework for automating the construction of a business semantic layer from raw telemetry data using hierarchical LLM inference and RAG abstraction, significantly improving semantic quality and reducing maintenance effort.
The author critiques current AI agent memory systems for following old data engineering patterns, highlighting the risk of gaps in memory and advocating for auditable raw trajectories before layered summaries.
The author critiques web agent demos for failing on real-world URLs due to fetch-layer issues, emphasizing that data engineering is the harder problem than agent reasoning.
A tweet discusses how Databricks has simplified Spark, enabling non-specialists like GTM Engineers to build data pipelines without deep technical knowledge.
This blog post details incremental optimizations to reduce ClickHouse query latency from over 85 seconds to sub-second, including changes to partition keys, join elimination, and aggressive merges.
The article shares lessons learned from operating petabyte-scale ClickHouse clusters for five years, discussing best practices and promoting Tinybird's managed data services.
This paper proposes a neurosymbolic layer for LLMs that improves logical reasoning accuracy by 8.5% without finetuning and reduces token usage by over 50% on long context tasks.
A Twitter user contemplates obtaining a second Spark instance, likely referring to Apache Spark, and asks for persuasion on the decision.
Snowflake's engineering blog details the new Postgres data mirroring feature that pushes changes via a snowflake_cdc extension into Iceberg tables, aiming for resilient, low-lag transactional replication.
TechAscent illustrates how DuckDB's high-performance vectorized SQL engine can now be leveraged from Clojure via tech.ml.dataset (TMD), enabling large out-of-memory joins and 50GB CSV ingestion that compresses to 18GB in about two minutes.
A comprehensive guide for software developers entering the data field, explaining the data tools landscape, key concepts, and workflows from ingestion to visualization.
This freeCodeCamp handbook teaches readers how to build a production-grade ETL pipeline in Python using real flood data, covering incremental loading, type coercion, deduplication, and idempotency.