Tag
This freeCodeCamp handbook teaches readers how to build a production-grade ETL pipeline in Python using real flood data, covering incremental loading, type coercion, deduplication, and idempotency.
A SQL tutorial for building a logistics and shipment analytics database, covering schema design, sample data, and key business KPIs for supply chain optimization.
LangChain rebuilt its data stack around an AI agent that handles ~40x the request volume of its 3-person data team, enabling self-serve analysis and shifting the team's focus to models, context, and guardrails.
A comprehensive guide for software developers unfamiliar with data tools, covering the data lifecycle, data professions, and the tool landscape, with insights from the author's experience at Deepnote and Metabase.
Prefect announces acquisition of Dagster, combining two leading open-source data orchestration platforms.
Introduces a new coding agent designed to assist with data engineering, analytics, and data science tasks.
A tweet thread from Zach outlines the skills and interview process for senior data engineers at Netflix, including SQL, data pipelines, and system design, while promoting DataExpert.io's mock interview service.
Data engineering has spent years building orchestrators like Airflow and Dagster, and now the same pattern is emerging for AI agents with projects like Agor, Agent Teams, and Omnigent from major companies.
There is a meticulously compiled data engineering interview question bank on GitHub called data-engineering-interview-questions, containing over 2,000 questions covering databases, big data frameworks, cloud platforms, data visualization, and other core areas.
Anthropic shared best practices for implementing self-service data analysis with Claude, achieving 95% automation of business analysis queries with an overall accuracy of about 95%, and detailed the agent analysis tech stack, three main failure modes, and corresponding countermeasures.
databow is a new open-source Rust CLI tool that provides a unified interface for querying any database with an ADBC driver, supporting over 30 databases including PostgreSQL, DuckDB, and Snowflake.
Ingestr is an open-source CLI tool for high-speed data movement between any source and destination, supporting numerous databases, data warehouses, and SaaS applications.
This paper formalizes Autonomous Agentic Data Engineering, where LLMs act as autonomous data engineers to curate and optimize training data for specialized domains, showing a 57.29% improvement in student model performance using GPT-5.2.
Streambed is an open-source CDC engine that streams Postgres WAL changes to Iceberg tables on S3, with a built-in query server using DuckDB that speaks the Postgres wire protocol.
GitHub's Open Source Friday event features Elvis Kahoro from dltHub discussing dlt, an open-source Python library for building data pipelines without complexity.
This paper introduces Autonomous Agentic Data Engineering, a task where LLMs autonomously execute end-to-end data curation pipelines for model specialization, showing significant performance gains (e.g., GPT-5.2 improves a student model by 57.29%).
This article provides a comprehensive step-by-step breakdown of how modern Large Language Models like ChatGPT and Claude are built from scratch, covering data collection, tokenization, transformer architectures, training, alignment, and deployment.
Major improvements to session storage and access for Hermes Agent, saving 20-40% disk space and improving speed.
DuckDB introduces 'Quack', a new client-server protocol that enables DuckDB instances to communicate via HTTP, supporting concurrent writers and remote access while maintaining simplicity and performance.
DataTalksClub is offering a free 9-week data engineering zoomcamp course covering containers, orchestration, data warehousing, analytics, batch and streaming processing.