Tag
A tweet discusses how Databricks has simplified Spark, enabling non-specialists like GTM Engineers to build data pipelines without deep technical knowledge.
This blog post details incremental optimizations to reduce ClickHouse query latency from over 85 seconds to sub-second, including changes to partition keys, join elimination, and aggressive merges.
The article shares lessons learned from operating petabyte-scale ClickHouse clusters for five years, discussing best practices and promoting Tinybird's managed data services.
This paper proposes a neurosymbolic layer for LLMs that improves logical reasoning accuracy by 8.5% without finetuning and reduces token usage by over 50% on long context tasks.
A Twitter user contemplates obtaining a second Spark instance, likely referring to Apache Spark, and asks for persuasion on the decision.
Snowflake's engineering blog details the new Postgres data mirroring feature that pushes changes via a snowflake_cdc extension into Iceberg tables, aiming for resilient, low-lag transactional replication.
TechAscent illustrates how DuckDB's high-performance vectorized SQL engine can now be leveraged from Clojure via tech.ml.dataset (TMD), enabling large out-of-memory joins and 50GB CSV ingestion that compresses to 18GB in about two minutes.
A comprehensive guide for software developers entering the data field, explaining the data tools landscape, key concepts, and workflows from ingestion to visualization.
This freeCodeCamp handbook teaches readers how to build a production-grade ETL pipeline in Python using real flood data, covering incremental loading, type coercion, deduplication, and idempotency.
A SQL tutorial for building a logistics and shipment analytics database, covering schema design, sample data, and key business KPIs for supply chain optimization.
LangChain rebuilt its data stack around an AI agent that handles ~40x the request volume of its 3-person data team, enabling self-serve analysis and shifting the team's focus to models, context, and guardrails.
A comprehensive guide for software developers unfamiliar with data tools, covering the data lifecycle, data professions, and the tool landscape, with insights from the author's experience at Deepnote and Metabase.
Prefect announces acquisition of Dagster, combining two leading open-source data orchestration platforms.
Introduces a new coding agent designed to assist with data engineering, analytics, and data science tasks.
A tweet thread from Zach outlines the skills and interview process for senior data engineers at Netflix, including SQL, data pipelines, and system design, while promoting DataExpert.io's mock interview service.
Data engineering has spent years building orchestrators like Airflow and Dagster, and now the same pattern is emerging for AI agents with projects like Agor, Agent Teams, and Omnigent from major companies.
There is a meticulously compiled data engineering interview question bank on GitHub called data-engineering-interview-questions, containing over 2,000 questions covering databases, big data frameworks, cloud platforms, data visualization, and other core areas.
Anthropic shared best practices for implementing self-service data analysis with Claude, achieving 95% automation of business analysis queries with an overall accuracy of about 95%, and detailed the agent analysis tech stack, three main failure modes, and corresponding countermeasures.
databow is a new open-source Rust CLI tool that provides a unified interface for querying any database with an ADBC driver, supporting over 30 databases including PostgreSQL, DuckDB, and Snowflake.
Ingestr is an open-source CLI tool for high-speed data movement between any source and destination, supporting numerous databases, data warehouses, and SaaS applications.