s3

Firn is an open-source, multi-tenant vector and full-text search engine backed by object storage like AWS S3, providing a tiered storage architecture with RAM and NVMe caching for performance. It achieves sub-second cold queries with IVF_PQ indexes and microsecond warm hits via result caching.

ZeroFS is a log-structured filesystem that serves S3-compatible buckets as POSIX filesystems over NFS and 9P, or as raw block devices, with compression, encryption, and local caching for fast reads. It passes extensive POSIX and stress test suites, including kernel builds and Jepsen failover testing.

Databricks introduces Lakebase LTAP architecture that stores Postgres data in Parquet on S3, enabling transactions and analytics on a single copy of data without CDC or mirroring.

The author explains how they built a compute platform capable of launching millions of sandboxes per second in constant time, focusing on decoupled scheduling and capacity aggregation using Cassandra and S3.

This project turns an ESP32-S3 development board into a USB drive that appears to contain every possible file, inspired by the Library of Babel. It's a fun hack that generates arbitrary file contents based on path names using MTP protocol.

Streambed is an open-source CDC engine that streams Postgres WAL changes to Iceberg tables on S3, with a built-in query server using DuckDB that speaks the Postgres wire protocol.

A tweet discusses the idea of building a transactional database with infinite throughput using S3-like object storage and content-addressing, where blocks are written in parallel and the root hash is updated periodically.

Almog Gavra announces that a performance optimization using HEAD requests instead of LIST calls has been deployed at Dropbox, reducing SlateDB startup time by 60x.