sharding

BriskDB is an alpha-stage database tool that transforms SQLite files into a sharded system with parallel writes, PostgreSQL compatibility, and embedded Rust/Python APIs, aiming to scale SQLite while maintaining its simplicity.

This paper from Carnegie Mellon researchers shows that giving an LLM judge more compute doesn't fix oversight failures when it must check many requirements in one call. It proposes sharding—dividing requirements into smaller groups handled by separate calls—which improves accuracy, resists presentation-based adversarial attacks, and can make a weaker sharded judge match a more capable holistic judge.

PlanetScale describes how it performs massively parallel backups for sharded Postgres databases by spinning up per-shard EC2 instances, restoring prior backups from object storage, and replaying WAL, achieving petabyte-scale backup speeds over 50 GB/s.

This documentation introduces hijax types, a new feature in JAX that allows defining custom types with their own invariants, tangent types, batching, and sharding behavior, illustrated with an example of quantized arrays.

Manticore Search 27.1.5 introduces built-in authentication, sharded tables, conversational search, and faster HNSW vector search, along with numerous fixes and improvements across replication, KNN, and more.

This paper addresses the 'Lost in Conversation' problem where LLMs struggle with information revealed across multiple turns. It proposes a scalable sharding pipeline to create multi-turn training data from single-turn QA datasets and uses reinforcement learning with verifiable rewards to train a memory-augmented policy that maintains a compact rolling memory, improving multi-turn reasoning accuracy and generalizing zero-shot to harder tasks.

This article describes how Akvorado, a network flow analysis tool, scales its BMP Routing Information Base (RIB) by implementing sharding to handle tens of millions of routes, improving concurrent updates performance.

Ben Dicken emphasizes that sharding is essential for building scalable databases and architecting data-intensive applications.