sharding

Manticore Search 27.1.5 introduces built-in authentication, sharded tables, conversational search, and faster HNSW vector search, along with numerous fixes and improvements across replication, KNN, and more.

This paper addresses the 'Lost in Conversation' problem where LLMs struggle with information revealed across multiple turns. It proposes a scalable sharding pipeline to create multi-turn training data from single-turn QA datasets and uses reinforcement learning with verifiable rewards to train a memory-augmented policy that maintains a compact rolling memory, improving multi-turn reasoning accuracy and generalizing zero-shot to harder tasks.

This article describes how Akvorado, a network flow analysis tool, scales its BMP Routing Information Base (RIB) by implementing sharding to handle tens of millions of routes, improving concurrent updates performance.

Ben Dicken emphasizes that sharding is essential for building scalable databases and architecting data-intensive applications.