We’ve released lkv, a new embedded DB implementation in Rust. It’s a lightweight KVS specialized for Point Lookup, achieving high read performance by significantly limiting its features. Compared to SQLite’s default settings, it delivers about 165 times the speed, and about 15 times the speed compared to LMDB. ■Repository https://github.com/nuskey8/lkv

nuskey8/lkv

Source: https://github.com/nuskey8/lkv

lkv

A lightweight and fast embedded key-value store for Rust.

Overview

lkv is a lightweight and fast embedded database implemented in Rust. It is designed specifically for read performance based on hash tables and memory efficiency, featuring faster lookups than LMDB, sled, redb, and others.

To maintain structural simplicity and read performance, lkv comes with a very limited set of features. Here is what lkv supports:

• Writing and reading arbitrary byte sequences [u8] as Key/Value
• Fast, unordered scans
• Fast and zero-copy lookups
• Transactions
• Snapshots
• Explicit compaction

On the other hand, the following features are not supported:

• Reading and writing from multiple processes
• Multiple writers
• Advanced queries such as ranged or prefix queries
• Automatic compaction

Due to its performance characteristics, it is suitable for lightweight configuration persistence or managing master data that rarely changes. Conversely, using it as a general-purpose database with frequent updates is not recommended.

Much of lkv’s design is inspired by LinkedIn’s PalDB and Bitcask. In addition, some ideas are based on FASTER.

For details, refer to docs/design.md.

Installation

cargo add lkv

Quick Start

use lkv::{Database, Result};

fn main() -> Result<()> {
    let mut db = Database::create("./example.lkv")?;

    let mut write = db.begin_write()?;
    write.put("name", "lkv")?;
    write.commit()?;

    let read = db.begin_read()?;
    assert_eq!(read.get("name")?, Some(b"lkv".as_slice()));
    for item in read.iter()? {
        let (key, value) = item?;
        println!("{} = {}",
            String::from_utf8_lossy(key),
            String::from_utf8_lossy(value));
    }
    drop(read);

    Ok(())
}

Snapshots

When writing to the database using WriteTransaction in lkv, you cannot perform reads at the same time. If you want to read from the database during a write, you can create a snapshot with snapshot() and read through it.

let snapshot = db.snapshot()?;
let old_value = snapshot.get("key")?;
for item in snapshot.iter()? {
    let (key, value) = item?;
}

Compaction

Updates to lkv are appended to the Overlay located behind the Base. As updates and deletions increase, old records remain within the file, and the Overlay index also consumes memory.

compact() rebuilds the Base with valid Key/Values, deletes the Overlay and older generations, and physically shrinks the database file.

// Ensure no snapshots are referenced during compaction
drop(snapshot);
db.compact()?;

Compaction is not performed automatically. If the Overlay exceeds the configured memory limit, new WriteTransactions will return Error::MaintenanceRequired.

During compaction, a backup Base is constructed at the end of the same file, so additional disk space is required until completion. In the event of a crash, recovery is possible from the synchronized Base upon reopening, though the file reduction may not be complete. In such cases, compact() can be run again.

In-Memory

It is also possible to use lkv as an in-memory database.

let mut db = Database::memory();

The API is identical to that of a regular database, but it operates in memory without creating a file.

Benchmark

The benchmarks were conducted using a MacBook Pro with an Apple M2 chip and 24 GB of RAM.

License

MIT