Witchcraft, fast local semantic search on top of SQLite [P]

Reddit r/MachineLearning Tools

Summary

Witchcraft is an open-source re-implementation of Stanford's XTR-Warp semantic search engine in Rust, using SQLite for fast local search. It includes Pickbrain CLI for indexing code session transcripts and equipping AI agents with global memory.

**Witchcraft (https://github.com/dropbox/witchcraft)**, an open source project that I built at Dropbox, is a from-scratch re-implementation of Stanford's XTR-Warp semantic search engine ( [https://github.com/jlscheerer/xtr-warp](https://github.com/jlscheerer/xtr-warp) ) in safe rust, using a single-file SQLite database as backing storage, making it suitable for client-side deployment. It runs completely stand-alone on your device, needs no API keys, no vector database, no chunking strategy, no fancy re-rankers, and it is lightning fast (20ms p.95 end-to-end search latency on NFCorpus, at 33% NDCG@10, on an Apple Macbook Pro M2 Max, more than twice as fast as the original XTR-WARP on server-class hardware, at similar accuracy.) The project also includes **Pickbrain**, a CLI that indexes your Claude Code and OpenAI Codex session transcripts, memory files, and authored documents into a Witchcraft database for fast semantic search. Ever wondered "what was that conversation where I fixed the auth middleware?" — pickbrain finds it, and lets you resume the session directly. There is also a /pickbrain skill for both Claude and Codex, which equips those tools with global memory across all sessions. You can use pickbrain directly from the command line, e.g., to rediscover a previous agent session and directly resume it, or you can have your agent invoke it via the supplied skill, e.g.,. "use /pickbrain to read up on our previous efforts on training with XTR token masking", to easily populate a new session with previous context.
Original Article

Similar Articles

Rethinking Search as Code Generation (25 minute read)

TLDR AI

Perplexity introduces Search as Code (SaC), a new architecture that atomizes search primitives for AI agents to compose via code, moving beyond traditional monolithic search pipelines to enable fine-grained control over retrieval.

I built a rust database for agent traces (sub-ms p95 at 1B rows)

Reddit r/AI_Agents

ZenithDB is a new open-source Rust database designed for storing and querying AI agent traces. It achieves sub-millisecond trace fetch latency at 1 billion rows by colocating all spans of a trace during compaction, and includes embedded full-text search and late materialization.

Sage launched! Local AI inference engine

Reddit r/AI_Agents

Sage is a local AI inference engine built in Rust that runs entirely on the user's machine with kernel-level sandbox security, no telemetry, and no cloud dependency.