Audio perception layer for LLM agents, with a memory that grows through use

LLMs handle speech well once you run speech-to-text. They don't hear the rest: a bird outside, a glass breaking two rooms away, a smoke alarm two floors down. I've been working on an experimental open-source framework that aims to close that gap: continuous audio in, event-gated recognition out, a concept memory that grows through use. The whole perception stack runs locally: CLAP, Whisper, Silero VAD, sqlite-vec. The classifier goes through fast-agent, so any LLM provider works (including Ollama-local). https://i.redd.it/6rv9qvv0cfdh1.gif The loop mirrors how hearing roughly works: Gate — energy/novelty/Silero-VAD trim the continuous stream to candidate regions. Most audio never reaches the rest of the system. Fingerprint — CLAP embedding + ~two dozen symbolic features (spectral shape, harmonics, peaks, temporal drift). Recognize — sqlite-vec top-k cosine over learned concepts. Strong match against a calibrated concept fires server-side, no LLM call. Weak/miss surfaces to the agent. Unhandled — anything the agent can't recognize goes into a queue with its fingerprint + specialist guesses, waiting for a teacher to label it. Label → new concept. The agent reasons over three symbolic layers, never raw audio: signal features as JSON, embedding similarity scores against memory, and specialist labels. After enough examples, a concept auto-calibrates a threshold and moves to the server-side fast path; no LLM call for subsequent matches. No training; the pretrained CLAP embedding does the work. Today, it ships with three specialists (Whisper, BirdNET, AST); registry is extensible with more classifiers. Each declares a plain-English contract so an LLM knows how and when to use them. Some potential applications: a robot that reacts to a crash it wasn't expecting, an assistant that hears the oven still beeping, an industrial monitor catching a bearing failure before it happens. More of a fun research project; lightly evaluated on my own recordings, no formal benchmark yet. Feedback welcome! - Repo: https://github.com/es617/audient - Writeup with diagrams and demo: https://es617.dev/2026/07/07/audient-ears.html