A team from Nanyang Technological University, NUS, Tsinghua and CUHK has released VoiceMem, a fast memory framework for real-time voice assistants. VoiceMem keeps facts and an emotional profile of the user in two parallel graphs and manages to retrieve what it needs while the user is still finishing a sentence, in 134 ms to be exact. Here is how it works: a speech model cannot hold an entire conversation history in its context, so after every session a separate LLM pulls short statements out of it and files them into a database, one record per statement: “I’m a vegetarian”, “I’m allergic to nuts”, “my sister’s name is Anya”. When the user asks a new question, several matching records are retrieved and appended to the prompt as plain text, right before the question itself.
On LoCoMo, a long-term memory benchmark, VoiceMem answers correctly in 91.2% of cases while retrieving five records that add up to 430 tokens. Mem0, which runs underneath it as the storage backend, scores 61.68% on the same questions and spends 6,956 tokens; EverMemOS reaches 83.13% at 1,899 tokens. These records are inserted into the prompt again on every turn, so their combined length determines both the price of a request and how much context room is left for the conversation itself. Retrieval takes 134 ms against 1,440 ms for Mem0 and fits entirely inside the pause that a voice activity detector (VAD) waits out anyway. The project is open under Apache 2.0: the code, fine-tuning setup and evaluation scripts are on GitHub, the weights and auxiliary models are on Hugging Face, and the ChatMem-400K training dataset is published separately. Demos and video are available on the project page.
Why voice needs its own memory
Ordinary memory engines for text agents are built for a different operating mode. First, they return up to 100 records at a time, which is more context than a speech model can digest. Second, retrieval takes them 2–3 seconds, whereas the pause between turns allows only 100–200 ms before the rhythm of the dialogue breaks down. The authors state the task as follows: hold accuracy at a budget of 5 records and add no audible delay. On top of that, a voice assistant needs to remember not only facts but also how the user feels about people and events, and that is a separate kind of memory with its own update logic.
Left brain: schema, entity, record
Instead of searching the whole database, VoiceMem builds a lightweight two-level index on top of the store. The upper level holds schemas (coarse semantic slots such as work, health, daily_life), the lower one holds entities (specific people, events, concepts). Each entity belongs to exactly one schema and keeps pointers to records in the backend. While the user is speaking, a streaming matcher works from the partial transcript to find relevant schemas and entities, expands them with one-hop neighbours along strong and weak links, and ends up with a compact candidate pool. The search runs only over that pool.
This yields an important property: latency does not depend on K. Retrieval stays around 134 ms at K=3 and at K=100 alike, because schema routing bounds the candidate pool before ranking begins. Schemas are not fixed once and for all: if a group of entities is regularly retrieved together, an LLM judge checks it for relevance, importance and completeness, and it is promoted into a new cluster.
Right brain: who the user is
While the left side records what happened, the right side describes who is in front of the model. There are two types of nodes here. Independent nodes hold stable traits (“he often feels lonely”), while cross-entity nodes are tied to a specific object in the left graph (“he dislikes the weekly meeting with Bob”). The separation matters: merging them would either turn a situational reaction into a permanent character trait or strip an emotion of its cause.
The right graph is updated over two horizons. Within a turn, an affect estimator captures the current emotion together with its target; after a session, long-horizon attribution reviews the whole sequence and keeps only recurring patterns rather than every fleeting state.
Four stages inside the pause
Retrieval is broken into stages and starts before the turn ends. While the person is speaking, streaming recognition, entity detection and emotion detection run. At 200 ms of silence, the query embedding is computed and both graphs are expanded. At 400–500 ms, only the backend call is left. The VAD threshold is usually set at 500 ms, so there is a 400 ms window before the reply has to start, and a dense search across both graphs costs 134 ms.
Results
Across the factual benchmarks (LoCoMo, LongMemEval, Memora), VoiceMem averages 76.39. That is 24.12 points above Mem0 and 15.90 above feeding in the entire dialogue history. The margin is widest where an answer requires several memories at once: on temporal reasoning the gap reaches 54.9 points. On persona benchmarks the average is 74.16 with GPT-4o-mini as the responding model, 1.89 points above the previous leader MemOS, and 76.56 with the fine-tuned model.
The authors also assembled ChatMem-Bench: 316 questions drawn from 15,314 turns and 53 hours of audio. VoiceMem leads in 11 of the 14 categories, averaging 68.73 against 53.95 for MemOS. The widest gap is in paralinguistics and acoustic environment, where a transcript simply carries none of the needed information: text-based approaches score between 3.23 and 26.92, VoiceMem between 45.16 and 53.84.
What each mechanism contributes
Ablations across four datasets show that every component pulls its weight. The biggest loss comes from removing the upper-layer index (minus 9.9 on LoCoMo), followed by the right brain (minus 6.3), meaning that affect and persona carry information the factual store does not. Raising the budget adds almost nothing: going from K=5 to K=100 gains only 2.3 points for eight times the tokens.
The resulting structure of the store is worth a look too. The left graph holds 510 records across six preset slots, plus two more that emerged on their own and cover 254 records, nearly half the store. Neither of them fits inside a preset slot: “Pets & Outdoor” draws 48% of its records from daily_life, 27% from health and 17% from relationships. The index is not tied to a particular store: with the default Mem0 the gain is 29.52 points, with Zep 22.92, with LangMem 15.76.
Training and limitations
To make speech models capable of taking memory as input at all, the authors fine-tuned Qwen2.5-Omni, Qwen3-Omni and Step-Audio2-Mini through online distillation from proprietary teachers. The data was generated in a loop: building a synthetic user’s memory world, generating memory-dependent dialogues, filtering, and manual refinement. That produced ChatMem-400K.
Some of the weak points the authors name themselves. A detailed description of ChatMem-Bench is deferred to a separate technical report, which makes it harder to reproduce than a LoCoMo run. Backend quality still bounds the result: LangMem and Mem0 start within 5.50 points of each other but finish 19.26 apart. It is also worth keeping in mind that part of the gain on the acoustic categories comes from more than a good architecture: competitors work from a transcript, which contains no audio evidence at all.
