memvault panorama: three tracks × three levels of mindfulness
Trilogy Omnibus - Stringing Write, Organize, and Recall into a Map; Complementing the Three Layers of CLT, Event Flows, and Reactive.
After writing the trilogy, I needed a map that put the "Write, Organize, Recall" tracks on the same screen in alignment, rather than repeating details.
But I haven't mentioned it: the three tracks.Behind the DesignThere are three big ideas that cut across the whole picture. There are three cross-cutting rules that have guided my decision-making when I've developed my business. I'm going to wrap this up and tell you all about it at once.
Organizational Overview#
A panoramic view of the three tracks. The solid line is the main flow, and the cross-track dotted lines are the 4 event streams that allow the system to run on its own. 13 group brackets encircle the related components, and each group is followed by a detailed description of the components.
I. Write Track - Conversation into the Vault#
Memory into the vault track#
At the end of a conversation, I'm afraid that the air will suddenly quiet down, and the content of the conversation will disappear if I don't take it all in. But if you don't accept everything, you'll be drowned out by a lot of noise, and you won't be able to find anything that's really useful. This track is designed to solve these two problems.
I liken memory writing to goods entering a warehouse. There are three hurdles at the entrance: the first hurdle is to filter out the "uh-huh, okay" and such nutritious nonsense; the second hurdle is to block suspicious commands to prevent the system from being damaged by strange contents; and the third hurdle is to block out the duplicated information that has already been there before. You can't archive until you've passed all three levels.
There are also two routes to archiving: one is based on semantic similarity, while the other breaks down the relationships between people, events, and objects into separate files. Each record is labeled with its source to ensure that the original conversation can be found later.The complete process of this three-level, two-track archive is written in detail in this article.The
Someone's talking.#
As soon as the conversation hits the ground, the background begins to move.
Three security gates#
The order of the three filters before entering the vault cannot be changed.
Proofreading and Shelving#
The text is written in a standardized way and then compressed into two indexes to be shelved together.
Cover knowledge diagram#
Break down each passage into relationships, note the source, and run parallel to the main line.
Background Track - Learning to Forget#
Learn to forget the track.#
It's not over when the memories go into the warehouse. If the warehouse is all in and all out, in six months it will be so cluttered that you won't be able to find anything. So this track is quiet, but important: it's learning how to "forget".
I think of the process as that of a night shift stocktaker who regularly patrols the warehouse late at night. He has a five-step process: first, he takes stock of the inventory, then he identifies what's hot in the recent past, then he compares and integrates what may be duplicates or contradictions, then he condenses the identified highlights into a summary, and finally he marks the old, unattended data as pending.
I had two helpers on hand: one to nitpick and pull out contradictory memories, and the other to keep a quiet record of my recent interests so the inventory taker could prioritize my areas of concern.The complete process of this night shift operation, and the logic of how it determines whether to stay or go, is in this article.The
It's not until midnight.#
It will only be activated at 4:00 a.m. when all the conditions are in place.
Five Dreams#
The night is like sleep, with five stages to clean up the day's events.
Real or fake proofreader#
In the dream, you will find four more gatekeepers, and you will find contradictions and mistakes.
Interests & Explorations#
Not only facts, but also preferences, gaps, surprises, and how to store old photos.
III. Recall Tracks - Letting the Right Memories Come to the Surface#
Memory is recalled on this track#
Once the memories are stored and organized, the last thing to be solved is "recall" - how to make the right memories come up automatically at the right time. It's no use having a good warehouse if you have to go through the trouble of searching for it every time.
I liken this track to the front of a restaurant. When a customer asks a question, someone first determines what type of question it is: fact checking, looking for a connection, or just chatting? Next, the back of house synchronizes the delivery of goods from all warehouses, and when the goods arrive, they are screened by quality control, eliminating those that don't fit and leaving the most relevant ones behind. Before serving, they go through another security check to make sure the contents are in order.
What's even better is that there's a character in the background who secretly prepares information. He will anticipate what I want to ask next and prepare the relevant memories in advance. As soon as I say something, he can serve it directly to the table, eliminating the whole process.This is the complete logic of the recall, from ordering to serving, and the details are all in this article.The
(dialect) treat sb according to their social status, relationship with them etc#
When visitors enter, they first judge the map and then decide which floors to look for.
quick search#
The inventory is first roughly sifted and then weighted with 11 layers, and the commonly used fingerprints are left for quick pickup.
second best#
The rough picks go to the picker, but he can skip and stop the machine.
Five Tips for Slow Thinking#
Slow thinking before opening the backup five strokes, engaged in the realization of the full picture and then self-check.
Pre-Shipment#
Shaping and sterilizing once more, and secretly preparing the next course in the background.
Four, three tracks through the design layer#
Where does the pulse of the track come from? In fact, there are three layers of mindfulness, and every detail is a manifestation of these mindfulnesses.
Three Layers of Mind#
Cut redundancy, remove complexity, and standardize your home to make it work for you.
The map is complete here; the Write track collects conversations, the Background track organizes them and learns to forget them, and the Read track lets the right memories come to the surface at the right time.
What makes the three tracks work together is: cut redundancy, remove complexity, and standardize. Shades of this principle can be seen in each track.
Of course, no tool can be used forever. Theory, technology, and usage are always evolving, and memvault will continue to grow with time. This is a snapshot of the progress so far - if there is a major revision in the future, we will talk about it in a new article. memvault trilogy, here it is for now.
Organizational Overview#
Panoramic view of the entry-agnostic pipeline on three tracks. 13 group brackets correspond to the component cards underneath; 4 cross-track dashed event flows mark the system's self-driven backbone.
I. Write Track#
Write Track#
The memory write path is provided bySanitizeGate Coordinate and run three lines of defense before persistence:
NoiseFilter: Weed out the uninformative chit-chat.InjectionGuard+Poisoning Detection: Block prompt injection and content poisoning.DedupOp: Use embedding cosine to compare the existing data, if it is similar enough, then just block it out.
After passing, enter theHybrid Indexing: Vector Inventorymemvault.blocks(pgvector 768d), the knowledge map is split intoL0 Triple(subject/predicate/object) write tokg-opsThe Each piece of information carriesProvenance(session_id / turn_index) for easy backtracking, and after first going through theContentNormalizer Unified format to avoid noise polluting the embedding space.
The theoretical basis for this two-track writing comes from the paperDon't Forget to Connect! Improving RAG with Graph-based Reranking(2405.18414). The complete pipeline, the three lines of defense judgments and the KG extraction strategy are documented in the memvault Write Track DetailsThe
Trigger#
Conversations are triggered when they hit the ground, and the background pipeline fire-and-forget.
asyncio.ensure_future() - fire-and-forgetSanitize Gate#
The three sequential gates, non-commutative.
core.noise.NoiseFilter - low-info regex + densitymemvault.security.poisoning - authority/role/markdown/temporal/base64 + Shannon entropyNormalize & Index#
Content normalization + Hybrid dual indexing on the shelf.
libs/text-ops - 4 submodules + preprocess_chineseomlx_bridge.py - 1024d - persistent subprocesssparse_tokenizer - per-service avgdlqdrant_search - Reciprocal Rank FusionKG Construction#
L0 triad extraction + provenance, event-driven parallelism.
source_tracker - trust_scoreII. Background Track#
Background Track#
Offline Organizing byDreamLoop Triggered at 4AM daily, run in thecore/src/modules/memvault/dream/ The five stages of the pipeline:
orient: Captures the current snapshot of the memory hierarchy.signal: PolymerizationMultiSignalEdges(access / cite / co-occur).reflect: EnforcementTemporal ConflictThe detection of bi-temporal contradictions.consolidate: produce a hierarchical summary (TierDigest) and merge semantically identical memories.prune: Mark the candidates to be eliminated according to their attenuation scores.
There's a bypass.KnowledgeLint Error correction through four levels of progressive review (diagram structure → grounding → LLM → three-stage validation), andInterestProfile Track the popularity of my topics and give back to theSurpriseDiscoveryThe first thing you need to do is to proactively tap into memories that are low in interaction but high in potential.
This design partially corresponds toHippoRAG: Neurobiologically Inspired Long-Term Memory for LLMs(2405.14831) integration concepts.The Dream Loop five-stage thresholds, conflict resolution strategies, and KnowledgeLint v2 details are documented in the memvault Background Track DetailsThe
Trigger#
Cronicle scheduling + dual condition gate.
(now - last) > 24h ∧ sessions_since ≥ 5Dream Loop#
OODA-like five stages: Orient → Signal → Reflect → Consolidate → Prune.
MultiSignalEdges - PPR seedKnowledge Lint#
Four levels of progressive auditing, graph structure → truthfulness → LLM → human review.
Interest & Discovery#
Attention Layering + Interest/Gap/Surprise Tapping + tier descending.
surprise_ops - Leiden weighted triggersIII. Read Track#
Read Track#
The entry point to read the path isQueryClassifyIt chooses different strategies depending on the type of question, but the backbone is an 11-stage scoring pipeline:
QueryRouter: Divide the questions into factual, associative, and chatty.ScoringPipeline: Combining BM25, sense retrieval, and knowledge map jumps with parallel scoring, AttnRes intent-dependent weighted merging.CascadeRecall: When a vector search fails to find a vector, it will automatically drill down into the knowledge graph for multi-step inference.Reranker: Reordering using Jina Reranker v3 MLX.OutputFormatter+ReadTimeSanitize: Run another injection check before shipping.
There's another one in the background.SlowThinkerIt listens to the conversation's SSE stream, predicts the next query, and prefetches the candidate block; if it guesses, it saves the cost of a full pipeline.
where the theoretical basis for intent-dependent weighted mergers comes from the paperAttention as a Hint: Detecting Irrelevant Contexts via Attention Weight(2603.15031). The three-way shunt criterion, the 11-stage scoring function, and the SlowThinker prediction model were memvault Read Track Details Inside.
Query Routing#
Intent classification + attention prior + layer routing + HyDE.
Fast Search#
Hybrid recall + 11-stage weighting + intent-tuned blending + Redis cache.
qdrant_search - hybrid + RRFReranking#
Cross-encoder 2-stage fine-tuning + skip + circuit breaker.
Cascade Recall#
Slow-only five moves: L0 → L1 → L2 → PPR → CRAG.
(s, p, o) lookupOutput#
Budget-aware shaping + shipping side sanitize + Slow Thinker bypass.
is_unsafe_for_injection + sanitize_for_injectionFour, three tracks through the design layer#
The three tracks share a common design layer - CLT as the philosophical frame, 4 Event Flows as the self-driven backbone, and Reactive abstraction as the function synthesis base.
Cross-cutting#
CLT triple principle + 4 event flows + Reactive abstraction.
libs/ops-core - RxJS-inspiredThe three tracks of deep dive are scattered in the trilogy. This post brings them together under one panorama, and pulls out the design layers that run through the whole picture: CLT is my design philosophy, 4 Event Flows is the self-driven backbone, and Reactive abstraction is the basis for function synthesis.
The ones that are not written in are mostly parameters that need to be tuned over time - scoring weights, prefetch hit rates, cascade bounds, knowledge graph extraction reliability thresholds. These values are fine-tuned monthly as the data is distributed, and are not considered new components.
Of course, no tool can be used forever. Theory, technology, and tools are all still breaking through, and memvault will continue to iterate along with them. This blog will serve as an inventory of the progress so far - if there are any major revisions in the future, we'll explain them in a new post. memvault trilogy, here it is for now.
Panorama View Tips for AI Agent#
If you're building your own personal memory system / RAG / AI assistant, post this to your AI assistant and ask it to look at your design in terms of its overall architecture to see if it stands up.
Extended Reading#
Trilogy In Depth Chapter + Theoretical Roots of Cross-Track Mindfulness.
| Resources | Why is it important? |
|---|---|
| memvault Write Track - after the conversation ends | The first in-depth version of the trilogy. Three gate (noise / injection + disguise / de-duplication) + Dual write (vector + KG) + Provenance + Content Normalizer 4 submodules |
| memvault Background Track - After saving | Trilogy Part II In-Depth Edition. 12 elements of Dream Loop 5 Stages, Knowledge Lint 4 Stages, Multi-Signal Edges, Temporal Conflict, Tier Digest, Surprise Discovery, and more! |
| memvault Read Track - After Thinking About It | The third in-depth version of the trilogy. Intentional Sorting + Personalized Router + 11-stage scoring + Cascade Recall + Read-Time Sanitize + Slow Thinker prefetching |
| Cognitive Load Theory (Sweller, 1988) | Three tracks sanitize / segment / verify the roots of the design philosophy of the three gates - write a little harder, read the talent is not tired! |
| Enterprise Integration Patterns (Hohpe & Woolf, 2003) | 4 Event Flows' pub-sub / pipe-and-filter design pattern source. memvault builds the EventBus and Subject abstractions on top of this pattern. |
| RxJS Reactive Extensions | Operator / Pipeline / Observable / Subject are four abstraction specific references. memvault doesn't use RxJS directly, but the abstraction level is aligned to it. |