MemVault Overview: Three Tracks × Three Core Principles
Trilogy Compilation — Weaving “Writing,” “Organizing,” and “Retrieving” into a Map; Supplementing with the Three Core Principles of CLT, Event Flows, and Reactive Programming.
After finishing the trilogy, I need a map that aligns the three tracks—“writing, organizing, and recalling”—on the same canvas, rather than repeating the details.
But there’s one thing I haven’t mentioned: the design context behind these three tracks. There are three core principles that run throughout the entire project, and they served as my decision-making guide during development. In this concluding post, I’ll explain them all in detail.
Architecture Overview
A panoramic view of the three tracks: The solid lines represent the main workflow, the dashed lines crossing the tracks represent the four event streams that run autonomously, and the 13 group brackets enclose related components, with a detailed description of each component following its respective group.
I. Getting on Track — Putting the Dialogue in the Vault
"Memories in the Vault" track
The write path—the process of saving memories—is metaphorically like cargo entering a warehouse, guarded by three checkpoints at the entrance: the first filters out nutritional-less nonsense like "uh-huh, okay"; the second blocks suspicious instructions to prevent the system from getting corrupted by strange content; and the third stops duplicate information that is already on file. Archiving only happens once all three gates are passed. These three gates are set up because accepting everything would drown everything in noise, making it impossible to find what is actually useful, while accepting nothing risks having the recent conversation vanish into thin air.
Archiving is divided into two approaches: one organizes records based on semantic similarity, while the other breaks down the relationships between people, events, and objects and archives them separately. Each record is labeled with its source to ensure that the original conversation can be located later. This article details the entire process, which consists of three checkpoints and a dual-track filing system..
Someone spoke up.
As soon as the dialogue began, the background started to move.
Three security checkpoints
The three-step screening process before entering the vault must be followed in the specified order.
Proofreading and Publishing
Standardize the text formatting, then compile it into two indexes and shelve them together.
Cover Knowledge Graph
Break each paragraph down into its key points, note the sources, and work on them in parallel with the main storyline.
II. Background Track — Learning to Forget
Learn to Let Go of This Track
I think of the background track as a night-shift inventory clerk teaching the vault how to "forget": making routine late-night patrols in five steps—first taking stock of the current inventory, then identifying recently popular information, next comparing and integrating potentially overlapping or contradictory content, then distilling the verified essence into a summary, and finally flagging neglected old data as pending. Without this track, the warehouse would only take in without ever releasing, and after half a year, it would become so cluttered that nothing could be found.
I also have two assistants by my side: one specializes in nitpicking and pointing out contradictory memories; the other quietly keeps track of my recent interests so that the inventory taker can prioritize the areas I care about. This article covers the complete process for night shift operations, as well as the decision-making logic behind determining whether to stay or leave..
Didn't move until the middle of the night
It will only start up at 4 a.m. when all conditions are met.
Five-Part Dream
At night, just like when you sleep, the five stages help you process and clear your mind of the day’s events.
Real and Fake Proofreaders
In my dreams, I go through four more rounds of scrutiny to weed out contradictions and errors.
Interests and Exploration
Not just facts, but also preferences, gaps, surprises, and how to store old photos.
III. The Path of Recall — Bringing the Right Memories to the Surface
"Memories Are Recalled"—This Track
I compare the retrieval track to a restaurant’s front desk: when a customer asks a question, someone first determines the type of question (checking facts, finding connections, or just chatting). Then, the back kitchen simultaneously retrieves items from various storage areas. Once the goods arrive, quality control screens and discards anything unsuitable, and right before serving, a final security check ensures the content is problem-free. Stored and organized memories finally rely on this stage to automatically surface at the right time, saving the trouble of laborious searching every single time.
What’s even more amazing is that there’s a “prep team” in the kitchen working quietly in the background: they anticipate what I’m going to ask next, prepare the relevant memories in advance, and as soon as I open my mouth, they can serve them right up, skipping the whole process. This article covers all the details of the complete recall process, from order placement to serving..
Tailor one's approach to the person
As soon as a visitor walks in, we first assess their intent, then decide which floors to check.
Quick Search
Upon arrival at the warehouse, the items undergo an initial screening, followed by sorting using an 11-tier weighted system; frequently used items are set aside in a quick-access bin.
Carefully Selected a Second Time
After the items have been roughly sorted, they are handed over to the final sorter, but he can skip them or stop the machine.
Five Tips for Reflective Thinking
Five Strategies for Thinking Things Through: From the Facts to the Big Picture, Then Self-Reflection.
Before shipment
Rearrange and sanitize again; in the background, they’re secretly preparing the next course.
IV. Design Layer Penetrated by Three Tracks
The three tracks—writing, context, and recall—are all guided by the same set of principles: cut out the excess, break down complexity, and refine examples. Every detail is, in fact, a manifestation of these three principles.
The Three-Tiered Method
Cut out the excess, simplify the complex parts, and refine the examples—so your home can run on its own.
This completes the map. The “Write” track is responsible for storing conversations; the “Background” track organizes them while learning to let go; and the “Read” track allows the right memories to surface at the right time.
What ties these three tracks together is: cutting out the excess, simplifying complexity, and refining examples. Traces of this set of principles can be seen in each track.
Of course, no tool is a silver bullet. Theories, technologies, and usages are always advancing, and memvault will continue to grow over time. Consider this post as a snapshot of its current progress—if there is a major overhaul in the future, I'll cover it in a new article. The memvault trilogy rests here for now.
Architecture Overview
13 group brackets correspond to the component cards beneath this entry-agnostic pipeline overview, and 4 cross-track dashed event flows mark the system's self-driving backbone.
I. Write Track
Write Track
The memory write path is coordinated by SanitizeGate, which runs three lines of defense before persistence:
NoiseFilter: Eliminate small talk that lacks substance.InjectionGuard+Poisoning Detection: Prevents prompt injection and content poisoning.DedupOp: Compare the new data against existing data using embedding cosine similarity; if the similarity is high enough, block it immediately.
Once approved, proceed to Hybrid Indexing: vectors go into memvault.blocks (pgvector 768d), while the knowledge graph is broken down into L0 Triple (subject/predicate/object) and written into kg-ops. Each piece of data carries Provenance (session_id / turn_index) for easy tracing, and first passes through ContentNormalizer, which standardizes the format to avoid noise contamination in the embedding space.
The theoretical basis for dual-track writing comes from a research paper Don't Forget to Connect! Improving RAG with Graph-Based Reranking(2405.18414). The complete pipeline, the three-line defense criteria, and the KG extraction strategy are all documented in MemVault Write Track: A Detailed Look.
Trigger
Conversations are triggered as soon as they land, with a "fire-and-forget" background pipeline.
asyncio.ensure_future() · fire-and-forgetSanitize Gate
Three sequential gates, non-commutative.
core.noise.NoiseFilter · low-info regex + densitymemvault.security.poisoning · authority/role/markdown/temporal/base64 + Shannon entropyNormalize & Index
Content Normalization + Hybrid Dual-Index Listing.
libs/text-ops · 4 submodules + preprocess_chineseomlx_bridge.py · 1024d · persistent subprocesssparse_tokenizer · Average delay per serviceqdrant_search · Reciprocal Rank FusionKG Construction
L0: Triplet extraction + provenance, event-driven and parallel.
source_tracker · trust_scoreII. Background Track
Background Track
Offline consolidation is triggered daily at 4 a.m. by DreamLoop, running the five-stage pipeline in core/src/modules/memvault/dream/:
orient: Capture the current snapshot of the memory hierarchy.signal: AggregationMultiSignalEdges(access / cite / co-occur).reflect: ExecuteTemporalConflict, detect bi-temporal inconsistencies.consolidate: Output Hierarchical Summary (TierDigest) and merge memories with the same meaning.prune: Identify the candidates to be eliminated based on their attenuation scores.
There's also a bypass: KnowledgeLint corrects errors through a four-tier progressive review process (graph structure → grounding → LLM → three-stage verification), and Interest Profile tracks the popularity of my topics and feeds that back to SurpriseDiscovery, which actively seeks out memories that have low engagement but high potential.
The design of this background layer partly corresponds to the integration concept in HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models(2405.14831). The thresholds for the five stages of Dream Loop, conflict resolution strategies, and details about KnowledgeLint v2 are all documented in memvault Background Track: A Closer Look.
Trigger
Cronicle Scheduling + Dual-Condition Gate.
(now - last) > 24h ∧ sessions_since ≥ 5Dream Loop
The five stages of the OODA-like cycle: Orient → Signal → Reflect → Consolidate → Prune.
MultiSignalEdges · PPR seedKnowledge Lint
A four-tier progressive audit process: graph structure → authenticity → LLM → human review.
Interest & Discovery
Attention Stratification + Identifying Interests, Gaps, and Surprises + Tier Downgrading.
surprise_ops · Leiden-weighted triggersIII. Read Track
Read Track
The entry point for the read path is QueryClassify, which selects a different strategy based on the question type, though the core is always an 11-stage scoring pipeline:
QueryRouter: Break the problem down into three categories: facts, connections, and small talk.Scoring Pipeline: Combines BM25, dense retrieval, and knowledge graph hopping with parallel scoring, followed by intent-dependent weighted merging using AttnRes.CascadeRecall: When a vector search yields no results, the system automatically delves into the knowledge graph to perform multi-step reasoning.Reranker: Use Jina Reranker v3 MLX for re-ranking.OutputFormatter+ReadTimeSanitize: Run the injection check one more time before outputting.
There's also another one running in the background: SlowThinker. It listens to the SSE stream of the conversation, predicts the next query, and prefetches candidate blocks. If the prediction is correct, it saves the cost of running a full pipeline.
The theoretical basis for intent-dependent weighted merging is derived from the paper Attention as a Hint: Detecting Irrelevant Contexts via Attention Weights(2603.15031). The three-way split criterion, the 11-stage scoring function, and the SlowThinker prediction model are all described in memvault Read Track: In Detail inside.
Query Routing
Intent classification + attention prior + layer routing + HyDE.
Fast Search
Hybrid recall + 11-stage weighting + intent-tuned blending + Redis cache.
qdrant_search · hybrid + RRFReranking
Cross-encoder two-stage fine-tuning + skip + circuit breaker.
Cascade Recall
Five Slow-only Strategies: L0 → L1 → L2 → PPR → CRAG.
(s, p, o) lookupOutput
Budget-aware reshaping + sanitize on the output side + Slow Thinker bypass.
is_unsafe_for_injection + sanitize_for_injectionIV. Design Layer Penetrated by Three Tracks
A design layer that unifies three tracks—CLT serves as the philosophical framework, 4 Event Flows as the self-driving backbone, and Reactive abstraction as the foundation for functional composition.
Cross-cutting
CLT's Three Principles + 4 Event Flows + Reactive Abstraction.
libs/ops-core · RxJS-inspiredThe deep dives into the three tracks are spread across the trilogy. This article brings them together under a single panoramic view and extracts the design layers that run through the entire architecture: CLT is my design philosophy, the 4 Event Flows form the self-driven backbone, and Reactive abstraction serves as the foundation for function composition.
Most of the parameters not listed here are those that require long-term tuning—such as scoring weights, prefetch hit rates, cascade boundaries, and confidence thresholds for knowledge graph extraction. These values are fine-tuned monthly based on data distribution and do not count as new components.
Of course, no tool is a silver bullet. Theories, technologies, and tools are still breaking through constantly, and memvault will continue to iterate along with them. Consider this blog as a review of our current progress—if there is a major overhaul in the future, we will explain it in a new article. The memvault trilogy concludes here for now.
Check the symptoms you've actually experienced to see which track to start with.
This article argues that “most people’s self-built memory systems have only two tracks—input and retrieval—and lack a context track, which is why they become more disorganized the more they use them.” For the eight symptoms listed below, you don’t need to know the names of any components—just check the ones you’ve actually encountered. The tool will determine which track to check first based on the causes outlined in this article. Everything runs within the browser; no data is sent out.
This is a comparison table based on the text, not a measurement—it can't read your data and it can't run your queries. The exact same symptom often stems from causes this article doesn't cover (the data volume was small to begin with, the embedding model is mismatched, or the query is written too vaguely); the tool can't tell the difference and will simply point in one direction based on this article's attribution. This article doesn't provide any threshold for "severe enough to fix," so it only ranks by order and doesn't assign scores.
Prompt for a Comprehensive Overview for the AI Agent
Most people’s self-built memory systems have only two tracks—“writing” and “retrieval”—and lack a “context” track, which is why they become more and more disorganized the more they use them. This article breaks down a self-organizing memory system into a three-track × three-layer framework, which serves as a handy reference chart.
Send the text below to your AI and have it compare your system point by point against the design outlined in this article, pointing out which track or layer is missing:
Further Reading
The Trilogy: In-Depth Edition + The Theoretical Roots of the "Cross-Track" Method.
| Resources | Why It Matters |
|---|---|
| memvault Write Track — After the conversation ends | Depth version of Part 1 of the trilogy. Three gates (noise/injection + camouflage/deduplication) + dual-track write (vector + KG) + Provenance + Content Normalizer 4 submodules |
| memvault Background Track — After Saving | Part two of the trilogy, deep-dive edition. Features 12 components including the Dream Loop 5 phases, Knowledge Lint 4 layers, Multi-Signal Edges, Temporal Conflict, Tier Digest, and Surprise Discovery. |
| memvault Read Track — After I Remembered | Part 3 of the Trilogy: Deep Dive Edition. Intent Classification + Personalized Router + 11-stage Scoring + Cascade Recall + Real-Time Sanitize + Slow Thinker Prefetching |
| Cognitive Load Theory (Sweller, 1988) | The Design Philosophy Behind the Three-Track "Sanitize/Segment/Verify" Gate System — Write it in a way that’s a bit challenging to write, so readers won’t get tired |
| Enterprise Integration Patterns (Hohpe & Woolf, 2003) | 4 Origins of the pub-sub / pipe-and-filter design patterns of Event Flows. memvault abstracts EventBus and Subject on top of this pattern |
| RxJS Reactive Extensions | Concrete implementation references for Operator / Pipeline / Observable / Subject. memvault doesn't use RxJS directly, but the abstraction level aligns with it |