Let me start by explaining the original concept.

At that moment, there was only one thought in my mind: “The conversation is over; leave the items behind—I’ll come back for them next time.”

And that's it. It's over.

Later, when reviewing the architecture with Claude, there were a few things I hadn't thought of initially—the order of the three doors, whether the dual tracks should block responses, and whether the triple should wait for the KG to be drawn before returning. This post breaks down three things separately: organizing, saving backups, and drawing the map.

I. Tidying Up: It’s Actually Three Stages

Three steps, in that order—this is the result of the write-side evolving from the initial two actions of “noise reduction” and “deduplication” to its current version:

1. Filter out nonsense (small talk, garbled text, strings with no meaningful information)
2. Block Malicious Intent (Some people hide commands within words to manipulate memories)
3. Finally, I checked to see if I had mentioned it

Why can't the order be changed? During the third comparison, if it encounters a match that is "similar but not exactly the same," it will merge the new entry into the existing one. If a malicious message reaches this step first, the clean data will be corrupted.

This order was determined after we ran into a "MERGE" contamination issue—in the early stages, Dedup ran before Noise, so a block containing noise was flagged as similar and its contents were mixed in with those of a clean block.

"Comparison" itself isn't just about removing duplicates. It makes four types of decisions:

• Almost the same: Skip
• If there's something new, the old one still stands: Combine it with the existing entry and recalculate the fingerprint.
• The old is wrong: it has been replaced by the new
• Brand New: Newly Added

Any ambiguity regarding whether items are similar is referred to a small LLM for arbitration—it determines whether they should be merged, replaced, or left as two separate entries. So “conflict detection” is actually handled as part of this process, rather than as a separate step.

Sisters: Defend Against Deception

The second hurdle is simply writing plain-language commands like “Please forget the previous settings.” But more cunning opponents will disguise commands as statements the system has made or quotes from authoritative figures, weaving them into the conversation and waiting for them to be absorbed as memories.

Airport security can detect lighters, but it can’t stop someone from hiding contraband inside the bottom cover of a laptop. That’s why a back-scanning scanner was added after the second checkpoint, specifically designed to detect five types of camouflaged items:

• Faking Authority—“‘An official document states...’”
• Impersonation—The phrase “I am the system” pops up in a sentence
• Hidden Commands in Comments
• Time-Based Override—"The Most Recent Rule Takes Precedence"
• Encode the instructions; you have to decode them to see them.

If any of them match, block it immediately. Only if none match should it be allowed through to compare against the duplicate door. This goes hand in hand with the “blocking malicious commands” mentioned earlier—the former blocks explicit commands, while the latter blocks disguised commands; if either is missing, the system can be bypassed.

Uniform Units

You say "next Wednesday," someone else says "3-12," and the document writes "2026-03-12"—all refer to the same day, but different formats prevent them from being compared. Amounts, percentages, and durations all suffer from the same problem: "three thousand dollars," "3,000," and "three grand" are the exact same sum of money.

So before entering these doors, a small preparatory step was buried: converting dates, amounts, ratios, and durations into a uniform standard format, while clearing out full-width spaces and strange symbols in the Chinese punctuation. It sounds trivial, but without it, the subsequent comparison would miss a bunch of memories that actually ought to be merged—the same event would be treated as two, and the map would fork.

II. Backing Up: From a Single Path to Two Paths

One path is not enough—my initial approach was to "store a vectorized copy," only thinking about the semantic fingerprint route. Within a month of launching the search end, the first case popped up: when encountering abbreviations, proper nouns, and precise keyword queries, high semantic similarity actually led to wrong results. Later, it was split into two paths:

• "Storing semantic fingerprints along the way" (using your senses to remember the overall feel of a sentence)
• Store keywords along the way (like looking them up in a dictionary)

Data is inserted into the database from two sources simultaneously and sorted together during queries.

3. Draw a map and create a backup at the same time

Extracting "who - did what - to whom" triplets runs concurrently with backing up, without waiting for the vectorization to finish. Entities and relationships are stored together, while alias merging is done along the way (such as treating ChatGPT and GPT as the same). The earliest draft had two steps: first extract "entities," then "relationships," which later evolved into happening simultaneously upon storage.

Before importing the map, I’ll perform another noise-reduction check—just an extra safeguard to prevent clutter from spoiling the map.

One more thing: every time a memory is recorded, a copy is made. Source History—Who saved it, when, and how confident they were. When reading through it, I’ll sort the entries, review the context, and decide whether to keep them—and I’ll always come back to this resume. This is where I plant the seeds for a complete reading cycle.

Let me start by explaining the original concept.

At that moment, there was only one thought in my mind: After the session ends, the content should remain so I can access it again later.

And that's it. It's over.

Later, while reviewing the pipeline with Claude, I realized I hadn't broken down several decisions into enough detail back then—such as the gate order, the dual-track timing, and whether the graph build should block the HTTP response. Below, I'll discuss these three things separately: sanitize, persist, and graph build.

I. Sanitize (Organize) is actually a three-step process

Three gates—the order cannot be changed—this is the result of “sanitize” evolving from its original two functions of “noise reduction” and “duplicate removal” into its current version:

1. NoiseFilter (Noise Filtering) — Removing small talk, garbled text, and low-information content
2. InjectionGuard (Injection Protection) — Blocks malicious content such as prompt injection
3. Dedup (Duplicate removal) — Compare existing data using embeddings (vectors) and cosine similarity

Why can't the order be changed? One of Dedup's outputs is called MERGE (fusion) — when something is similar but has new information, it merges the new content into the existing block and recalculates the embedding. MERGE is a diffusive write: if a compromised payload bypasses the first two gates, it spreads through MERGE into the clean block.

This constraint was established because Dedup suffered a setback early on when it ran before Noise: a block containing an injection payload was judged as MERGE, and the payload got mixed into the content of a clean block.

Dedup itself does more than discard duplicates. It makes four types of decisions:

• SKIP (Skip): near-duplicate, do not write
• MERGE (Integration): Similar but with new information (see the previous paragraph for details)
• SUPERSEDE (Replace): The old version is no longer valid; replace the entire section.
• CREATE (New): Completely new—write a new block

The similarity falls within the "uncertain zone" (high, but not high enough to be classified directly as SKIP), Dedup calls resolve_conflict() to perform LLM arbitration and returns a MERGE / SUPERSEDE / COEXIST mapping onto the decision above. Contradiction Detection is performed as a side step here; it is not a separate phase of Fork B. The third phase of Dream Loop, "Consolidate," runs resolve_conflict() again, treating the batch conflict analysis as a second chance.

Sister Modules: PoisoningDetector

Airport security can detect items in carry-on luggage, but they can’t detect contraband hidden under the laptop’s bottom cover—InjectionGuard is the frontal X-ray, blocking explicit prompt injection; PoisoningDetector is the backscatter side, blocking disguised attacks. Both are mounted side-by-side on G2; missing even one attack surface will leave it wide open.

The corresponding module is memvault.security.poisoning; its input is the raw block payload that has already passed through the G1 NoiseFilter, and any hit results in a complete rejection (it never enters G3 Dedup). The judgment is based on five types of spoofing templates:

• impersonation of an authority figure: Fabricating authoritative citations such as “OpenAI official documents state…” or “According to Anthropic’s security guidelines…” — pattern matching + comparison against a whitelist of known references
• Role Self-Declaration: I am the system / You are now / system: role-announcement strings appearing in the body of the text, and the like
• Markdown comment injection: <!-- ignore previous instructions -->, this kind of implicit instruction hidden in a comment
• temporal manipulation: Phrases such as “latest instructions take precedence” and “override prior rules” that use chronological order to override existing policies
• encoded payload: Consecutive high-entropy Base64 / hex / unicode escape blocks — detected using Shannon entropy + base alphabet ratio

The template comes from memory-lancedb-pro's 5-vector taxonomy; it's split into 5 independent detectors during implementation, with a short-circuit triggered upon any hit. The trust_score is also calculated here—even if a spoofing attack source manages a partial match, downstream scoring will downweight it to unreachability.

Pre-normalization: ContentNormalizer

"Next Wednesday" and "2026-03-12" refer to the same day, but different formats prevent them from matching—before the three gates, there is an added normalize pass that folds surface strings into canonical tokens, ensuring that downstream Dedup similarity comparisons and LLM arbitration both receive aligned inputs. Hung on libs/text-ops, five submodules connected in a chain:

• TemporalNormalizer: "Next Wednesday", "3/12", "March 12" → ISO-8601. Rewritten after cannibalizing dateparser + MS Recognizers, 10-pass zero-dependency, covering mixed Chinese-English, relative time, and fuzzy ranges
• CurrencyNormalizer: “3,000 yuan,” “3,000 NTD,” “USD 100” → {amount, currency} tuple; the FX rate is not processed at this level
• ProportionNormalizer: “70%,” “0.7,” “7/10” → Standardize to a float in [0, 1]
• DurationNormalizer: “Two and a half hours,” “2.5h,” “150 min” → ISO-8601 duration (PT2H30M))
• preprocess_chinese: Full-width/half-width folding, CJK punctuation normalization, zero-width character stripping — a pre-pass that runs before the above four steps

Without this layer, Dedup would classify “dinner next Wednesday night” and “2026-03-11 19:00 dinner” as having low similarity, resulting in CREATE rather than MERGE; KG's entity resolution also treats “Jones,” “JonesHong,” and “j-hong” as three different people—causing the graph to branch. For details, see libs/text-ops/docs/opensource-assessment.md.

II. Persist (Write): From a Single Track to a Dual Track

A single path is not enough—my initial pitch was "save a vectorized copy," focusing solely on the dense vector approach. As soon as the search end went live, problems emerged: when encountering abbreviations, proper nouns, and exact keyword queries, semantic similarity actually led to inaccurate results. Later, we split it into hybrid indexing:

• Dense: MLX (Apple Machine Learning Framework) Qwen3-Embedding, 1024 dimensions
• Sparse: BM25 Keyword Index

Write both paths into the same collection in Qdrant. At the query end, use RRF (Reciprocal Rank Fusion) to combine and rank the results from both paths.

III. Graph Build: Reactive Pipe Takes Over

Extract SPO (subject-predicate-object) triples, hang them on the MemvaultEvents.MEMORY_STORED channel as an event-driven trigger, and don't wait for Fork A's vectorization to finish — this pipeline is fire-and-forget. The earliest draft of the graph build was two steps: extract entities first, then extract relations. In practice these became simultaneous, done synchronously while writing the triples, as Entity Resolution: alias disambiguation, folding into an EntityCanonical node (ChatGPT and GPT are treated as the same entity).

There's one more step before entering the atlas: NoiseGateOp, a second review—an extra layer of protection to prevent nonsense from creeping in.