Semantic Deduplication of Agent Memories

Objective

An agent that writes a memory every time the user mentions something soon has ten near-identical rows saying "the user likes dark mode." That bloats recall, biases scoring toward whatever's repeated, and wastes tokens. Exact-match dedup can't catch paraphrases. Here you'll dedup by meaning: compare new memories against existing ones with vector similarity, skip writes that are semantically redundant, and collapse the duplicates already in the store. The same store works from any framework or a voice agent — see Use it from your agent at the end.

Step 1: Create the agent memory store

One row per memory, embedded so we can measure semantic closeness between rows.

CREATE TABLE IF NOT EXISTS recipe_agent_dedup (
  memory_id  INTEGER PRIMARY KEY,
  agent_id   TEXT,
  content    TEXT,
  is_active  INTEGER DEFAULT 1,                       -- 0 = retired as a duplicate
  embedding  VECTOR(384)
);

Step 2: Insert memories — several are near-duplicates

The same preference and the same fact arrive phrased differently across sessions.

INSERT INTO recipe_agent_dedup (memory_id, agent_id, content) VALUES
 (1,'assistant','The user prefers dark mode.'),
 (2,'assistant','The user likes the app in dark theme.'),
 (3,'assistant','User said they want the interface in dark mode please.'),
 (4,'assistant','The user is allergic to peanuts.'),
 (5,'assistant','The user works as a data engineer.'),
 (6,'assistant','The user has a peanut allergy.'),
 (7,'assistant','The user is planning a trip to Japan.');

Step 3: Embed the memories

The embedding model runs in-database, so semantically equal rows land close in vector space.

UPDATE recipe_agent_dedup SET embedding = EMBED(content);

Step 4: Detect near-duplicate pairs

Self-join the table and surface pairs whose meaning is nearly identical — the candidates to merge.

SELECT a.memory_id AS keep_id, b.memory_id AS dup_id,
       a.content AS kept, b.content AS duplicate,
       COSINE_SIMILARITY(a.embedding, b.embedding) AS similarity
FROM recipe_agent_dedup a
JOIN recipe_agent_dedup b
  ON a.agent_id = b.agent_id AND a.memory_id < b.memory_id
WHERE COSINE_SIMILARITY(a.embedding, b.embedding) >= 0.80
ORDER BY similarity DESC;

Step 5: Check a new memory before writing it (dedup on ingest)

Before inserting "user wants dark theme," ask whether the store already knows it — and skip if so.

SELECT memory_id, content,
       COSINE_SIMILARITY(embedding, EMBED('the user wants a dark theme interface')) AS similarity
FROM recipe_agent_dedup
WHERE agent_id = 'assistant' AND is_active = 1
ORDER BY similarity DESC
LIMIT 1;

Step 6: Retire the duplicates already in the store

Flag every memory that has a higher-priority near-twin as inactive, keeping the earliest of each cluster.

UPDATE recipe_agent_dedup
SET is_active = 0
WHERE memory_id IN (
  SELECT b.memory_id
  FROM recipe_agent_dedup a
  JOIN recipe_agent_dedup b
    ON a.agent_id = b.agent_id AND a.memory_id < b.memory_id
  WHERE COSINE_SIMILARITY(a.embedding, b.embedding) >= 0.80
);

Step 7: Confirm a clean, deduped memory set

List the surviving active memories — one row per distinct fact.

SELECT memory_id, content
FROM recipe_agent_dedup
WHERE agent_id = 'assistant' AND is_active = 1
ORDER BY memory_id;

Cleanup (Optional)

DROP TABLE IF EXISTS recipe_agent_dedup;

Expected Outcomes

• Step 4 pairs the three dark-mode memories and the two peanut-allergy memories as near-duplicates, with high similarity scores.
• Step 5 finds an existing dark-mode memory above threshold for a new "dark theme" candidate — so the agent skips the redundant write.
• Step 6–7 retire the duplicate rows and leave one memory per distinct fact: dark mode, peanut allergy, data engineer, Japan trip.

You now keep agent memory clean — duplicates are caught by meaning on ingest and collapsed in the store, so recall stays sharp and unbiased.

Use it from your agent (framework-agnostic — this is the whole point)

Dedup is just a similarity check before write + a cleanup pass, so any agent uses it with no framework lock-in:

• REST / SDK — POST /v1/query/execute (any language), or @synapcores/sdk client.executeQuery(...). Before writing a memory your agent runs the Step-5 check and only inserts when the top similarity is below threshold; a scheduled Step-6 pass collapses anything that slipped through.
• MCP (native, on by default) — point any MCP client (Claude Code, Cursor, a custom loop, a voice runtime) at ws://<your-instance>/mcp?token=<jwt> (JWT from one POST /v1/auth/login → access_token). The query tool runs the pre-write similarity check; the execute tool inserts or retires rows — dedup as tool calls.
• Any framework — OpenClaw, LangChain / LlamaIndex memory stores, a custom loop, or a voice agent all dedup against the same vector store. The database is the brain; the framework is swappable.

Key Concepts Learned

• Exact-match dedup misses paraphrases; vector similarity catches "same fact, different words."
• A self-join on COSINE_SIMILARITY finds near-duplicate clusters in one query.
• Checking similarity before writing prevents the bloat instead of cleaning it up later.
• Because it's plain data ops (SQL / REST / MCP), semantic dedup works for any agent — the agent-agnostic backend pattern this cluster builds on.