RAG Answer Gate: Retrieve, Verify, Then Ship

Objective

Your RAG pipeline retrieves passages, an LLM drafts an answer, and the answer ships — whether or not the evidence actually backs it. The classic fix is a second service that re-checks the answer; here the whole gate is one SQL statement: COSINE_SIMILARITY retrieval feeds NLI_ANCHORS, which feeds NLI_VERIFY, and only a TRUTH verdict lets the answer through. Along the way you'll hit the trap everyone hits — top-k retrieval feeding the verifier junk — and fix it with a similarity floor.

Step 1: The evidence table

CREATE TABLE IF NOT EXISTS recipe_raggate_evidence (
    id INT PRIMARY KEY,
    source TEXT,
    body TEXT,
    embedding VECTOR(384)
);

Step 2: Seed retrieved-world content — relevant and irrelevant

Real corpora are mostly irrelevant to any one question. Seed both kinds.

INSERT INTO recipe_raggate_evidence (id, source, body, embedding) VALUES
  (1, 'deploy-log',  'The deployment finished at 14:02 with zero failed checks.',
      EMBED('The deployment finished at 14:02 with zero failed checks.')),
  (2, 'deploy-log',  'All 46 post-deploy probes passed.',
      EMBED('All 46 post-deploy probes passed.')),
  (3, 'facilities',  'Lunch was served in the cafeteria.',
      EMBED('Lunch was served in the cafeteria.')),
  (4, 'hr-wiki',     'New hires must complete security training in week one.',
      EMBED('New hires must complete security training in week one.'));

Step 3: Ground truth — what does retrieval actually return?

Before trusting any gate, look at the similarity scores it will act on.

SELECT id, source, COSINE_SIMILARITY(embedding, EMBED('deployment result')) AS sim
FROM recipe_raggate_evidence
ORDER BY sim DESC;

Expected: the two deploy-log rows score far above the rest (observed ~0.48 and ~0.36 on the default 384-dim model) while the cafeteria row scores near zero (~0.01). Note the scale: similarity is not a probability, and useful floors on short text sit well below 0.5.

Step 4: The trap — top-k feeds the verifier junk, and junk can read as contradiction

Verify the answer against the top 3 retrieved rows, k chosen carelessly:

SELECT NLI_VERIFY('nli-default',
  'The rollout completed without failures.',
  NLI_ANCHORS(body)) AS gate
FROM (
  SELECT body
  FROM (
    SELECT body, COSINE_SIMILARITY(embedding, EMBED('deployment result')) AS sim
    FROM recipe_raggate_evidence
    ORDER BY sim DESC
    LIMIT 3
  ) ranked
) top_k;

Expected: verdict LIE — with violated_anchor = "Lunch was served in the cafeteria.". The third-ranked row is irrelevant, and the model scores the unrelated sentence as a contradiction of the claim (observed contradiction ≈ 0.9998). Contradiction takes precedence over entailment, so one junk anchor outvotes two perfect ones. Top-k with no floor is how a correct answer gets blocked — or, with the polarity flipped, how a wrong one sneaks through.

Step 5: The fix — a similarity floor decides what the verifier sees

SELECT NLI_VERIFY('nli-default',
  'The rollout completed without failures.',
  NLI_ANCHORS(body)) AS gate
FROM (
  SELECT body, COSINE_SIMILARITY(embedding, EMBED('deployment result')) AS sim
  FROM recipe_raggate_evidence
) scored
WHERE sim > 0.3;

Expected: verdict TRUTH, matched_anchor = "The deployment finished at 14:02 with zero failed checks.", confidence ≈ 0.998. Only the two deploy-log rows clear the floor; the verifier sees clean evidence.

Calibrate the floor on Step 3's observed scores for your corpus and embedding model — 0.3 is right for this seed data, not a universal constant.

Step 6: The gate as an INSERT — nothing unverified reaches the answers table

CREATE TABLE IF NOT EXISTS recipe_raggate_answers (
    id INT PRIMARY KEY,
    question TEXT,
    answer TEXT,
    verdict TEXT,
    confidence DOUBLE
);

INSERT INTO recipe_raggate_answers (id, question, answer, verdict, confidence)
SELECT 1,
       'Did the rollout succeed?',
       'The rollout completed without failures.',
       JSON_UNQUOTE(JSON_EXTRACT(gate, '$.verdict')),
       CAST(JSON_EXTRACT(gate, '$.confidence') AS DOUBLE)
FROM (
  SELECT NLI_VERIFY('nli-default',
    'The rollout completed without failures.',
    NLI_ANCHORS(body)) AS gate
  FROM (
    SELECT body, COSINE_SIMILARITY(embedding, EMBED('deployment result')) AS sim
    FROM recipe_raggate_evidence
  ) scored
  WHERE sim > 0.3
) verified
WHERE JSON_UNQUOTE(JSON_EXTRACT(gate, '$.verdict')) = 'TRUTH';

SELECT id, answer, verdict, confidence FROM recipe_raggate_answers;

Expected: exactly 1 row, verdict TRUTH, confidence ≈ 0.998. Re-run the INSERT with a statement the evidence does not back and zero rows land — the gate is the WHERE clause.

Step 7: The empty-retrieval case fails closed

If the floor filters out everything, NLI_VERIFY does not guess:

SELECT COUNT(*) AS evidence_rows
FROM recipe_raggate_evidence
WHERE COSINE_SIMILARITY(embedding, EMBED('quarterly revenue figures')) > 0.3;

Expected: 0. Running the verify over that empty set raises NLI truth_anchors requires 1..=32 entries — an error, never a silent pass. Treat "no evidence retrieved" as its own outcome in your application (serve nothing, or serve with an explicit unverified banner); check the count first as above.

Cleanup (Optional)

DROP TABLE IF EXISTS recipe_raggate_evidence;
DROP TABLE IF EXISTS recipe_raggate_answers;

Use it from your app

• SQL (shown here): retrieval + verification + the gating INSERT is one round trip — no verifier sidecar, no second vector store.
• REST: retrieve however you like, then POST /v1/ai/nli/verify with the passages as truth_anchors. Same model, same verdicts.
• Why in-DB: the evidence, the embeddings, the verifier, and the gated write share one transaction boundary and one auth token.

Key Concepts Learned

• NLI_ANCHORS turns retrieved rows into the anchors array NLI_VERIFY consumes — retrieval-to-verification is one statement.
• Top-k is not a relevance guarantee. One irrelevant anchor can score as a contradiction and veto a correct answer; gate what the verifier sees with a similarity floor calibrated on observed scores.
• Verify-then-write: making the verdict a WHERE clause on the INSERT means unverified text never reaches the serving table.
• An empty anchor set is a loud error, not a default verdict — fail-closed by design; count evidence before verifying.