Contradiction Mining Across Statements — and Why It Needs a Similarity Gate

Objective

Discovery sets, witness statements, and interview transcripts contain the same facts told different ways — and sometimes told incompatibly. Finding the contradictions by hand is the expensive part of review. You can screen every pair with NLI_CLASSIFY, but this recipe's real lesson is the trap you hit first: a pairwise NLI model scores unrelated sentences as contradictions, so raw n² screening is mostly noise until a vector-similarity pre-filter removes the pairs that were never comparable. The working pipeline is similarity-gate → NLI → human review.

Step 1: The statements, with embeddings

CREATE TABLE IF NOT EXISTS recipe_contra_statements (
    id INT PRIMARY KEY,
    speaker TEXT,
    statement TEXT,
    embedding VECTOR(384)
);

INSERT INTO recipe_contra_statements (id, speaker, statement, embedding) VALUES
  (1, 'Witness A', 'I left the office at 6pm on the night of the incident.',
      EMBED('I left the office at 6pm on the night of the incident.')),
  (2, 'Witness B', 'Witness A was still at the office at 9pm that night.',
      EMBED('Witness A was still at the office at 9pm that night.')),
  (3, 'Witness C', 'Witness A departed the office in the evening.',
      EMBED('Witness A departed the office in the evening.')),
  (4, 'Witness D', 'The north parking garage was closed for repairs all week.',
      EMBED('The north parking garage was closed for repairs all week.'));

Statement 2 contradicts 1; statement 3 corroborates 1 (and so conflicts with 2); statement 4 is about something else entirely.

Step 2: The trap — raw pairwise contradiction flags everything

Score every unordered pair by contradiction, highest first:

SELECT a.id AS a_id, b.id AS b_id,
       ROUND(CAST(JSON_EXTRACT(NLI_CLASSIFY('nli-default', a.statement, b.statement),
             '$.scores.contradiction') AS DOUBLE), 3) AS contradiction
FROM recipe_contra_statements a
JOIN recipe_contra_statements b ON a.id < b.id
ORDER BY contradiction DESC;

Expected (observed on v2.1.2-ce, nli-default):

Three of the top four "contradictions" involve the parking-garage statement, which contradicts nothing — it's just unrelated. A cross-encoder trained on sentence pairs was never asked "are these two about the same thing?"; feed it two unrelated sentences and it still returns a confident label. Raw n² contradiction screening is noise.

Step 3: The fix — gate on similarity first, then contradiction

The unrelated pairs are exactly the low-similarity pairs. Require both a vector-similarity floor (same topic) and a contradiction threshold:

SELECT a.id AS a_id, a.speaker AS a_speaker, b.id AS b_id, b.speaker AS b_speaker,
       ROUND(COSINE_SIMILARITY(a.embedding, b.embedding), 3) AS similarity
FROM recipe_contra_statements a
JOIN recipe_contra_statements b ON a.id < b.id
WHERE COSINE_SIMILARITY(a.embedding, b.embedding) > 0.4
  AND CAST(JSON_EXTRACT(NLI_CLASSIFY('nli-default', a.statement, b.statement),
           '$.scores.contradiction') AS DOUBLE) > 0.6
ORDER BY similarity DESC;

Expected: exactly two rows — (2, 3) and (1, 2). Every garage pair sits below the 0.4 similarity floor (observed similarities: (1,4)=0.14, (2,4)=0.18, (3,4)=−0.02) and is gone; what remains is the genuine Witness A timeline dispute. The corroborating pair (1, 3) is on-topic but has near-zero contradiction, so the contradiction threshold drops it.

Step 4: Materialize the review queue

CREATE TABLE IF NOT EXISTS recipe_contra_queue (
    a_id INT, b_id INT, contradiction DOUBLE,
    PRIMARY KEY (a_id, b_id)
);

INSERT INTO recipe_contra_queue (a_id, b_id, contradiction)
SELECT a.id, b.id,
       CAST(JSON_EXTRACT(NLI_CLASSIFY('nli-default', a.statement, b.statement),
            '$.scores.contradiction') AS DOUBLE)
FROM recipe_contra_statements a
JOIN recipe_contra_statements b ON a.id < b.id
WHERE COSINE_SIMILARITY(a.embedding, b.embedding) > 0.4
  AND CAST(JSON_EXTRACT(NLI_CLASSIFY('nli-default', a.statement, b.statement),
           '$.scores.contradiction') AS DOUBLE) > 0.6;

SELECT q.a_id, sa.speaker AS a_speaker, q.b_id, sb.speaker AS b_speaker,
       ROUND(q.contradiction, 3) AS contradiction
FROM recipe_contra_queue q
JOIN recipe_contra_statements sa ON q.a_id = sa.id
JOIN recipe_contra_statements sb ON q.b_id = sb.id
ORDER BY q.contradiction DESC;

Expected: the two Witness A timeline conflicts, strongest first. A reviewer opens two pairs, not six — and the two are real.

Step 5: Calibrate both thresholds on your own data

The 0.4 similarity floor and 0.6 contradiction threshold are starting points for this seed set, not constants. Raise the similarity floor if unrelated pairs still leak through; lower it if genuinely-related statements phrased very differently get dropped. Record the scores your corpus and embedding model produce (Step 2's matrix) before fixing either number.

Cleanup (Optional)

DROP TABLE IF EXISTS recipe_contra_statements;
DROP TABLE IF EXISTS recipe_contra_queue;

Use it from your app

• Embed statements at ingest; run the Step 3 gated query to build the review queue, and let reviewers work it top-down by contradiction score.
• Swap COSINE_SIMILARITY scan for an HNSW index on large sets — the similarity gate is also what keeps the NLI cost linear instead of n².
• Why in-DB: embeddings, pairwise scores, and the reviewer's queue are one store; no export of sensitive statements to an external service.

Key Concepts Learned

• A pairwise NLI model has no notion of topic relevance — unrelated sentences score as confident contradictions, so raw n² mining is noise.
• The similarity floor is the relevance test NLI lacks: gate on COSINE_SIMILARITY first, run NLI_CLASSIFY only on same-topic pairs.
• Rank the survivors by contradiction for a conflict queue; near-zero contradiction on an on-topic pair is corroboration, not a conflict.
• Both thresholds are corpus-specific; derive them from the observed score matrix, and keep the queue as human triage, not an automated verdict.