The NLI Pitfalls Test Battery

Objective

Before wiring NLI verdicts into anything automated, learn where the model class is weak — on your engine, your model, your thresholds. Each probe below has a defensible correct answer; run them all, record what your deployment actually returns, and let the misses define the boundary of what you automate.

Step 1: Direction — entailment is one-way

curl -s -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{
    "model": "nli-default",
    "pairs": [
      {"premise": "A red car is parked.",  "hypothesis": "A vehicle is parked."},
      {"premise": "A vehicle is parked.",  "hypothesis": "A red car is parked."},
      {"premise": "No vehicle is parked.", "hypothesis": "A vehicle is parked."}
    ]
  }' http://localhost:8080/v1/ai/nli/pairs

Correct: entailment / neutral / contradiction. A model that scores pair 2 as entailment is pattern-matching overlap, not reasoning. Observed on v2.1.1-ce with nli-default: all three correct, each above 0.99.

Step 2: The probes that fail in production

Each is one /verify call; the body's statement and anchor are shown as claim ⟂ evidence:

# Ambiguous pronoun — correct: neutral ("they" has no resolved referent)
#   "Jordan approved the request." ⟂ "Morgan spoke to Jordan after they approved the request."
# Numeric upper bound — correct: neutral (at most 10 ≠ exactly 10)
#   "The service limit is exactly 10 users." ⟂ "The service limit is at most 10 users."
# Numeric conflict — correct: contradiction (5 kg vs 10 kg, same parcel)
#   "The parcel weighs exactly 10 kilograms." ⟂ "The parcel weighs exactly 5 kilograms."
# Role reversal — correct: neutral (Alice paying Bob ≠ Bob paying Alice)
#   "Bob paid Alice." ⟂ "Alice paid Bob."
# Quoted ≠ asserted — correct: neutral (quoting a claim is not asserting it)
#   "The system is secure." ⟂ "The reviewer wrote the sentence \"The system is secure\" as an example of an unsupported claim."
# Nested negation — correct: contradiction
#   "The backup is enabled." ⟂ "It is not true that the backup is enabled."
curl -s -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{
    "model": "nli-default",
    "statement": "The backup is enabled.",
    "truth_anchors": ["It is not true that the backup is enabled."],
    "closed_world": false,
    "contradiction_threshold": 0.6, "entailment_threshold": 0.6
  }' http://localhost:8080/v1/ai/nli/verify

Run each pair the same way and record the scores — misses here are normal for this model class, and knowing which ones miss is the deliverable.

Observed on v2.1.1-ce with nli-default (your results may differ):

The quoted-claim miss is the one to design around: text that mentions a claim ("the reviewer wrote the sentence '…' as an example of an unsupported claim") can certify the claim as TRUTH. Strip quotations and reported speech from evidence before verification, or exclude such passages from anchor selection entirely.

Step 3: No multi-premise proofs

curl -s -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{
    "model": "nli-default",
    "statement": "Parcel P1 is held in room R2.",
    "truth_anchors": [
      "Parcel P1 is held in locker L1.",
      "Locker L1 is located in room R2."
    ],
    "closed_world": false,
    "contradiction_threshold": 0.6, "entailment_threshold": 0.6
  }' http://localhost:8080/v1/ai/nli/verify

Correct: neutral on both anchors — each anchor is evaluated independently; the API deliberately does not chain them into a proof. Observed on v2.1.1-ce with nli-default: contradiction (1.00) on both anchors, verdict LIE — without the chain, each individual location fact reads as conflicting with the composed claim. Either way the composed fact is never proven; if your use case needs the chain, do the join in SQL or Cypher and verify the final sentence.

Step 4: Thresholds are policy, not truth

curl -s -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{
    "model": "nli-default",
    "statement": "The standard plan includes email support.",
    "truth_anchors": ["The standard plan includes email support."],
    "closed_world": false,
    "contradiction_threshold": 0.0, "entailment_threshold": 0.6
  }' http://localhost:8080/v1/ai/nli/verify

Expected: verdict LIE on an identical sentence — with a zero contradiction threshold, any nonnegative contradiction score "qualifies". This demonstrates that the verdict is your policy applied to probabilities. Choose thresholds with the same care as the evidence.

Honest boundaries

Oversized inputs are refused (byte and token limits), and a failed request, missing model, or disconnect must never be turned into a success verdict by your integration. Pronouns, numbers, dates, and role direction are the known weak spots — handle them with typed logic and clarification, and retest the battery whenever you change model or thresholds.

Key Concepts Learned

• Acceptance-test the model class's known failure modes on your own deployment before automating.
• Independent-anchor evaluation is a feature: it keeps verdicts auditable.
• Thresholds are product policy; probabilities are the only thing the model outputs.