Citation Integrity: Does the Citing Sentence Say What the Paper Says?

Objective

A measurable fraction of citations misrepresent the work they cite — a hedged finding becomes a proven fact, a null result gets cited as support, a correlation becomes a cure. Editors and reviewers can't re-read every cited paper, so the misrepresentations propagate.

Checking a citing sentence against the cited abstract is a textual-entailment problem, which is exactly what the in-process NLI model scores. This recipe builds a screening pass: every citing sentence is classified against its target abstract, and the ones the abstract doesn't support — or outright contradicts — land in a review queue with scores attached.

Step 1: The cited abstracts

CREATE TABLE IF NOT EXISTS recipe_citecheck_abstracts (
    id INT PRIMARY KEY,
    paper TEXT,
    abstract_finding TEXT
);

INSERT INTO recipe_citecheck_abstracts (id, paper, abstract_finding) VALUES
  (1, 'Rivera 2024',
      'In a small observational cohort, participants taking the supplement showed a modest association with improved sleep quality; the authors note the design cannot establish causation and call for randomized trials.'),
  (2, 'Chen 2023',
      'The randomized trial found no statistically significant difference in recovery time between the treatment and placebo groups.');

Step 2: The citing sentences — faithful, inflated, and reversed

CREATE TABLE IF NOT EXISTS recipe_citecheck_citations (
    id INT PRIMARY KEY,
    cites_paper_id INT,
    citing_sentence TEXT
);

INSERT INTO recipe_citecheck_citations (id, cites_paper_id, citing_sentence) VALUES
  (1, 1, 'An observational study reported an association between the supplement and better sleep quality.'),
  (2, 1, 'The supplement has been proven to cause improved sleep.'),
  (3, 2, 'Prior work demonstrated that the treatment shortens recovery time.');

Citation 1 is faithful. Citation 2 inflates an association into proven causation. Citation 3 cites a null result as if it were positive evidence.

Step 3: Score every citation against its cited abstract

The abstract is the premise (the evidence); the citing sentence is the hypothesis (the claim):

SELECT c.id,
       a.paper,
       CAST(JSON_EXTRACT(NLI_CLASSIFY('nli-default', a.abstract_finding, c.citing_sentence),
            '$.scores.entailment') AS DOUBLE)    AS entailment,
       CAST(JSON_EXTRACT(NLI_CLASSIFY('nli-default', a.abstract_finding, c.citing_sentence),
            '$.scores.contradiction') AS DOUBLE) AS contradiction
FROM recipe_citecheck_citations c
JOIN recipe_citecheck_abstracts a ON c.cites_paper_id = a.id
ORDER BY c.id;

Expected shape (observed on v2.1.2-ce — your exact decimals will match on the same pinned model): citation 1 scores high entailment; citation 2 scores low entailment (the abstract does not entail "proven to cause"); citation 3 scores high contradiction (the abstract says no significant difference). Record your observed values before wiring thresholds.

Step 4: The review queue — unsupported and reversed citations, worst first

SELECT c.id,
       a.paper,
       c.citing_sentence,
       CASE
         WHEN CAST(JSON_EXTRACT(NLI_CLASSIFY('nli-default', a.abstract_finding, c.citing_sentence),
                   '$.scores.contradiction') AS DOUBLE) > 0.6 THEN 'REVERSED'
         WHEN CAST(JSON_EXTRACT(NLI_CLASSIFY('nli-default', a.abstract_finding, c.citing_sentence),
                   '$.scores.entailment') AS DOUBLE) < 0.6 THEN 'UNSUPPORTED'
         ELSE 'FAITHFUL'
       END AS screen
FROM recipe_citecheck_citations c
JOIN recipe_citecheck_abstracts a ON c.cites_paper_id = a.id
ORDER BY c.id;

Expected: citation 1 → FAITHFUL, citation 2 → UNSUPPORTED, citation 3 → REVERSED. Two of three citations of real-looking text need an editor's eyes — which matches what citation-integrity studies keep finding.

Step 5: Ground truth the screen never replaces

The screen compares sentences to abstracts. It cannot know whether the abstract itself is the right target (a citation may point at the full text), and it reads rhetoric, not statistics:

SELECT COUNT(*) AS flagged
FROM recipe_citecheck_citations c
JOIN recipe_citecheck_abstracts a ON c.cites_paper_id = a.id
WHERE CAST(JSON_EXTRACT(NLI_CLASSIFY('nli-default', a.abstract_finding, c.citing_sentence),
           '$.scores.entailment') AS DOUBLE) < 0.6;

Expected: 2 — the queue size an editor actually works through, instead of re-reading every cited paper for every submission.

Cleanup (Optional)

DROP TABLE IF EXISTS recipe_citecheck_abstracts;
DROP TABLE IF EXISTS recipe_citecheck_citations;

Use it from your app

• Run the screen at submission time; attach the screen column to the reviewer's checklist rather than auto-rejecting — this is triage, not adjudication.
• Store abstracts once and screen every new manuscript's reference list against them; the pinned bundle_sha256 in every result makes screening runs comparable across time.
• Why in-DB: reference lists, abstracts, and scores live in one store — the integrity report is a JOIN, not a pipeline.

Key Concepts Learned

• Citation misrepresentation is an entailment question: premise = cited finding, hypothesis = citing sentence. Direction matters.
• Overclaiming ("proven to cause") shows up as missing entailment, not contradiction — screen on both scores, not contradiction alone.
• A null result cited as support is the easy catch: it scores as a near-pure contradiction.
• Thresholds (0.6 here) are starting points; calibrate on a labeled sample of your own field's citations before trusting the queue boundaries.