PPubMed21 May 2026

Logic, inference, understanding: cross-domain generalization for generative language models

Rasmus Blanck, Bill Noble

Abstract

Neural systems for Natural Language Inference (NLI) have seen impressive performance over the last ten years, but their ability to generalize beyond their training data has repeatedly been questioned. The NLI task has long been considered as a proxy for the wider problem of Natural Language Understanding (NLU), implicitly motivated by relying on an inferentialist conception of semantics. This paper draws on insights from work in formal logic and semantics to introduce distinctions between different notions of generalizations (in-domain vs. cross-domain, and linguistic vs. inferential) in an attempt to disentangle the problem of generalization. We leverage the theoretical contributions in experiments addressing the inferential generalization power of autoregressive NLI models.

§ The Valyu brief

The paper, in five minutes

Read from the full text via Valyu Contents

Generative decoder-only models show limited cross-domain NLI performance, with modest drops compared to encoder models, highlighting challenges in inferential generalization.

What’s new

i.Define formal distinctions between in-domain vs. cross-domain and linguistic vs. inferential generalization for NLI.
ii.Link inferentialist philosophy to the NLI/NLU relationship through theoretical analysis.
iii.Empirically evaluate generative decoder-only models (Pythia) on zero-shot, in-domain, and cross-domain NLI tasks.
iv.Show that generative models experience smaller cross-domain drops than encoder models, but still significant.
v.Analyze factors (parameter count, pre‑training steps, annotation artifacts) affecting cross-domain generalization.

How they did it

The authors fine‑tuned Pythia decoder‑only transformer models of four sizes (14M–410M parameters) on SNLI or SICK datasets and evaluated zero‑shot, in‑domain, and cross‑domain accuracy using standard NLI labels (E, C, N) and macro‑average recall. They also ran hypothesis‑only baseline experiments to assess annotation bias.

What they found

Zero‑shot macro‑average recall is ~33.3% on SNLI (chance) and up to 38.4% for the 410M model; on SICK the 160M model reaches 4.3 points above chance.
In‑domain macro‑average recall reaches up to 79.5% for SICK‑fine‑tuned models.
Cross‑domain performance drops 21.8–33.5 percentage points, e.g., the 160M model drops 21.8 points (SNLI→SICK) versus ~30+ points for encoder models.
Increasing model size does not consistently improve cross‑domain generalization.

Caveats

✕Cross‑domain accuracy still declines sharply, indicating limited inferential generalization.
✕Findings are restricted to English NLI datasets (SNLI, SICK) and the Pythia model family.
✕No evidence that larger parameter counts improve cross‑domain performance.
✕The study does not resolve how different abstract consequence relations affect evaluation.

§ The Valyu brief

Reading the full paper and taking notes. This takes a few seconds…

§ Ask this paper

Ask a question about this paper

Valyu reads the full text and answers from what the paper actually says.

§ Related work in other archives

Similar papers in all four archives

The same idea searched in arXiv, PubMed, bioRxiv and chemRxiv at once, so a method can lead you to where it is being applied.

AarXiv4 found

PPubMed3 found

BbioRxiv4 found

CchemRxiv4 found

Searching the other archives…