CATT tests whether attribution finds the training facts behind an answer

A controlled test

The authors fine-tune models on fictitious knowledge, then vary repetition and distracting documents separately. This makes the intended source known and exposes attribution failures that a mixed real-world dataset can obscure.

Controlled Attribution Benchmark paper ↗

Keep the relevant tokens visible

The study changes gradient scoring and combines the strongest token scores instead of averaging a whole document. Reported gains concern training-data attribution. They do not establish that a chatbot citation supports a particular answer or that its full training corpus can be recovered.

Controlled Attribution Benchmark paper ↗CATT project explanation ↗

What you can do

1. Compare the controlled-data assumptions with the attribution task you need to evaluate. Open reference ↗

Where the evidence stops

No experiment was reproduced. Synthetic facts and the tested retraining benchmark limit generalization to arbitrary deployed models.

Sources read. Abstract and project page read October 31, 2026. No model training, inference or attribution workflow executed.

Sources & further reading