- Who it helps
- Researchers evaluating coding models and interpreting repair benchmark scores
- Availability
- Public paper, posted to arXiv October 14 and listed as USENIX Security 2026 work. Runnable artifacts were not checked.
Repository and patch evidence
One stage probes a model with code prefixes to infer repository inclusion. Another checks whether a patch persists, using similarity and tests. The result is evidence about contamination under the study’s assumptions.
Interpreting reported scores
The authors report strong detection on their tested models and reduced repair scores after filtering suspected examples. This does not establish the training membership of every commercial model or every benchmark task.
What you can do
- Examine the detection assumptions before filtering a coding benchmark. Open reference ↗
Where the evidence stops
Abstract checked November 11. No detection pipeline, repository-history analysis or benchmark filtering reproduced.
Sources read. Public source text checked November 11, 2026 UTC. No third-party project or complete workflow executed.
Sources & further reading
Paper · paper
BenchChecker evaluation ↗
October 14 arXiv abstract describes repository and patch-presence tests; inference about contamination has evaluation limits.
Di Wu and coauthors · page checked 2026-11-11 · Published 2026-10-14
Coverage note: Abstract checked November 11. No detection pipeline, repository-history analysis or benchmark filtering reproduced.
The record over time (1)
Dated source checks and published explanations. An observation date can differ from the original release date.
- · Added to Current
BenchChecker uses model outputs and public repository history to investigate whether bug-fixing evaluation examples may have appeared in training data.