Benchmark Dataset Contamination Detection Methods

Benchmark contamination means the test has leaked into the study guide. A model's training corpus already contains the questions (or close cousins of them) that will later be used to grade it, and the score that results tells you more about memorization than reasoning. Given how web-scale pretraining corpora get built, this is close to the default state of affairs rather than an edge case: any benchmark released publicly gets mirrored, quoted, and re-posted across code-hosting sites, forums, and blog write-ups within weeks, and it ends up baked into the next crawl. An audit against DCLM-baseline (arXiv 2506.12229) found 27.70% of MMLU entries already dirty, along with 32.6% of ARC-Challenge, 32.3% of ARC-Easy, and 16.20% of MMLU-Pro. Even explicit exclusion filters, the lists of "do not train on this" hashes that some labs maintain, do not stop the leakage, because contamination now arrives sideways: through data aggregation pipelines that repackage a benchmark under a new name, through distillation from a contaminated teacher model, through continual pretraining runs that quietly reintroduce material a base model never saw. Contamination is a decay curve. It is a decay curve. Newer benchmarks tend to start out clean and get dirtier as they age into the crawl: on the more recent CC-2025-05 corpus, AIME-2024 shows only 10.00% contamination and GPQA just 0.89%, numbers that will almost certainly look different a year from now.

Detecting this leakage is its own subfield at this point, with five distinct method families, each resting on different assumptions about what evidence is available and what "contamination" even means at a technical level. None of them solve the problem. Understanding what each one can and cannot see is the actual skill here.

A four-tier contamination taxonomy that determines which detection methods apply

Before comparing detection methods, it helps to fix what they're detecting. The most widely cited framework as of 2026 comes from Nourbakhsh et al., a systematic review spanning 55 studies through late 2025, which sorts contamination into four tiers of increasing subtlety.

Tier one, exact contamination, is a benchmark question sitting verbatim in the training data, word for word. It's the easiest case to reason about and the easiest to catch, because a direct string match will find it. Tier two, syntactic contamination, covers test items that survive a light rewrite: someone reordered the clauses, swapped in synonyms, or changed the surface phrasing while keeping the underlying question intact. A naive string match will usually miss this, though n-gram overlap can sometimes still catch fragments of it. Tier three, semantic contamination, is where surface-level tools stop working altogether: the meaning of the question is duplicated even though nothing about the wording matches, so only something like embedding similarity comparison can flag it. The review also names a fourth tier, task-level contamination, where the general skill or format tested by a benchmark has been trained on repeatedly, even absent any single overlapping item. That fourth tier is arguably the hardest of all to pin down, since it has no discrete instance to point at.

The tiers matter because they set a ceiling on every method that follows. A detection technique built to catch tier-one contamination will report a clean bill of health on tier-three contamination every time, because the method was never built to see it.

String-matching and n-gram overlap: the baseline method and its hard ceiling

The oldest and still most common detection method is direct overlap calculation: take the benchmark, take the training corpus, and check for shared n-grams. Touvron et al.'s Llama papers used this approach, as did the technical report for another major lab's flagship model, and many labs still lean on various indexing approaches to make the search over trillions of tokens computationally tractable.

The appeal is obvious. N-gram matching is cheap relative to the alternatives, easy to audit, and gives a clean yes-or-no answer for a given string. The ceiling is just as obvious once you hold it up against the taxonomy above: it catches tier-one contamination reliably and picks up some tier-two cases if the rephrasing is minor enough, but it is structurally blind to tier three and tier four. A paraphrase, a translated version, a reformatted table, none of these leave an n-gram trail, even though the model may have absorbed the exact same information.

There's also a scale problem that gets worse every year rather than better. As pretraining corpora grow past the trillion-token mark, running exhaustive n-gram comparisons against every benchmark of interest becomes an expensive, slow operation in its own right, one that a lot of teams simply skip or run on a sample instead of the full corpus.

The clearest illustration of the ceiling comes from a documented case involving a 13-billion-parameter Llama model trained on paraphrased versions of MMLU, GSM8K, and HumanEval test sets. Because none of the surface text matched the originals, the paraphrased material was not flagged by standard overlap-based checks. And yet the model came out performing at roughly the level of that other flagship model on those same benchmarks, having absorbed the benchmark material through the paraphrased exposure. The filter did its job perfectly and still missed the contamination completely, because the job it was built to do was never big enough for the problem.

Likelihood-based and perplexity methods: probing what the model has internalized

If string matching looks at the training data, likelihood-based methods look at the model itself. The core intuition: a model that has seen a specific example during training will usually assign it anomalously low perplexity compared to a similar, uncontaminated example it has never encountered. Loss thresholding, in other words, flags items the model finds suspiciously easy to predict.

Raw perplexity comparisons are noisy on their own, since sequence-level averaging can wash out a strong signal sitting in just a handful of tokens. Refinements in this space address the noise problem by focusing on the most informative tokens in a sequence rather than treating every token as equally diagnostic, sharpening the signal considerably.

A second branch of this family, perturbation-based detection, has been developed in parallel by several research groups. Instead of reading perplexity directly, these methods perturb a candidate example, swap a word, alter a clause, and measure how sharply the model's confidence drops in response. A memorized example tends to show a steep, brittle drop once perturbed, because the model was relying on the exact surface form rather than a general understanding of the content. A genuinely unseen example degrades more gently.

Both branches can in principle be extended toward semantic-level (tier three) contamination by pairing them with embedding-based similarity measures that compare meaning rather than surface tokens. This is one of the few detection strategies in the current toolkit that reaches past tier two at all, which is precisely why serious contamination audits use it so often despite its being more computationally demanding than a simple string match.

Membership inference attacks: the theoretical ideal versus observed performance

Membership inference attacks, MIAs, come from the privacy attack literature, where the original goal was to prove that a specific record had or had not been used to train a model, in service of proving a privacy violation. Applied to contamination detection, the logic flips slightly: a membership inference attack tries to give concrete, defensible evidence that a specific benchmark item was or was not part of the training set.

MIAs come in three flavors depending on what the attacker can see. White-box access means the attacker has the model weights themselves. Gray-box access means the attacker sees logits and output probabilities but not the weights. Black-box access means the attacker only sees generated text, with no internal access. Black-box is naturally the most useful setting in practice, since it's the only one that matches what most people actually have when auditing a closed, proprietary model, and it requires the fewest assumptions to run.

The theory reads cleanly. The empirical results do not. Studies running membership inference attack methods against models with known training corpora, such as the Pythia family pretrained on the Pile, have found that performance is often weak, with ROC-AUC scores that in many settings remain close to chance. For context, a score representing a coin flip on the standard classifier-performance metric marks the low end and a perfect classifier marks the high end, so sub-0.6 means the attacks are only marginally more useful than not running them.

A structural cause explains it. It's structural. Most large language models are pretrained for a single epoch over an enormous corpus, so any individual example gets seen exactly once during training. MIAs tend to depend on strong, repeated memorization signatures; a model that sees each example only once during pretraining may not produce the kind of fingerprint these attacks are built to find. A single pass over trillions of tokens just doesn't leave that kind of fingerprint behind, at least not one current membership inference techniques know how to find.

Temporal signal analysis: post-cutoff performance decay as a contamination probe

Temporal signal analysis takes a different angle entirely: instead of inspecting the model or the training data, it watches how performance changes across time. The logic is straightforward. If a model performs meaningfully worse on benchmark questions published after its training cutoff than on those published before, that gap is read as evidence that the pre-cutoff questions leaked into training, since nothing else should explain the asymmetry.

This has produced real, repeated successes: post-cutoff performance decay appears consistently on verbatim benchmarks across math, coding, and reasoning tasks, and it's cheap to run since it requires nothing more than a well-dated set of questions on both sides of the cutoff.

But the method turns out to be far more sensitive to how a question is constructed than to what source material it's drawn from, which is a much bigger problem than it sounds. A large validation study built 1,643 questions from 20,277 arXiv papers, spanning 26 months, tested across 8 models in mathematics and physics. Cloze-style questions (fill-in-the-blank, drawn straight from the paper text) showed the expected clean post-cutoff decay. But LLM-generated questions built from those exact same time-stamped papers showed no decay signal at all, even though the underlying source material was identical in both cases.

Influence function analysis points to why: the act of generating a new question via an LLM transforms the base material just enough that the resulting phrasing no longer maps back to anything the model can recall from its training corpus, even when the underlying facts are the same facts. The same pattern held up in a second test: running LiveCodeBench problems, which had previously shown a clear and reliable temporal decay signal, through a transformation step powered by one language model erased that pattern almost completely.

The practical upshot cuts in two directions at once. Temporal analysis stays a genuinely useful, low-cost probe for verbatim or near-verbatim benchmarks. But the current trend toward using LLMs to synthesize new benchmark questions, specifically in order to dodge contamination, may be quietly destroying the one signal that would have revealed contamination. A benchmark can look clean under temporal analysis because its construction method happens to suppress the very decay pattern the test is looking for.

CoDeC: an in-context learning approach that reframes what the signal looks like

CoDeC (Contamination Detection via Context), from Zawalski et al. and accepted at ICLR 2026, takes a different variable entirely as its signal: how in-context learning shifts a model's confidence.

The mechanism runs like this. Feed a model some in-context examples alongside a test item, then check whether its confidence goes up or down relative to a zero-shot baseline. For a genuinely unseen dataset, in-context examples typically boost the model's confidence, since they're giving it useful task structure it didn't have before. For a memorized dataset, the opposite tends to happen: the in-context examples actually disrupt the model's established memorization pattern, pulling confidence down rather than up. That reversal, confidence rising for unseen data and falling for memorized data, is the tell that CoDeC is built to catch.

The headline number is striking: 99.9% accuracy distinguishing memorized from truly unseen data, tested against open-weight models whose training corpora were never disclosed, which is exactly the hard case every other method in this list struggles with. The validation setup paired known training data, drawn from subsets of the Pile, Dolma, and Nemotron-CC, against unseen data pulled from benchmarks and sources published strictly after each model's training cutoff. The method was run across more than 40 recent open-weight models, spanning the Llama, Gemma, Phi, and Qwen families, tested on a range of standard benchmarks including coding, instruction-following, math, and knowledge tasks.

That combination, a genuinely novel signal plus a large and varied validation set, is what makes CoDeC one of the more closely watched entries in this space heading into 2026.

Dynamic benchmark construction as a structural alternative to post-hoc detection

Every method covered so far tries to detect contamination after the fact. A separate line of work argues for skipping detection altogether and designing benchmarks that resist contamination structurally, from the start.

This reflects a broader documented shift from static to dynamic evaluation. A static benchmark is a fixed target, and a fixed target is exactly the thing that eventually gets absorbed into a crawl and memorized. Dynamic benchmarking tries to break that by keeping the target moving. One version continuously updates the dataset to stay ahead of the training cutoffs of models being evaluated, so new items always postdate whatever cutoff a given model was trained against. A second approach periodically refreshes the test data itself, replacing benchmark items over time so there's no single fixed answer key sitting around to be memorized.

A related and more aggressive proposal, contamination-resistant benchmark design from Al-Lawati et al. (arXiv 2605.19999), pushes the idea further: release benchmark data in a form that's "unlearnable," built around the asymmetry between how a transformer is trained and how it's queried at inference. The goal is a benchmark that still functions perfectly well for evaluation purposes but actively resists being absorbed usefully into a pretraining corpus.

None of this comes free. Continuously refreshing a benchmark costs real compute and real maintenance effort, on an ongoing basis rather than once. The whole approach also depends on having clean, reliable timestamp metadata, a dependency that introduces its own reliability questions when that metadata is incomplete or imprecise. Longitudinal comparisons get harder too: if the benchmark target itself keeps shifting, comparing a model's score today against its score from a year ago stops being a clean apples-to-apples comparison. And Chen et al.'s survey flags a more basic gap: there's currently no standardized way to evaluate the dynamic benchmarks themselves, no agreed criteria for judging whether a given "unlearnable" or continuously-updated benchmark is actually doing its job.

Persistent blind spots the systematic review identifies across all five method families

Line all five families up next to each other, string matching, likelihood and perplexity methods, membership inference, temporal analysis, and dynamic construction, and the pattern that emerges is not one method winning and the others losing. Nourbakhsh et al.'s review is explicit on this point: no single method holds up reliably across every contamination tier, every model-access setting, and every training stage. Each one buys precision in a narrow slice of the problem and gives up coverage elsewhere.

One gap that stands out as genuinely under-studied: instruction-tuned and RLHF'd models receive far less contamination scrutiny than base pretrained checkpoints. Almost the entire detection toolkit described above, string matching, perplexity thresholds, MIAs, temporal decay, was developed primarily with pretraining-stage contamination in mind, and none of it transfers cleanly onto the post-training stage, where the data, the objectives, and the failure modes all look different.

A separate and, in some ways, more unsettling finding concerns disclosure rather than detection. Angulo et al. (arXiv 2608.29463) built a pre-registered coding instrument and applied it to 41 published documents to see whether independent expert reviewers, working from the disclosures those papers actually provide, could agree on a paper's contamination status. Inter-rater agreement, measured by linear-weighted kappa on a per-variable basis, ranged from 0.00 to 0.35, with a median of 0.21. Compare that against a same-coder test-retest ceiling of 0.84, the level of agreement a single expert reaches with their own earlier judgment, and the gap is stark. Even trained reviewers, reading the same published disclosure, routinely disagree with each other about whether a given benchmark result should count as contaminated.

That is not a detection failure in the way the other four sections describe one. It's a reporting failure sitting upstream of detection altogether: the methods in this piece can only be as good as the disclosures they're run against, and right now, those disclosures are not consistent enough for experts to agree on what they even say.