Essay · 17 July 2026

Your Model Already Knows the Answer

The best test of an AI model is a real world problem with a known answer. It is also the easiest test to cheat. An AI model has read much of the internet, so it may have met your question, the answer or both long before you asked. A high score then hides two very different things: a model that reasoned its way to the answer, and one that is repeating something it already read. Telling those apart is the contamination problem, and it sits under a surprising share of the benchmarks the field relies on.

Imagine you want to know how reliably a model can predict whether a drug will succeed in the clinic. Rather than wait years for fresh trials, you test on ones that have already finished: take drug programmes that reached late-stage trials, hide whether each worked, and ask the model to predict. On paper it looks like a clean test, fast and scored on real outcomes, but it is not: the trials worth testing are often the famous ones, and their outcomes recur across the textbooks, reviews, news and patents the model trained on. When the model says a drug will succeed, the score cannot separate a model that reasoned its way there from one that recognised the name and recalled what happened.

Clinical trials are our field, but the trap is general: it springs on any benchmark built from resolved events, in finance, law, forecasting or anywhere the outcome is already public. These benchmarks are built from public history for a good reason: a real outcome is a better test case than any expert can invent, because it reflects true biology and the real ways a drug fails, not just what a question-writer thought to ask. Those outcomes are the gold standard we want to score against, and being public is exactly what makes them leak. So, what can we do?

The three ways the answer gets in

The answer can reach the model by three routes. This split is our own way of organising the problem rather than a named taxonomy, though each route has an established basis.

Input leak happens at test time, through the documents the model reads for the case in front of it, whether you feed those documents in or the model fetches them itself with a search tool. Any document written after the decision date can announce the very outcome the model is meant to predict, which is as good as letting it read the future (Kaufman et al., 2012).

Benchmark leak happens when a model is released after your benchmark and the model might have been trained on the benchmark set itself, memorising both the questions and answers. This is the most common way in which a benchmark decays over time. Balloccu and colleagues traced more than four million test samples leaking into GPT-3.5 and GPT-4 within a year of their release (Balloccu et al., 2024), and when other researchers rewrote a set of grade-school maths problems from scratch, several model families scored markedly lower on the fresh version (GSM1k, Zhang et al., 2024).

Outcome leak happens when the outcome is itself a public fact and the model has trained on it: a completed trial, a regulatory approval or a drug that worked is repeated across thousands of papers and filings, and the model absorbs it during training, whether in pre-training or in the later fine-tuning that follows, with no contact of any kind with your particular benchmark. The model comes to hold the result in the same way that it holds any other widely reported fact, such as the year in which a war ended. As a consequence, you can write a completely fresh test set that the model has never seen and date-blind every input you give it, and the model will still know that this well-known drug succeeded. Keeping the benchmark secret does not help either, because the answer was never contained in your file in the first place; it was already part of the model’s general knowledge of the world.

Waiting for fresh outcomes works, until the outcome is a decade away

The cleanest way to prevent any type of leak is to stop testing on history altogether and to score the model only on events that have not yet happened, so that there is simply no answer available to leak. Live benchmarks refresh their questions on a fixed schedule (LiveBench, White et al., 2024), and forecasting benchmarks like ForecastBench score models on events that have not yet resolved at the time of the test (ForecastBench, Karger et al., 2024). CT Open is a live platform that scores models on clinical trials before their results are public: predictions go in before each challenge opens, and an automated pipeline, validated against expert annotation, searches out the earliest public mention of every outcome, so a trial counts only if its answer was nowhere to be found when the prediction was made (Wang et al., 2026; ct-open.net). Trial results land all year round, and CT Open scores four times a year to match. The decision we care about is a slower one: the commitment that costs a billion dollars is taken about a decade before the readout that settles it, so put that decision to a live benchmark and your first score arrives in 2036.

So we are stuck with history, and with the outcome leak that comes with it. That leaves eight methods worth knowing. Each one either prevents or detects a leak. The split that matters more, though, is which of the three leaks a method tackles. Most of the field’s work targets the benchmark leak, but for a benchmark built from real world outcomes, the methods that matter most are the three aimed at the outcome leak.

The eight methods, grouped by the leak each one works on

Input leak: shut it at the source.

1. Date-gated tools prevent give the model a search-and-retrieval tool that refuses anything published after the decision date, for example a web search with a before-date limit. Forecasting systems already work this way, restricting retrieval to news published before the question’s decision date (Halawi et al., 2024). This is the one move that closes the input leak, and because it works at the source there is nothing left to detect afterwards. It is only as good as the dates its sources carry, so in practice it leans on scholarly sources with reliable publication dates rather than the open web.

Benchmark leak: whether the model saw your file.

2. Fresh or private test set prevent is the clean move here: test on cases no model has been trained on, or keep the set unpublished so that none can be. A number of newer benchmarks now take exactly this route, collecting fresh problems released after each model’s training cutoff so the test set cannot have been trained on (LiveCodeBench, Jain et al., 2024).

The other three methods for this leak try to detect it after the fact. They make up the most developed part of the field, and yet for a benchmark of public outcomes they are the least useful of the eight, because a model can know an outcome perfectly well without ever having seen the specific file in which you posed the question.

3. Membership inference detect asks whether the model has seen a specific piece of text, by checking how confident it is on each word (log probability of each token in the sequence), since a model is unusually fluent on text it trained on (Min-K%, Shi et al., 2023). For us it is weak on two counts: a single file barely moves the statistics of a model trained on trillions of words, so the signal is close to chance, and the method needs the model’s word-by-word probabilities, which not every provider exposes (Anthropic’s API returns none, forcing sampling-based workarounds like SaMIA, Kaneko et al., 2024). Either way it tells you only whether the model read your file, not whether it knows the answer.

4. Perturbation detect rephrases the benchmark question while keeping its true answer, then checks whether accuracy drops, since a large drop means the model was matching remembered wording rather than reasoning. GSM-Symbolic does this by swapping the numbers in maths problems and catches models that had only appeared to master them (Mirzadeh et al., 2024). It works where the reworded question still has an answer you can work out, as with those maths problems. But a real trial gives you nothing to work out: change the drug or the patient group and you have described a trial that was never run, so nobody knows how it would have ended.

5. Watermarks and canaries detect plant a hidden marker in the benchmark before release, so that a model which later reproduces it must have trained on the file; the classic is the canary string buried in BIG-bench. Like keeping a benchmark private, this only protects a benchmark you have not published yet, and does nothing about outcomes that were public for years before any model trained.

Outcome leak: whether the model already knows the result.

These last three are the ones that matter for a benchmark built from cases that have already resolved, because they aim at the outcome leak directly rather than at whether the model happened to see a particular file.

6. The cutoff gap detect compares the model’s accuracy on cases decided before its training cutoff with cases decided after it. Strong before and near-chance after is a signature of memorised outcomes, since real skill should not care which side of a date a case falls on. Researchers have run this split on financial forecasts and on knowledge tied to publication dates (Lopez-Lira et al., 2025; Roberts et al., 2023). It is a diagnostic, not a design choice, and so is different from only scoring post-cutoff cases or from date-blinding the inputs; it works only if both halves are built identically, or you measure a change in format instead of memory.

7. Recognition tests detect ask whether the model is running on recognition rather than evidence, in three variants: ask for the outcome with no supporting material, hide the entity’s name but keep the facts, or give only the name and nothing else. A model that truly reasons should be steady when the name is hidden and helpless when the evidence is removed, while a model riding on a famous name does the reverse. Blinding the name this way, in a study of oncology targets, changed about one in six of the model’s top picks while its recovery of validated targets was unchanged (Cuccarese, 2026); the paper’s caveat is that this does not make the blinded answers better, because a gene name carries real biology, but that without blinding you cannot tell how much of an answer came from the data on the page and how much from the model’s memory of the name. That figure comes from four indications with a single run each.

8. Grading the reasoning detect ignores the final answer and judges the argument behind it, checking that every claim rests on a source that existed before the decision date, because a model that only memorised the outcome can state it but cannot build a dated case for it. Our own method sits here, the Receipts Test: we ask the model to show its receipts, a dated and checkable case using only evidence available at the decision date, count a correct answer it cannot defend as a miss, track the gap between how often it is right and how often it can prove it, and let it answer “not yet knowable” instead of guessing. Grading reasoning traces against dated sources is established (process supervision, Lightman et al., 2023), and TimeSPEC breaks a prediction into atomic claims and reports how much of the decision rested on post-cutoff information (Zhang et al., 2026).

No method can prove a model is completely clean

No method can prove that a model is completely clean. Prevention comes closest: a date-gated tool is built to keep post-dated documents out, and a test set no model has trained on cannot leak through your file, so those two leaks can be designed out rather than merely caught. The outcome leak is the one you cannot close. There is no way to demonstrate that the weights gave no help at all, and you can only catch the cases where they plainly did. Each method also has its own blind spot, and even the two prevention moves are not airtight: date-gating is only as reliable as the publication dates it trusts, and a fresh test set does nothing about an outcome the model already absorbed from the wider literature. This is not only our experience. An independent effort to train a language model on pre-1931 text alone, with temporal leakage named as its most important objective and a purpose-built anachronism filter enforcing it, still produced a model that knows the Second World War happened (Talkie, Levine et al., 2026). A decontamination filter leaks even when keeping the data clean is the whole point. Membership inference is weak at scale, and watermarks protect the file rather than the public outcome. Perturbation depends on an answer that can be regenerated, which a real trial does not provide, and the cutoff gap collapses into a measurement of formatting if the question format is allowed to drift across the boundary. Blinding the name flags a shift that genuine biology would also produce, and grading the reasoning can be defeated by a fluent argument constructed backwards from an answer the model already knew. For these reasons we do not rely on any single method, but run several of them together and pay particular attention to the cases where they disagree, which is what turns a number one cannot trust into a number one can defend. For a benchmark built from resolved outcomes, the three methods worth their weight are the ones aimed at the outcome leak.

We are building a benchmark that must survive every leak

The benchmark we are building for drug-development outcomes has to survive every one of these leaks at once, because the cases that are most worth scoring are precisely the famous ones that a model is most likely to have memorised, and waiting a decade for fresh outcomes is not a realistic option.

We are not alone here. CT Open avoids the outcome leak by going live (Wang et al., 2026). TimeSPEC grades the trace (Zhang et al., 2026). MemGuard-Alpha filters memorised signals out of a financial backtest (Roy and Roy, 2026). All three score whether the prediction was right; none scores why the trial turned out as it did. A verdict is not something a drug developer can act on. The mechanism is, because the mechanism is what you would change: the target, the cohort, the biomarker you select patients on. So that is what we are scoring, on a bet we would rather test than assert, that a model can memorise a verdict far more easily than a mechanism it never had to reason about.

If you build or evaluate models on historical data, in drug discovery or anywhere else, tell us what catches memorisation and what slips through, especially where it contradicts what we have written here. And if you hold outcome data with accurate dates attached, that is the raw material a benchmark of this kind is made of. We would love to hear from you!