My GO/NO-GO Gate: Pre-Registration for Traders

The most dangerous moment in strategy research is not the generation run itself. Set a template loose and most of what comes back is obvious junk — you discard it without a flicker of temptation. The dangerous moment comes later, when one candidate finishes just short of your bar. Close enough that you can see what you wanted it to be. And every researcher I have ever met, myself included, hears the same whisper at that point: the threshold was arbitrary anyway — lower it a touch, just this once.

It is never just once. Science ran headlong into this exact disease — in clinical trials, in psychology, in every field where a researcher gets to see the data before deciding what counts as success — and eventually found a cure for it. It is called pre-registration, and I have built a trading equivalent of it, because I do not trust my own judgement any more than I would trust a researcher who got to pick the finish line after watching the race.

What pre-registration actually means

Borrow the idea straight from clinical trials: before a drug trial enrols its first patient, the researchers file a document stating exactly which tests they will run, exactly which thresholds count as success, and exactly what happens if the trial fails. Once that document is filed, it is not up for negotiation. If the primary endpoint misses, it missed — nobody gets to go rummaging through the secondary endpoints afterwards, find one that happened to clear the bar, and quietly report that instead.

The logic carries over to strategy research without needing to be forced. Post-hoc threshold shopping — running the numbers, disliking the verdict, and reaching for a kinder statistic — is overfitting, just relocated. You already went to great lengths to stop your strategy from overfitting the data. Negotiate with the results afterwards and you have simply overfit yourself instead, which is the same failure in different clothes.

So I did what a trial registry does. I wrote the protocol before I had results in front of me to be tempted by, and I froze it.

What the protocol fixes in advance

The document nails down everything that is otherwise dangerously easy to adjust after the fact:

The statistical thresholds a population has to clear, decided with no results on the table to negotiate against.
How the search size is accounted — every strategy the generation process produced counts, discards included. A population is measured by everything it took to find its survivors, not by the survivors alone.
The failure protocol — precisely what I am allowed to change after a NO-GO. The strategy grammar and the generation setup are fair game. The judge is not, ever.
The data schedule — which slices of history exist, what each one may be used for, and the conditions under which the final, untouched “vault” slice may be opened at all.
Incubation rules — how long a strategy has to survive on paper before it goes anywhere near live capital.
Portfolio rules — how surviving strategies may be combined once they are through the gate.

None of that makes exciting reading, and that is rather the point. A protocol written in a hurry, in the afterglow of a strategy you already love, is not a protocol. It is a rationalisation wearing a template.

The gate itself

Every population I generate — inside StrategyQuant (affiliate link — if you buy through it I earn a commission at no extra cost to you), the platform I build in — has to clear four checks, in order, before anything downstream of it matters.

Coverage validity comes first, because it can invalidate everything that follows before a single statistic gets computed. Was the full generated population exported, discards included, or only the survivors that already looked promising? Survivor-only in means invalid out. No amount of clever statistics downstream can rescue a sample that was filtered before it ever arrived — you would only be measuring the filter.

The Deflated Sharpe Ratio comes next. It deflates the best result in the population by the size of the search that produced it, using the effective number of independent trials rather than the raw count — correlated near-duplicates are not separate tries, and counting them as such flatters the result — plus a false-discovery-rate correction applied across every candidate, not just the winner. I go into this properly in the Deflated Sharpe Ratio in practice; the short version is that it answers one question honestly: given how hard I searched, how surprised should I really be by the best result I found?

The Probability of Backtest Overfitting is the third check, computed through combinatorially symmetric cross-validation. The returns history gets split into many combinations of in-sample and out-of-sample partitions; strategies get ranked in-sample on every split; and I measure how often that split’s in-sample winners land in the bottom half once you look out-of-sample. PBO is not really a verdict on any single strategy — it is a verdict on the selection process that produced it. If a process’s in-sample winners keep collapsing out-of-sample, the process is what is overfit, not merely the strategies it happened to produce.

A verdict, on the record, closes the gate, and it is neither optional nor informal. GO or NO-GO gets written to a verdict file, and every run — pass or fail — gets appended to an append-only ledger, together with a fingerprint, a hash, of the exact configuration that produced it. History cannot be quietly rewritten. If I ever loosen a threshold, that change is dated, hashed, and sitting in the ledger forever, next to every verdict it was and was not used to produce.

Separation of powers

I keep the factory and the judge deliberately apart. The factory is the generation side — the process that builds and searches over strategy populations. The judge is the gate above it. The factory can call the judge and read its verdicts. It cannot edit its thresholds, cannot see inside its statistics, cannot negotiate.

That is not an org-chart nicety. It is a statistical control. The entity being graded must not hold the red pen — the moment generation can adjust the standard it is measured against, the standard has stopped measuring anything.

Two rules with teeth

Everything above is machinery. These two rules keep me honest when it hands back an answer I do not like.

If it almost passed, it failed. There is no category between GO and NO-GO for “so close it basically counts.” A near-miss gets exactly the same treatment as a strategy that missed by a mile: I am allowed to revise the strategy grammar or the generation setup that produced it. I am never allowed to revise the judge that graded it.

Judge populations, never the best offspring. A generation template is graded on the statistics of its entire population, not on whichever strategy happened to come out on top. Pointing at your single best result and calling the template validated is the multiple-testing trap wearing a different coat: somewhere in a large enough population, something was always going to look good by chance alone. The template earns credit only when the population as a whole clears the bar — not when one lucky member of it does.

Ground truth

I did not take the gate on faith. Before trusting it with real research, I tested it against answers I already knew. Fed it a population built from pure noise, and it returned NO-GO. Planted a genuine edge inside an otherwise noisy population, and it returned PASS — and pointed at the exact strategy I had planted, not a neighbour. Exported survivors only, hiding the discards the way a tempted researcher might, and it returned INVALID before the statistics ever ran. Three answers I already knew, and the gate produced all three without being told which was which.

The point

This is exactly what my Strategy Validation Audit is. Your databank runs through my gauntlet, not a softer version assembled for the occasion. The thresholds were fixed before I ever looked at your results, exactly as they are fixed before I look at my own. What you get back is a written GO or NO-GO and the reasoning behind it — not a reassuring conversation, a verdict with a hash attached to it.

I have made the underlying case for treating a backtest with this much scepticism in the manifesto, and covered the stress-testing half of my own process in Monte Carlo robustness testing. Pre-registration is the piece that sits above both of them — not one more test added to the pile, but the discipline that stops me from quietly failing all the others once I dislike what they are telling me.

If you want your own strategy population through the same gate, fixed thresholds and all, see how the audit works or get in touch.