A Backtest Is a Statistical Test, Not a Proof

The most expensive misunderstanding in retail algorithmic trading is not a bad indicator or a mistimed exit. It sits further upstream than that: treating a good backtest as proof. A backtest is not proof of anything. It is a statistical test — one sample, drawn under specific conditions, carrying its own error rate — and like any statistical test, it can hand you a false positive and dress it up as an edge.

That distinction barely mattered when a strategy was something a person built by hand, one rule at a time. It matters enormously now, because that is not how most systems get built any more. Strategy generators — StrategyQuant (affiliate link — if you buy through it I earn a commission at no extra cost to you) and its kind — are mass-generation machines: describe the behaviour you are curious about, point the software at a data set, and it will hand you back thousands of complete, fully specified trading systems, each with its own backtest, before you have finished your coffee.

That capability quietly changes which problem you are solving. Build one strategy by hand and curve-fitting is the risk everyone warns you about — too many parameters bent to fit the noise in one data set. Generate 10,000 strategies and curve-fitting inside any single one of them stops being the main danger. The dominant risk becomes selection overfitting: picking the luckiest of the 10,000 and mistaking its luck for skill. That is strategy development recast as a multiple-testing problem, and it is a different failure mode with a different fix — one most retail traders are not defending against, because they are still braced for the first one.

The arithmetic of luck

The reason selection is so dangerous is not psychological, it is arithmetic. The expected value of the best result out of N tries rises with N, even when every single try is pure noise with no edge inside it anywhere. Generate 10,000 random strategies against real price data and the best backtest in that batch will look brilliant — guaranteed, not because anything real was found, but because 10,000 places were searched and only the best one got reported back.

It is the same trick as running enough clinical trials on a drug that does nothing: run enough independent trials and chance alone will eventually produce one where the drug appears to work. Nobody treats that single trial as proof; the discipline of clinical statistics exists specifically to catch that inference. Strategy generation runs the identical experiment, at a scale medicine would never tolerate, and most of us report the winner without asking how many losers it beat.

Which is why the right question is never “is this backtest good?” A good-looking backtest is the least surprising output a generator can produce — give it enough tries and one is inevitable, whether or not any edge exists underneath it. The only question worth asking is whether this backtest is better than the best of 10,000 lucky accidents would look, by chance alone. Everything downstream of that question is what actually separates an edge from a well-dressed coincidence.

What 1.2 million strategies taught me

I did not arrive at this by reading someone else’s paper. I arrived at it by generating and evaluating roughly 1.2 million FX strategies and tracking what happened to each one once it left the flattering territory of the backtest and met data the generation process had never seen. What 1.2 million FX strategies taught me about overfitting is the detailed version of that study; the short version is blunter. Backtest performance on its own predicted almost nothing about which strategies would survive out-of-sample. If the number you are proudest of does not predict what happens next, it was never evidence — it was the artefact of having looked in enough places.

The two rules everything else follows from

Accept that strategy development is a multiple-testing problem and almost every piece of validation discipline I now use falls out of two rules.

Any data you select on is consumed. The moment you use a slice of data to pick winners — to rank, to filter, to eyeball and discard — that slice stops being evidence. It has been spent. Test enough variants against the same stretch of history and keep the best, and that history can no longer tell you anything honest about the winner, because the winner was chosen specifically because it flattered that history. This is why you always need one more untouched slice than you think you do — something the selection process never touched, held back until after the decision is made, not before it.

Count the search and pay for it. Keep a record of N — everything the process generated, including every strategy you discarded, not only the ones that made the cut. Then correct the winner’s statistics for the size of that search: deflate the Sharpe ratio, and demand enough clean out-of-sample data to support the claim. Survivor-only records make this arithmetic impossible. Throw away the discards and there is no way to reconstruct what best-of-N actually means — you have deleted the exact information the statistics need in order to correct for the luck.

What this changes in practice

Take those two rules seriously and a handful of concrete habits follow, not as nice-to-haves but as consequences:

Deflate the Sharpe ratio by the size of the search that produced it, instead of reporting the raw number — see the deflated Sharpe ratio in practice.
Measure the probability that your selection process overfits (PBO) and fix the pass/fail threshold before you look at the result, not after — that is my pre-registered GO/NO-GO gate.
Stress-test whatever survives against alternative histories and messier execution, because a strategy that only works along one flattering path through the past is not robust — see Monte Carlo robustness testing.
Give survivors incubation time on paper before real money follows them, because forward time is the one kind of out-of-sample data no selection process can have already spent.
Judge the generation process by its whole population, not its best offspring — a generator that produces one dazzling strategy out of thousands of attempts has told you more about its search space than about that one strategy.

None of this is my invention. The deflated Sharpe ratio and the probability of backtest overfitting both come from the research of David Bailey and Marcos López de Prado, and their colleagues — I have simply built the habits above around applying that work to strategy generation specifically.

Research is a pipeline, not a treasure hunt

There is a reason all of this matters beyond the moment of selection. My own testing has found that strategies, on average, start losing performance after around two years live — markets are not stationary, and whatever regularity a strategy captured erodes as the world that produced it changes. That finding reframes what a validation process is for. It is not a one-time filter you run to find The Strategy and then retire undefeated. It is a pipeline that has to keep running: continually producing new candidates, continually retiring the ones whose edge has decayed, because the survivor you validated two years ago is already, on average, going stale. Research is not a hunt for one trophy. It is a production line, and the discipline above is what keeps its output honest.

The point

None of this makes strategy development slower for its own sake. It makes the difference between a track record and a lucky draw legible, which is the only way to know which one you are holding before the market finds out for you. A backtest is a statistical test. Treat it like one — count what you searched, spend your data on purpose, judge populations instead of trophies — and whatever survives will have earned that survival rather than won a lottery it did not know it was entered in.

That discipline — auditing the search behind a strategy, not just the strategy in isolation — is exactly what I offer as an independent validation audit.