What 1.2 Million FX Strategies Taught Me About Overfitting

By 2016, something had quietly stopped working. In 2014 and 2015, running a standard strategy search across a handful of major FX pairs tended to turn up systems that looked genuinely strong — clean equity curves, comfortable profit factors, the kind of results that make you want to risk real money on them. A year or two later, the same workflow, the same pairs, the same generation settings, stopped producing anything I trusted. The backtests still looked fine. That was exactly the problem: a good-looking backtest had never been proof of anything, and for two lucky years I had let myself forget that.

I could have told myself a story about why the edge had gone — markets got more efficient, retail flow changed, brokers tightened their books. Instead, in 2019 and into 2020, I did something less comfortable than trusting my own instinct: I ran a large empirical study. I generated and evaluated roughly 1.2 million FX strategies and tracked, for every one of them, not just how it performed in the backtest but how it performed on data the generation process had never seen — its true out-of-sample result, measured via walk-forward out-of-sample testing (WFOS). The question I wanted answered was narrower than “why did my intuition stop working,” and more useful: across a huge population of strategies, what actually predicts whether a backtest survives contact with unseen data, and what turns out to be irrelevant?

This article is the consolidated answer. I have written elsewhere about individual pieces of this puzzle — Monte Carlo stress testing, walk-forward validation as a discipline — but this study is what sits underneath all of it. It is the reason I trust those later techniques at all: they kept showing up as decisive factors in a dataset large enough that I could stop arguing from intuition and start arguing from evidence. Four factors came out of that dataset with enough consistency that I now use them as a standing filter on everything my generation process produces, inside StrategyQuant X, the platform the whole study ran in.

What “true out-of-sample” actually means here

It is worth being precise about the target, because it is the one part of this article that is not a matter of interpretation. A backtest measures a strategy’s performance on the data used to build and select it, which means it will always flatter the strategy somewhat — the generation process searched, by construction, for the parameters that fit that exact data best. True out-of-sample is a different, harder measurement: performance on a segment of history the strategy never touched during generation or optimisation, walked forward through time so the “future” segment is always genuinely unseen at the point the strategy was built. Every finding below is a statement about that second number, not the first. A factor that makes the backtest prettier and does nothing to true out-of-sample is noise wearing a costume, and the entire point of the study was to stop being fooled by that costume.

1. Complexity — the sweet spot between naive and overfit

The finding: strategy complexity — the number of building blocks and conditions strung together in the entry and exit logic — has a real relationship with generalisation, and that relationship is not a straight line. It is a curve with a sweet spot, roughly 4 to 6 building blocks, and strategies built on either side of that range performed worse where it counts: their walk-forward out-of-sample median net profit and profit factor were both weaker.

The overfit side of that curve is the intuitive half. Every additional building block — another indicator, another filter, another AND/OR condition — is another degree of freedom the generation process can bend to fit the noise in the in-sample data. Stack enough of them and you can describe almost any historical price path in exhaustive, specific detail, the same way a high-order polynomial can be made to pass through every point of a noisy scatter plot. It looks like an excellent fit. It has learned nothing that will still be true next month.

What surprised me more was the other side of the curve. I went into the study more or less assuming that simpler was strictly better — Occam’s razor applied to trading logic — and expecting the data to show a steady decline in performance as complexity increased from the simplest possible logic upward. It did not. Strategies built on very little logic at all generally underperformed too, and for a sensible reason: logic that thin usually is not describing an edge, just triggering on noise with almost no selectivity. The strategies that generalised best were not the simplest strategies in the population. They were the strategies with just enough structure to describe a real, repeatable market behaviour, and not one condition more.

That reframed how I think about complexity entirely. It is not a dial you turn down as far as it will go. It is a target you aim for, and 4 to 6 building blocks is where I now expect that target to sit for a typical FX system, before I even look at anything else about it.

2. Trade count — why a thin sample is not evidence

The finding: strategies with too few trades in their track record are not conservative or cautious — they are statistically meaningless, and the data pointed to roughly 30 to 80 trades per year as the band where a backtest starts to become a trustworthy estimate of anything at all.

The logic here is not really about trading; it is about sample size. A strategy that produces very few trades a year can post an excellent profit factor purely because a small number of favourable outcomes landed the right way, the same reason a small run of coin flips can look wildly biased even from a fair coin. There is nothing in a track record built on a bare handful of trades for a walk-forward test to meaningfully validate, because there is barely a pattern there to validate in the first place. It is not that low-frequency strategies are always bad; it is that you cannot tell, from the backtest alone, whether a low-frequency strategy is good. The information simply is not in the data yet.

Push trade count up into a healthy range and that changes. With enough trades, results start to reflect the underlying behaviour of the logic rather than the idiosyncrasies of a handful of individual outcomes, and the backtest becomes something you can actually reason about rather than something you have to take on faith. That is the entire justification for treating 30 to 80 trades per year as a range worth respecting rather than an arbitrary threshold — it is roughly where a track record stops being an anecdote and starts being a sample.

The more useful discovery was how this factor interacted with the first one. Cutting the population by complexity alone helped. Cutting it by trade count alone helped. Cutting it by both at once — simple logic, built on 4 to 6 conditions, generating a healthy 30-to-80-trades-a-year cadence — produced a noticeably stronger pocket of true out-of-sample performance than either filter did on its own. Neither factor was a proxy for the other; they were catching different failure modes, and strategies that passed both were measurably healthier than strategies that only passed one.

3. Spread robustness — the fastest way to expose a fragile edge

The finding: re-running a strategy against a wider spread and comparing the result to its normal-spread performance — a comparison I think of as a spread ratio — was one of the single strongest fragility tells in the whole dataset. Strategies whose spread ratio held up above roughly 0.8 kept working; strategies whose edge collapsed as spread widened were, almost without exception, exploiting something that was never really there.

The mechanism is straightforward once you see it. A strategy’s backtest is priced against the historical spread in the data, which for most retail FX data means a spread that is thin, stable, and considerably kinder than what you will actually pay through periods of volatility, thin liquidity, or a less generous broker. If a strategy’s edge is only marginally larger than that spread, it does not take much of a change in trading cost to erase the edge entirely — and a generation process that does not care why a rule works will happily settle on a rule that only works because the built-in cost happened to be unusually forgiving. Widen the spread and you are not testing something exotic. You are asking the single most basic question a trading cost can ask: is there an edge here, or was the edge actually a subsidy from unrealistically good fills?

What made this factor stand out from the others in the study was how cleanly it separated the population. Complexity and trade count both describe the shape of the strategy’s logic — reasonable proxies for overfitting risk, but proxies all the same. Spread ratio is closer to a direct measurement: it re-tests the exact same rules under a harder cost environment and reports back whether the edge survived. A strategy can be simple and trade often and still fail this test, because none of that guarantees the edge was ever bigger than the cost of trading it. When it does fail, it fails for an easy-to-understand reason, which is exactly why I treat it as close to a veto rather than just one input among several.

4. Selection metrics — which ones actually predict what happens next

The finding: not every number on a backtest report carries the same amount of information about true out-of-sample survival, and ranking or filtering strategies on the wrong ones is a quiet, common way to sabotage an otherwise sound selection process.

I looked at how a handful of common performance metrics — profit factor, winning percentage, annual percentage return, a stability measure (SQ3 Stability), win/loss ratio, return-to-drawdown ratio, and Sharpe ratio — correlated with what a strategy actually went on to do out-of-sample. They did not all carry the same weight. Stability, return-to-drawdown, and profit factor turned out to be meaningfully more predictive than the others: strategies that scored well on those tended to keep scoring well on unseen data. Annual return and raw win rate were the opposite — easy numbers to be seduced by, and weak predictors of anything that mattered next. Sharpe ratio and win/loss ratio sat in between: informative, worth glancing at, but nowhere near as decisive as the top tier.

The reason, once I looked at it from the strategy’s side rather than the report’s side, is not mysterious. Annual return and win rate are both outcome measures — they describe what happened without much regard for how it happened, which means a strategy can post an excellent number on either by taking on a shape of risk that a generation process finds easy to stumble into: a long, quiet run interrupted by one outsized winner, or a high hit rate propped up by a run of trades with catastrophic downside that simply did not fire during the backtest window. Stability and return-to-drawdown are harder to fake that way, because they measure the character of the equity curve rather than its endpoint — how consistent the climb was, how much pain was required to earn the return. A strategy cannot back into a good stability score by accident nearly as easily as it can back into a good win rate.

The practical upshot is that the metric you sort by is itself a decision about what kind of overfitting you are willing to tolerate. Sort a large population of strategies by annual return and you will surface the ones that got luckiest. Sort by stability and return-to-drawdown, with profit factor as a sense check, and you surface the ones that behaved consistently — which, it turns out, is a far better proxy for “will keep behaving consistently” than any single-number description of past profit ever was.

Stacking the filters

None of these four findings is a silver bullet on its own, and I do not use any of them that way. What actually works is treating them as sequential filters applied to a large population of generated strategies, each one narrowing the pool and lifting the quality of what is left.

• Baseline — the full, unfiltered population straight out of the generator. This is what most people evaluate strategies from, usually by eye, and it is the weakest starting point there is.
• Filter to complexity 4–6 — remove everything that is either too thin to describe a real behaviour or complex enough to be describing noise in exhaustive detail. The surviving pool already generalises better than the baseline.
• Add a trade-count filter of 30–80 trades per year — remove the strategies whose track record is too short to trust, on top of the complexity filter. The pool shrinks further, and its true out-of-sample quality improves again.
• Add a spread ratio above roughly 0.8 — remove whatever is left that only works because of unrealistically favourable execution costs. This is usually the filter that does the most damage to a population that looked healthy a step earlier, which tells you how many “robust” strategies were quietly living on thin cost margins.

What is left after all three filters is a small fraction of where you started, and it is a genuinely better population — not because any individual strategy was improved, but because everything that survives has already demonstrated it does not depend on the specific failure modes the study identified. From there, the fourth finding decides which of the survivors gets built out further: rank what is left by stability and return-to-drawdown rather than by raw return, and you are choosing among strategies that were already filtered for structural soundness, using the metrics that this dataset actually proved matter.

Every step in that sequence cost something — fewer strategies, more rejected work, less flattering headline numbers along the way. And at every step, the out-of-sample quality of the surviving pool went up. That trade is the entire argument of this article in one sentence.

The point

You cannot optimise your way out of overfitting. Every optimisation pass, by construction, makes the backtest look better, and a better-looking backtest is not the same thing as a better strategy — that gap is precisely what this whole study measured. But you can select your way towards robustness, if you are willing to filter on the factors that actually predict generalisation instead of the ones that are merely satisfying to look at.

That is the real shift this study produced in how I work. I no longer trust a strategy because its equity curve is smooth, or because its win rate is high, or because I have a good feeling about the logic. I trust it because it kept a sensible amount of structure, traded often enough to mean something, held up when I made its costs worse, and scored well on the metrics that a population this large proved actually matter. None of that is intuition. It is a playbook, built from evidence, and it is the standard I now hold every strategy to before it gets anywhere near my own capital or a client’s.

If you want your own strategy population put through this same filtering process before any of it sees real money, get in touch.