Evidence-Based Technical Analysis: The Bias That Makes Backtests Lie

Every chart reader has had this moment: you look back at a turning point and it is obvious — the double top, the bearish divergence, the trendline that broke the day before the reversal. The pattern practically points at itself. Now do it forwards. Strip the labels off, stop the chart at today’s bar, and ask a room full of technicians to mark where that same pattern is forming right now. You will get a room full of confident, mutually exclusive answers, and most of them will be wrong by Friday.

That gap — between how obvious a pattern looks in hindsight and how useless it is as a forward-looking rule — is the entire subject of David Aronson’s Evidence-Based Technical Analysis, one of the few trading books I would call essential rather than merely useful. Aronson, a veteran market technician, published it in 2006, and the argument has aged better than almost anything else from that period, because it was never really about which indicator to use. It was about what counts as evidence at all.

His starting distinction is blunt. Most of technical analysis, he argues, is subjective: patterns and setups that depend on a human’s judgement to identify, that shift shape depending on who is looking and when, and that can always be rationalised after the fact because they were never precise enough to be wrong in the first place. A head-and-shoulders top is whatever a sufficiently motivated analyst decides it is. That is not analysis — it is storytelling with a chart as the prop, and it “works” only in the retelling.

The alternative is objective technical analysis: rules stated precisely enough to be coded, applied mechanically to historical data without any judgement calls along the way, and — critically — subjected to a genuine test of statistical significance before anyone believes them. Objective rules can be wrong in a way you can actually catch. That, to Aronson, is what makes them worth anything at all.

A rule like “buy when a shorter moving average crosses above a longer one” is objective in this sense — not because moving averages are magic, but because the rule leaves nothing to interpretation. Anyone coding it independently, from the same data, would produce identical trade signals. That is the low bar objective analysis has to clear before statistical testing can even begin, and it is a bar most textbook chart patterns never reach.

The bias hiding inside every backtest

Here is where the book earns its place on the shelf of anyone who codes rules for a living rather than eyeballing charts, because coding a rule precisely does not, by itself, solve the problem Aronson opened with. It just moves the danger somewhere subtler: data-mining bias.

Data mining, in the plain sense Aronson uses the term, is simply the practice of extracting patterns, rules, models and functions from a large database — trying many candidate ideas against historical data and keeping the ones that perform. There is nothing improper about that on its own; it is how most useful discovery happens, in trading and everywhere else. The bias creeps in at the selection step. When you test a large number of rules against the same data and report only the best one, that rule’s backtested performance is no longer an honest estimate of what it can do going forward. Some portion of it — sometimes all of it — is simply the residue of noise that your search process was, in effect, built to find. None of this requires bad intent. A careful, honest developer following a careful, honest process can still end up presenting a lucky rule as a discovered edge, simply because nothing in an ordinary backtest report distinguishes a real pattern from a well-disguised coincidence.

The mechanism is not exotic. Give a search enough purely random rules and a long enough stretch of history, and a handful of them will, by chance alone, produce a spectacular-looking equity curve over that specific data. They are not signalling anything real about the market; they are the predictable statistical debris of having looked hard enough. If your process selects the best-performing rule out of a large field, you have — deliberately or not — built a machine for finding exactly that debris and presenting it back to yourself as an edge. This is why the strategy that dazzled in-sample so often goes flat, or worse, the moment it meets data the search never touched: a meaningful share of its apparent edge was never really there.

What decides how much bias you are carrying

Aronson does not leave this as a vague warning; he sets out what actually drives the size of the bias, and three factors matter most.

How many rules you tried. This is the most intuitive driver: the more candidate rules you test, the more chances you give randomness to produce an accidental standout, and the larger the expected gap between that standout’s backtested performance and its true future performance. It is the same logic behind any multiple-comparisons problem — buy enough lottery tickets and one of them wins, but the win tells you nothing about your system for picking numbers.

How correlated those rules were with each other. This one is easy to miss, and it is why the raw rule count alone is not the whole story. If the variants you tested are really just the same idea wearing slightly different parameters, you have not run many independent trials — you have run a handful, dressed up in repetition, because a small move in a single parameter rarely uncovers a genuinely new pattern in the data. Bias tracks the number of effectively independent tests, not the number of buttons you clicked. A search that covers meaningfully different rule structures is a more powerful search — and, uncomfortably, a more bias-prone one, precisely because each new structure is a genuinely fresh roll of the dice.

How much randomness is in the outcomes you are searching over. Noisier data gives chance more room to work with. A rule tested against a choppy, low-signal series has a wider range of possible “great-looking” results available purely by luck than the same rule tested against a series with a strong, persistent underlying signal. The noisier the pond, the more spectacular the fish you can accidentally reel in.

Put together, these three explain something every systematic trader eventually notices: the best strategy in a large, diverse databank, tested over volatile, noisy markets, is exactly the result you should trust least at face value — even though it is the one that looks most impressive on the screen.

None of this shows up in a standard backtest report. A performance summary tells you what the winning rule did; it says nothing about how many rules lost, how similar they were to one another, or how much of the outcome space was noise to begin with. That context has to be sought out deliberately — it will not appear on the equity curve.

Testing the search, not just the winner

Aronson’s remedy is not to search less. It is to test the search itself, rather than evaluating the winning rule as though it were the only idea anyone had ever tried. This is the point of rigorous inferential statistics here: instead of asking “is this equity curve good?”, you ask “given how many rules I effectively tried, and how noisy this data is, how good a result would pure chance have produced anyway — and did my result clear that bar by a meaningful margin?”

This is the territory of bootstrap and Monte Carlo permutation methods, and of tests such as White’s Reality Check, which Aronson discusses as exactly the kind of tool this problem calls for: approaches that build up a picture of what the best result from your search would look like if there were no genuine edge anywhere in the rule set — purely from chance and the scale of the search — and then check whether your actual best result stands out from that picture or simply blends into it. A rule that cannot clear that bar has not demonstrated an edge, however good its curve looks. It has demonstrated that you searched.

What this means for how I develop strategies

I did not read this book as history. I read it as a description of my own workflow, because a modern strategy generator like StrategyQuant X does not test one rule and report back — it generates and ranks enormous numbers of candidate strategies across a vast space of rule structures, which is exactly the data-mining process Aronson describes, running at industrial scale. That is not a criticism of the tool. Breadth of search is genuinely powerful, and I have no interest in going back to hand-picking a handful of setups and hoping. But Aronson’s book is the clearest statement I know of for why that power has to be handled deliberately, not trusted on sight. A few habits follow directly from it:

• Treat the top of the databank as a hypothesis, not a conclusion. The best-ranked strategy out of a large population is, structurally, the most likely place for data-mining bias to be concentrated. Impressive rank is where you start being careful, not where you stop.
• Reserve data the search never touches. Not a chart you glanced at once, and not a period you happened to skim while building — a genuinely held-out stretch of history that played no role in generation, ranking or selection, so that testing on it actually means something.
• Account for how hard you searched. The size and diversity of the population you generated changes how good a result needs to be before it is believable. A standout from a small, narrow search and a standout from a huge, varied one are not the same claim, even when their equity curves look identical.
• Demand a significance test, not a good-looking curve. A smooth backtest is not evidence on its own; it is the starting point for the question Aronson insists you ask — could a search this size have produced this result by chance alone? — using the kind of Monte Carlo or permutation-based approach built for exactly that question.

None of this makes mass generation less useful. It makes the results I choose to trust considerably more honest, which is the entire point of running the search in the first place.

The point

The chart pattern that only reveals itself in hindsight and the top-ranked strategy pulled from a databank of countless variants are, statistically, the same animal wearing different clothes. Both are the product of a search finding what it was always going to find somewhere, and both borrow their credibility from a story told after the fact rather than a test that could have failed. Aronson’s contribution was to give traders a way to tell the two apart — not by trusting the curve, but by asking what chance alone, given the scale of the search, would have handed you anyway. That question does not get less relevant as generators get more powerful. It gets more urgent.

If you are generating strategies at scale and want an honest read on how much of your top performer’s edge would survive that question, I would be glad to take a look.