How I Evaluate AI Agents That Build Trading Strategies

Every AI demo you have ever watched is a backtest. Someone built something, ran it against a small set of cases they picked themselves, and showed you the ones where it worked. It looks brilliant, because it was tuned to look brilliant on exactly that evidence — and from the demo alone, you have no way of knowing whether any of it holds up on a case nobody showed you.

Quant traders solved this exact epistemology problem decades ago. We call it overfitting, and the market taught this industry, expensively, to distrust an in-sample result before anyone here would put real capital behind it. Almost nobody building on large language models applies that same discipline to their own agents. I do, because I did not arrive at AI from a computer science background. I arrived at it from trading, where trusting a result you have not tested out-of-sample is how you lose money you cannot get back.

The same disease, the same cure

Watch how most “AI agent” demos actually get built. Someone writes a prompt, runs it against a handful of examples, and when the output looks wrong, tweaks the prompt until it looks right. Do that over and over and you have a demo that reliably impresses a room. You also have something you probably did not set out to build: a prompt curve-fitted to your dev set, in exactly the sense that a strategy dragged through round after round of optimisation gets curve-fitted to its backtest window. It “generalises” the same way an overfit strategy generalises — which is to say, you do not actually know, and you will not find out until it meets a case it has never seen.

The cure is identical too, because the disease is identical. Ground truth in place of an opinion. Held-out data the thing under test never got to see. Repeated measurement instead of a single lucky run. Criteria fixed before you look at the result, not adjusted afterwards to fit whatever came out. None of that is exotic — it is the same argument I made in the manifesto on the trading side: a backtest is a statistical test, not a proof. I am simply applying it one layer up now, to the agents that do some of the strategy-building for me.

The four pillars

This is what that discipline looks like in practice, on my own agents, every time I change a prompt or a workflow and want to know whether the change actually helped.

• An oracle, not an opinion. I never grade an agent’s output by eyeballing it, and I never rely on another model’s opinion alone when objective ground truth is available instead. Every strategy my agent builds is loaded into a running StrategyQuant (affiliate link — if you buy through it I earn a commission at no extra cost to you) over its REST API, and the platform itself parses the file and accepts or rejects it. A LOAD PASS is a fact, not a vibe — the strategy either compiles inside a real trading platform or it does not, and no amount of confident prose changes that verdict. The newer layer of this goes further, driving the backtester itself headlessly to read back real trades and statistics, so the check moves from “is this real” to “is this any good.”
• Held-out, out-of-sample prompts. I keep a set of real user prompts that neither I nor the agent has ever tuned against. An improvement only counts if it generalises to those unseen prompts — a train/test split, applied to prompt engineering instead of feature engineering. Tuning and testing on the same cases is the same sin as optimising and validating a strategy on the same slice of history: it will reliably look like progress, and it will just as reliably mean nothing.
• Denoised measurement. Language models are stochastic. The same agent, given the same task, will vary from run to run — sometimes a little, sometimes enough to flip a verdict from pass to fail. A single measurement proves nothing, so I refuse to trust one. Speed and cost get measured across three passes and reduced to medians before I let myself believe a change actually helped.
• A/B with a no-regression rule. A candidate version of an agent runs the same case suite as the version it is trying to replace. It only ships if the denoised comparison improves the metric I am targeting, with zero loss of correctness anywhere else in the suite. A faster agent that breaks two cases it used to pass has not shipped an improvement — it has just moved the cost somewhere I had not thought to measure yet.

The numbers behind the discipline

I would rather show the receipts than repeat the pitch. Here is what that discipline currently reports about my own agent, and I trust every number below because of how it was produced, not despite it.

On a corpus of 30 unseen, real-user prompts — held out, never tuned on, never seen during development — the agent built 30 out of 30 strategies that loaded into a live StrategyQuant, single-pass, at a median of about 47 seconds each, start to finish. That is fast enough that testing a trading idea costs almost nothing, which changes how freely you are willing to test one in the first place. In a separate breadth battery designed to stress the range of what the agent can build rather than its speed, 100 out of 100 strategies traded. And when I turned this same discipline on the agent itself, trying to make it cheaper to run, one optimisation cycle cut its cost by 37% while a 48-case eval suite stayed perfect — not mostly held up, perfect, before and after. Cheaper, and not one case worse.

That last result is the one I trust most, precisely because it is the least dramatic. “Same suite, before and after, zero regressions” is not designed to impress anyone watching a demo. It is designed to be true.

Why this matters beyond trading

Most AI products ship on vibes: a curated demo, a confident narration, and the hope that the cases nobody showed you behave like the cases somebody did. That is not a jab from the outside — it is close to the industry default, and it is exactly the failure mode systematic trading spent decades learning to engineer out of itself, one blown-up account at a time.

Anything I ship sits behind a gate before it counts as shipped: the eval suite, the oracle, the out-of-sample check. That gate is slower than a demo, and it is the only reason I am willing to say, in public, that something works. Not because it impressed me once, but because I can show you what “works” means, measured, on cases I did not get to pick after the fact.

One discipline, not two

Markets taught me, the hard way, to distrust a beautiful backtest until it has survived data it was never fitted to. I apply that same distrust to my own AI, for the same reason: a demo and a backtest are the same object wearing different clothes, and both will tell you exactly what you want to hear if you let them. The two disciplines are not analogous to each other. They are one discipline, applied to two different kinds of model — one built from price data, one built from language.

If you are building AI agents — for trading or for anything else — and the only evidence you have for them is a demo, that is precisely the gap I help close. The agentic framework this evaluation discipline exists to keep honest lives on the strategy lab page; the full story of how it turns a single sentence into a finished strategy is in the agentic framework post.

If you want a second set of eyes on how you are evaluating your own agents, get in touch.