Why most backtests lie, and the one number that does not

A strategy that shows +1.13R per trade over four trades is not a strategy, it is a coin that landed well. Here is the number we plan on instead, and why the engine refuses to run without it.

4 min read

Backtestingbacktestingevidencecrypto futures

Also available in Bahasa Indonesia

Last week we ran 339 backtests across the 34 most liquid perpetual futures on Binance - every strategy the terminal ships, on the daily horizon, 19,482 simulated trades in total. The single best result was this:


+1.1310R per trade   PENGUUSDT   downtrend_rally   4 trades

That line would be the headline of most trading products. It is also worthless, and explaining why is the whole point of this post.

The average is not the answer

Four trades. Two of them happened to land well and the average is enormous. Run the same strategy across all 34 symbols, 408 trades, and it measures +0.036R - about thirty times smaller. Nothing changed about the strategy. What changed is that four coin flips stopped being mistaken for a measurement.

Every backtest table you have ever seen was sorted by that first column, and the top of the sort is always the smallest sample. That is not a coincidence; it is arithmetic. Small samples have the widest spread, so the best-looking row in any table is almost always the one you know least about.

The number we plan on

The terminal does not arm an automated strategy on its average. It arms it on the lower bound of a 99% confidence interval - the worst figure the data is still consistent with, given how many trades it rests on and how much they varied.

For the same sweep, here is what that does:


strategy            average    lower bound   trades
confluence          +0.1033R   +0.0605R      2,810
donchian            +0.0956R   +0.0577R      4,064
macd_momentum       +0.0994R   +0.0484R      1,917
cash_in_downtrend   +0.0867R   +0.0386R      2,612
squeeze_breakout    +0.1545R   +0.0296R        390
ema_cross           +0.0746R   +0.0069R      1,169
downtrend_rally     +0.0398R   -0.0612R        440

squeeze_breakout has the highest average in the table and only the fifth-best lower bound, because it rests on 390 trades rather than thousands. downtrend_rally - the strategy behind that +1.13R headline - has a lower bound below zero. The data cannot rule out that it loses money. So the engine will not run it, and the screen says exactly that.

Why a lower bound and not a p-value

A p-value tells you whether an effect is distinguishable from zero. It does not tell you how large it is, and a trader needs the size: a strategy that is very definitely +0.001R per trade is a strategy that very definitely does not cover its fees.

The lower bound answers both questions in one number. If it is above zero, the effect is real at that confidence. And its distance from zero is the smallest edge you should expect to keep. When you size a position or ask how long a target will take, that is the figure to use - not the average, which is the figure to hope for.

What this means for the account you actually have

Suppose you want to turn 150 USD into 500 - a little over three times. With confluence's lower bound of +0.0605R and 5% of equity risked per trade, that takes about 400 trades. On a daily horizon across the same 34 symbols, that is roughly a year.

A week earlier, before the sweep, the best futures lower bound on file was +0.0235R on 43 trades. The same target took eighteen years. Nothing about the market changed between the two numbers. The sample did.

That is the honest shape of this work: the most valuable thing you can do for a strategy is not tune it, it is measure it more.

What the engine does with all this

Every strategy, every market, every horizon is measured separately and the results are pooled across everyone who runs the terminal. When you press ARM, the engine looks up the combination you chose, checks its lower bound, and either runs or tells you which combinations it would run. It never picks the best row for you: naming one winner out of forty would be committing the exact error this post is about, one level up.

If you want to see the table for yourself, it is in the BACKTEST EVIDENCE panel on every automation screen, per horizon, with the sample size beside every number.

← Backtesting