Backtesting: What Historical Simulations Can and Cannot Prove

What is backtesting?

Backtesting applies a trading or investment process to historical data to estimate how that process would have behaved in the past. A useful backtest can expose weak assumptions, quantify turnover and drawdowns, compare alternatives, and reveal whether a strategy depended on a narrow market regime.

It cannot validate future profitability. Historical performance is evidence about a model under a particular research design, dataset, execution model, and market history. The more choices a researcher makes after seeing the data, the easier it becomes to produce an impressive result that does not generalize.

The goal is therefore not to make a backtest look good. The goal is to make it hard for the backtest to fool you.

Start with a testable trading process

A backtest needs rules that could have been followed at the time. At minimum, define:

• the investable universe;
• the information available at each decision point;
• entry and exit rules;
• position sizing and portfolio constraints;
• rebalance or order timing;
• transaction-cost and financing assumptions; and
• what happens when data, liquidity, or an intended trade is unavailable.

This matters because seemingly small implementation choices can dominate the result. A signal computed from the closing price cannot normally assume a fill at that same closing price unless the strategy could have known and acted on the signal before the execution opportunity disappeared.

A simple performance example

Suppose a hypothetical backtest grows from $10,000 to $18,000 over five years.

1CAGR = ($18,000 / $10,000)^(1 / 5) - 1
2     ≈ 12.47% per year

Now suppose the strategy also suffers a 25% maximum drawdown. Recovering from that loss requires more than a 25% gain:

1Value after drawdown = 1.00 × (1 - 0.25) = 0.75
2Gain needed to recover = (1.00 / 0.75) - 1
3                       ≈ 33.33%

Those two numbers already tell different stories. CAGR summarizes endpoint growth; maximum drawdown describes part of the path. Neither tells you whether the strategy was overfit, whether its fills were realistic, how much leverage it used, or whether the result came from one exceptional period.

The Research Tool below calculates both figures, but the harder work is deciding whether the simulation itself deserves trust.

Look-ahead bias: did the strategy know the future?

Look-ahead bias occurs when a simulation uses information before it would actually have been available.

Common examples include:

• using a company's quarter-end fundamentals on the quarter-end date even though the filing was published weeks later;
• using revised economic data rather than the release available in real time;
• calculating an indicator with a full day's closing price and assuming execution at that same close;
• selecting a historical universe using knowledge of which securities later survived or became important.

A robust backtester should model information time separately from the date a fact describes. The observation period and the publication timestamp are not the same thing.

Survivorship bias: who disappeared from the sample?

A current list of S&P 500 members is not the historical S&P 500. A current mutual-fund database omits funds that closed. A stock dataset can silently omit delisted companies.

If a historical strategy is tested only on securities that survived until today, weak or failed securities may be missing from the opportunity set. That can make the past look cleaner than it really was.

Survivorship bias is especially dangerous when the strategy selects among a changing universe. The backtest should reconstruct membership using the information and securities that existed at each historical point whenever the research question requires it.

Trading costs should get worse before you trust them

A backtest that assumes frictionless fills is usually an upper bound, not an investable result.

Depending on the strategy, realistic costs can include:

• commissions and exchange fees;
• bid/ask spread;
• slippage between decision and execution;
• market impact from the strategy's own order size;
• short-borrow costs and availability;
• margin or other financing costs; and
• partial fills or orders that never execute.

Turnover connects these costs to strategy economics. A 10-basis-point mistake in all-in execution cost matters little to a low-turnover portfolio and can destroy a strategy that repeatedly trades its capital.

When the exact cost is uncertain, sensitivity analysis is more informative than choosing one favorable number. If the strategy works at 5 basis points and collapses at 8, execution assumptions are part of the thesis.

In-sample, out-of-sample, and walk-forward testing

A basic research split separates data used to develop the strategy from data reserved for evaluation.

In-sample data is where you explore the idea and choose parameters. Out-of-sample data is held back so the final design encounters history that did not directly drive those choices.

That distinction is useful, but one train/test split is not magic. Researchers can indirectly overfit the test set by checking it repeatedly and changing the strategy after every disappointing result.

A walk-forward or rolling-window design is often stronger for time-series problems. The model is estimated using a historical window, applied to a later period, then rolled forward and repeated while preserving chronology. CFA Institute describes this as a closer proxy for an actual investing process because each evaluation period follows the data used to fit the model.

Parameter optimization can become data mining

Trying a few economically motivated parameter values is normal. Searching thousands of combinations and reporting only the winner creates a different statistical problem.

Imagine testing 500 unrelated strategy variants on noisy returns. Even if none has a true edge, some are likely to look unusually good by chance. Choosing the best historical result after the search means the reported performance no longer represents a single clean hypothesis test.

Useful defenses include:

• defining the hypothesis before a large parameter search;
• preferring broad regions of acceptable performance over one sharp optimum;
• retaining rejected trials rather than forgetting how many ideas were tested;
• reserving untouched evaluation data;
• using walk-forward analysis where appropriate; and
• demanding a plausible mechanism instead of accepting a statistical pattern solely because it backtests well.

For a deeper treatment, see Overfitting in Algorithmic Trading: How Good Backtests Go Bad.

Structural breaks and regime dependence

Markets change. Interest rates, tick sizes, transaction costs, regulation, participants, volatility, correlations, and available technology can all shift.

A strategy may therefore be historically real and still stop working. CFA Institute's backtesting guidance highlights structural breaks and scenario analysis for exactly this reason.

Instead of asking only for the full-period result, inspect subperiods that stress the mechanism. How did the strategy behave during high and low volatility? Rising and falling rate regimes? Liquidity shocks? Strong trends and range-bound markets? The relevant regimes depend on the thesis.

Consistency across regimes is not required for every strategy. A trend strategy may reasonably struggle in choppy markets. What matters is understanding why performance changes and whether that behavior matches the claimed source of return.

Metrics that deserve attention

No single backtest statistic summarizes investment quality. A useful review often includes:

• compound annual growth rate;
• volatility and downside volatility;
• maximum drawdown and time under water;
• Sharpe or other risk-adjusted measures, with their assumptions understood;
• gross and net exposure;
• leverage;
• turnover and modeled trading costs;
• trade count and concentration of profits;
• tail losses;
• performance by regime and subperiod; and
• sensitivity to nearby parameters and execution assumptions.

A 20% CAGR generated by extreme leverage is not directly comparable with an unlevered 20% CAGR. A high Sharpe ratio from 25 trades deserves different confidence than the same estimate from thousands of reasonably independent observations.

What a strong backtest can actually tell you

A carefully designed simulation can support statements such as:

• the rules are implementable on historical information;
• the strategy would have produced a particular sequence of hypothetical positions under stated assumptions;
• the result survives specified estimates of spread, slippage, and fees;
• performance is not isolated to one parameter point or one historical subperiod; and
• a reserved or walk-forward sample showed behavior consistent with the original hypothesis.

It still cannot prove that the future distribution of returns will resemble the past, that live execution will match the simulator, or that competitors will not erode the opportunity.

That is the right standard for algorithmic trading: use a backtest to eliminate bad ideas and build evidence for promising ones, then keep testing the system as it moves into paper trading and limited live deployment.

A practical Grizzly Bulls checklist

Before treating a backtest as decision-useful evidence, ask:

1. Could every input have been known at the simulated decision time?
2. Does the universe include relevant delisted, removed, or failed securities?
3. Are corporate actions and point-in-time fundamentals handled correctly?
4. Are spread, slippage, commissions, borrow, financing, and capacity realistic for the strategy?
5. How many hypotheses and parameter combinations were tried?
6. Is there genuinely untouched or walk-forward evidence?
7. Does performance survive nearby parameters and worse execution assumptions?
8. Is the return concentrated in a few trades, assets, or years?
9. How does the strategy behave in regimes that challenge its economic mechanism?
10. Can the live system monitor, reconcile, and stop the strategy safely if reality diverges from the simulation?

Passing the checklist does not validate future returns. It raises the quality of the evidence.

Related Grizzly Bulls research

• Algorithmic Trading: How Automated Strategies Actually Work
• Building an Algorithmic Trading Backtester with Node.js
• Overfitting in Algorithmic Trading: How Good Backtests Go Bad
• Machine Learning for Trading, Part 1: Build a Valid Experiment

Sources and further reading

• CFA Institute: Backtesting & Simulation
• FINRA: Algorithmic Trading
• FINRA Regulatory Notice 15-09: Effective Supervision and Control Practices for Algorithmic Trading
• SEC: EDGAR Search

Explore more topics in the Financial Research Encyclopedia.