Optimization Is Where Backtests Become Dangerous
Once a backtester can run a strategy repeatedly, it becomes easy to search parameter combinations until historical performance looks impressive. That is also one of the fastest ways to build a strategy that fails out of sample.
The core problem is simple: data used to choose a strategy cannot provide an unbiased estimate of that strategy's performance. If you try 500 configurations on the same history and keep the best one, the winning configuration has benefited from both signal and luck. Ranking those 500 trials by historical Sharpe ratio does not make the first row a reliable forecast.
In Part 2, we separated signal time from execution time and made the equity curve and trade ledger explicit outputs. Part 3 adds a second boundary that is just as important:
training data chooses the model; unseen test data evaluates the choice.
That boundary is the foundation for parameter optimization, walk-forward testing, and more credible research.
For a broader definition of the process, see backtesting and algorithmic trading.
Start with a Point-in-Time Strategy Interface
Before optimizing anything, make the strategy's inputs explicit. A moving-average example can be useful mechanically even though it is not evidence of an investable edge.
1function movingAverageSignal(history, { fastPeriod, slowPeriod }) {
2 if (fastPeriod >= slowPeriod) {
3 throw new Error('fastPeriod must be smaller than slowPeriod');
4 }
5 if (history.length < slowPeriod) return 'hold';
6
7 const closes = history.map((bar) => bar.close);
8 const fast = average(closes.slice(-fastPeriod));
9 const slow = average(closes.slice(-slowPeriod));
10
11 if (fast > slow) return 'buy';
12 if (fast < slow) return 'sell';
13 return 'hold';
14}
15
16function average(values) {
17 return values.reduce((sum, value) => sum + value, 0) / values.length;
18}The important detail is not the crossover. It is that the function receives only history through the current decision time. The optimizer may change fastPeriod and slowPeriod, but it should not change the information boundary.
Split Selection from Evaluation
A minimal research process needs at least two chronological segments:
- training segment: choose parameters or model rules;
- test segment: evaluate the frozen choice once.
For time series, random row shuffling is usually the wrong default because it destroys temporal order and can place future market regimes into the training set.
1function chronologicalSplit(data, trainFraction = 0.7) {
2 if (trainFraction <= 0 || trainFraction >= 1) {
3 throw new Error('trainFraction must be between 0 and 1');
4 }
5
6 const splitIndex = Math.floor(data.length * trainFraction);
7
8 return {
9 train: data.slice(0, splitIndex),
10 test: data.slice(splitIndex),
11 };
12}This is only a starting point. If your labels or trades overlap the split boundary, information can still leak across it. Strategies built from forward returns, event windows, or multi-day holding periods may require a gap, purge, or embargo around the boundary.
The rule is broader than any one implementation: the test segment should not influence the strategy-selection decision.
Grid Search Is Fine, In-Sample Grading Is Not
Grid search itself is not the problem. The problem is optimizing and reporting performance on the same observations.
First, generate candidate parameter sets:
1function parameterGrid({ fastPeriods, slowPeriods }) {
2 const candidates = [];
3
4 for (const fastPeriod of fastPeriods) {
5 for (const slowPeriod of slowPeriods) {
6 if (fastPeriod < slowPeriod) {
7 candidates.push({ fastPeriod, slowPeriod });
8 }
9 }
10 }
11
12 return candidates;
13}Then evaluate candidates on training data only:
1function selectParameters(trainData, candidates, runBacktest) {
2 const ranked = candidates
3 .map((params) => {
4 const result = runBacktest(trainData, params);
5 return {
6 params,
7 score: result.metrics.sharpe,
8 result,
9 };
10 })
11 .filter((trial) => Number.isFinite(trial.score))
12 .sort((a, b) => b.score - a.score);
13
14 if (ranked.length === 0) {
15 throw new Error('No valid parameter trials');
16 }
17
18 return ranked[0];
19}Finally, freeze the selected parameters and run them on the unseen test segment:
1const { train, test } = chronologicalSplit(bars, 0.7);
2const candidates = parameterGrid({
3 fastPeriods: [5, 10, 20],
4 slowPeriods: [40, 60, 100],
5});
6
7const selected = selectParameters(train, candidates, runStrategyBacktest);
8const outOfSample = runStrategyBacktest(test, selected.params);
9
10console.log({
11 selectedParameters: selected.params,
12 trainingSharpe: selected.score,
13 testSharpe: outOfSample.metrics.sharpe,
14});The out-of-sample result is the interesting one. A large collapse from training to test performance is not a nuisance to optimize away. It is evidence about robustness.
Do Not Optimize a Metric You Cannot Trust
An optimizer will exploit mistakes in the objective function just as aggressively as it exploits genuine edge. Before using Sharpe ratio, CAGR, drawdown, profit factor, or any custom score, verify that the underlying return series and accounting are correct.
At minimum, the scoring function should specify:
- whether returns are daily, hourly, or trade-level;
- whether returns include transaction costs and slippage;
- how the risk-free rate is aligned to that frequency;
- how open positions are marked to market;
- whether leverage is reflected in the equity curve;
- how missing observations are handled.
If those definitions drift between trials, the optimizer is comparing different experiments.
Prefer Robust Regions to a Single Lucky Peak
Suppose the best historical result occurs with a 17-day fast average and a 63-day slow average, but nearly every neighboring combination performs poorly. That isolated optimum is less convincing than a broad region where nearby parameters behave similarly.
One practical way to inspect this is to retain the full trial table rather than only the winner:
1function rankTrials(trainData, candidates, runBacktest) {
2 return candidates
3 .map((params) => {
4 const result = runBacktest(trainData, params);
5 return {
6 ...params,
7 cagr: result.metrics.cagr,
8 sharpe: result.metrics.sharpe,
9 maxDrawdown: result.metrics.maxDrawdown,
10 trades: result.trades.length,
11 };
12 })
13 .sort((a, b) => b.sharpe - a.sharpe);
14}Look for questions the ranking alone cannot answer:
- Are nearby parameter choices also acceptable?
- Does the strategy depend on one unusually profitable period?
- Does performance survive modestly higher costs?
- Is the result driven by a handful of trades?
- Does the selected configuration behave sensibly across distinct market regimes?
A backtest should help you reject fragile ideas, not just crown a winner.
Walk-Forward Validation
A single train/test split still depends heavily on one cutoff date. Walk-forward validation repeats the selection/evaluation sequence through time.
For each fold:
- choose parameters using only the training window;
- freeze those parameters;
- test them on the immediately following unseen window;
- move forward and repeat;
- combine only the out-of-sample folds for the final evaluation.
Here is a compact scaffold:
1function walkForward({
2 data,
3 trainBars,
4 testBars,
5 candidates,
6 runBacktest,
7}) {
8 const folds = [];
9
10 for (
11 let trainStart = 0;
12 trainStart + trainBars + testBars <= data.length;
13 trainStart += testBars
14 ) {
15 const trainEnd = trainStart + trainBars;
16 const testEnd = trainEnd + testBars;
17
18 const train = data.slice(trainStart, trainEnd);
19 const test = data.slice(trainEnd, testEnd);
20
21 const selected = selectParameters(train, candidates, runBacktest);
22 const testResult = runBacktest(test, selected.params);
23
24 folds.push({
25 trainStart: train[0].timestamp,
26 trainEnd: train.at(-1).timestamp,
27 testStart: test[0].timestamp,
28 testEnd: test.at(-1).timestamp,
29 params: selected.params,
30 trainingScore: selected.score,
31 testResult,
32 });
33 }
34
35 return folds;
36}This example uses a rolling training window of fixed length. An expanding window is another valid design: training starts at the beginning of the sample and grows with each fold. Which one is appropriate depends on what the live system is expected to do.
The key invariant is that each test fold is chronologically later than the data used to select its parameters.
Aggregate Only Out-of-Sample Performance
A subtle mistake is to run walk-forward folds correctly and then publish the attractive in-sample results alongside the out-of-sample results as if they were one continuous track record.
For model evaluation, the combined walk-forward series should be assembled from the test folds only. Training results are diagnostics for the selection process.
If folds overlap, be careful not to count the same observation more than once. If each test window immediately follows the prior one and windows do not overlap, concatenation is straightforward.
In a more complete engine, store a fold identifier on every out-of-sample equity or return observation. That makes it possible to audit which frozen model produced each result.
Understand the Multiple-Testing Problem
Even perfect train/test boundaries do not make unlimited experimentation free.
Imagine this workflow:
- try 100 parameter grids;
- inspect the test result after each one;
- change the strategy when the test result disappoints;
- repeat until the test period looks good.
The nominal test set has now become part of the training process. Human feedback leaked information from the test period into model development.
For serious research, keep a hierarchy of evidence:
- development/training data for model construction;
- validation data for model and parameter selection;
- a final holdout that is touched rarely;
- forward or live shadow performance after development.
You do not need an elaborate machine-learning platform to respect this principle. You do need discipline about which results are allowed to change the model.
Stress the Assumptions, Not Just the Parameters
Parameter sensitivity is only one kind of robustness test. A strategy that survives nearby parameter choices but fails under a tiny increase in trading cost is still fragile.
Useful stress tests include:
- higher commissions and bid/ask spread assumptions;
- worse fills or delayed execution;
- different starting dates;
- removal of the best few trades;
- different but defensible data-cleaning choices;
- regime-specific subperiods;
- lower position limits or leverage;
- alternative rebalance frequencies.
The objective is not to make every scenario profitable. It is to understand which assumptions the conclusion depends on.
Record the Research Trail
Optimization is much easier to trust when every run leaves an audit trail. For each experiment, consider storing:
1const experiment = {
2 strategyVersion: 'ma-crossover-v1',
3 datasetVersion: 'daily-bars-2026-09-11',
4 trainStart: '2010-01-01',
5 trainEnd: '2019-12-31',
6 testStart: '2020-01-01',
7 testEnd: '2022-12-31',
8 parameters: { fastPeriod: 20, slowPeriod: 60 },
9 executionModel: 'next-open',
10 commissionBps: 1,
11 slippageBps: 2,
12};The exact fields depend on the system, but the principle is stable: a result should be reproducible from its data version, strategy version, parameters, execution assumptions, and evaluation window.
That same philosophy drives the methodology and provenance sections in Grizzly Bulls' original research.
Next: Risk and Performance Measurement
A strategy can be genuinely out of sample and still be economically unattractive. It may require too much leverage, trade too often, suffer unacceptable drawdowns, or depend on execution assumptions that are impossible at scale.
Part 4 completes this series by tightening position sizing, slippage, transaction costs, drawdown, CAGR, volatility, and Sharpe-ratio calculations.
The progression across the series is deliberate:
clean data → point-in-time decisions → realistic execution → out-of-sample validation → risk-aware measurement
That sequence is more important than any particular JavaScript function in these examples.
Disclaimer: The examples in this tutorial are educational and intentionally simplified. Historical backtests do not guarantee future results, and optimization can amplify false confidence when data leakage or multiple testing is not controlled.
