Machine Learning for Trading, Part 1: Build a Valid Experiment

Machine learning can be useful in systematic trading. It can also produce extremely convincing nonsense.

Financial data gives a model thousands of opportunities to find relationships that happened by chance. Prices change over time, market regimes shift, neighboring observations are dependent, and researchers are tempted to keep trying features until something looks profitable.

That means the first machine-learning problem in trading is not choosing between a neural network and a random forest.

It is designing an experiment that gives a model a fair chance to fail.

This article builds that foundation. Part 2 applies it to a random forest.

Start with a question that can actually be tested

"Predict the stock market" is not a useful target.

A research target should specify an instrument, forecast horizon, outcome, and decision rule. Examples include:

• Will the next daily return be positive or negative?
• Will realized volatility over the next five sessions exceed a threshold?
• What is the expected return over the next hour?
• Is the expected excess return large enough to justify trading after costs?

The target determines almost everything that follows.

Suppose we want to predict the sign of the next daily return for an index future. For day t, we can define:

1forwardReturn[t] = close[t + 1] / close[t] - 1
2label[t] = 1 if forwardReturn[t] > 0 else 0

That is intentionally simple. A real strategy may use a cost-aware threshold rather than zero, but the label is at least explicit and reproducible.

Do not begin by predicting the raw price from time

A common toy tutorial feeds a model timestamps and historical stock prices, fits a line, and extrapolates the line into the future.

That can produce a number. It does not establish a trading edge.

Most financial price series trend over long periods, so a model can appear to fit the level of the series while learning little about future returns. What matters to a trading decision is usually a change in value, risk, relative value, or probability conditional on information available at the time.

Predicting returns is also difficult, which is exactly why the experiment needs strong controls.

Establish a baseline before using machine learning

A complicated model has not earned its complexity merely because its test accuracy is above 50%.

Compare it with simple alternatives first.

For a direction classifier, reasonable baselines might include:

• always predict the majority class;
• predict that tomorrow has the same sign as today;
• a simple momentum or mean-reversion rule;
• logistic regression on the same features; or
• no trade at all.

If a random forest, neural network, or gradient-boosted model cannot beat a transparent baseline after realistic costs, the added complexity has not helped.

A useful baseline also exposes class imbalance. If 54% of observations are positive, a model that predicts "up" every day already has 54% accuracy. A 53% accurate machine-learning model would be worse despite sounding impressive in isolation.

Features must exist before the prediction is made

This sounds obvious. It is one of the easiest rules to violate.

Suppose you compute a 20-day moving average for observation t. The feature may use prices from t-19 through t. It may not use t+1.

The same rule applies to:

• normalization;
• volatility estimates;
• rolling ranks;
• fundamental data;
• economic releases;
• analyst estimates;
• revised macroeconomic series; and
• labels created from future returns.

If a feature contains information that was not available at the decision timestamp, the model has look-ahead bias.

A subtler version occurs when you standardize the entire dataset before splitting it. The mean and standard deviation then contain information from the future test period. Fit transformations on the training set and apply those fitted transformations forward.

Split time-series data chronologically

Random train/test shuffling is appropriate for many independent machine-learning problems. It is usually a poor default for market data.

A trading system is trained on the past and deployed into the future. Its evaluation should respect that direction.

A basic split could look like:

12010-2018  training
22019-2021  validation
32022-2024  final test

The exact dates are illustrative. The structure is what matters.

Use the training period to fit parameters. Use validation data to make research decisions such as model family, features, and hyperparameters. Keep the final test period untouched until those decisions are substantially finished.

If you repeatedly inspect the final test result and then alter the model, it is no longer a final test set. You have optimized to it indirectly.

For strategies that require repeated retraining, walk-forward validation can better mimic deployment:

1train A -> test B
2train A+B -> test C
3train A+B+C -> test D

The information boundary must remain intact at every step.

Transaction costs belong inside the experiment

Prediction accuracy is not profit.

A model can be statistically better than a baseline while being economically useless. Frequent trading introduces:

• bid-ask spreads;
• commissions and exchange fees;
• slippage;
• market impact;
• financing costs; and
• taxes for some investors and account types.

If the model forecasts a 2-basis-point edge and it costs 4 basis points to enter and exit, the forecast is not actionable.

A research pipeline should convert predictions into explicit positions and then calculate returns after the costs that the strategy would reasonably incur.

This also changes model design. It may be better to trade only when predicted probability or expected return clears a meaningful threshold rather than forcing a position on every observation.

Financial observations are not independent

Market data has temporal dependence. A five-day feature window on Monday and another on Tuesday share most of their underlying observations.

Labels can overlap too. If each label measures the next five-day return, adjacent examples share future days.

That means naive cross-validation can leak information even when the rows themselves look separate.

Depending on the strategy, researchers may need gaps, embargo periods, non-overlapping labels, or other techniques that keep training observations from bleeding into validation outcomes.

Our guide to overfitting in algorithmic trading covers these failure modes in more depth.

What should the first model look like?

Boring is good.

Start with a small set of features that have a plausible economic or behavioral reason to matter. For example:

11-day return
25-day return
320-day realized volatility
4distance from a 50-day moving average
5recent volume relative to its trailing average

These are examples, not a recommended trading strategy.

Then fit a simple classifier or regressor and measure whether it improves on the baseline outside the training period.

You are trying to learn whether the research hypothesis has signal, not to maximize a leaderboard score.

A minimal TensorFlow.js setup

Node.js can be useful when the rest of a trading stack already uses JavaScript or TypeScript. TensorFlow.js provides an official Node binding.

The current TensorFlow.js setup documentation uses:

1npm install @tensorflow/tfjs-node

Then:

1const tf = require('@tensorflow/tfjs-node');

The following example only demonstrates how a small binary classifier can be fit to a feature matrix. The observations are synthetic. This is not a trading strategy or evidence that these features predict markets.

1const tf = require('@tensorflow/tfjs-node');
2
3// Synthetic observations: [shortReturn, mediumReturn, volatility]
4const xTrain = tf.tensor2d([
5  [0.004, 0.012, 0.011],
6  [-0.006, -0.010, 0.018],
7  [0.002, -0.003, 0.014],
8  [0.008, 0.019, 0.010],
9  [-0.003, 0.004, 0.016],
10  [-0.009, -0.015, 0.022],
11]);
12
13// Synthetic next-period direction labels.
14const yTrain = tf.tensor2d([[1], [0], [0], [1], [1], [0]]);
15
16const model = tf.sequential();
17model.add(tf.layers.dense({
18  units: 1,
19  inputShape: [3],
20  activation: 'sigmoid',
21}));
22
23model.compile({
24  optimizer: tf.train.adam(0.01),
25  loss: 'binaryCrossentropy',
26  metrics: ['accuracy'],
27});
28
29async function run() {
30  await model.fit(xTrain, yTrain, {
31    epochs: 50,
32    verbose: 0,
33  });
34
35  const xFuture = tf.tensor2d([[0.001, 0.006, 0.013]]);
36  const probability = model.predict(xFuture);
37  probability.print();
38}
39
40run();

A production research process would load chronologically ordered observations, fit preprocessing only on past data, preserve a validation/test boundary, and evaluate trading returns after costs. The example intentionally does none of those things because six synthetic rows cannot validate a market model.

TensorFlow maintains its current TensorFlow.js setup guide and Node.js guide.

Metrics should match the decision

Accuracy alone is rarely enough.

For a classifier, useful statistical metrics can include:

• precision and recall;
• ROC AUC;
• log loss;
• calibration; and
• confusion matrices by market regime.

A trading strategy then needs economic metrics such as:

• return after costs;
• volatility;
• maximum drawdown;
• turnover;
• exposure;
• Sharpe or another risk-adjusted measure; and
• performance stability across periods and regimes.

A model can have weak classification accuracy but still be useful if its highest-confidence forecasts contain economic information. A model can also have strong accuracy and lose money if its errors occur on large moves or it trades too often.

The final test should feel uncomfortable

A good research process creates a moment where you genuinely do not know whether the strategy will work.

You have chosen the target. You have selected features and hyperparameters using training and validation data. You have specified costs and trading rules. Only then do you expose the untouched test period.

If performance collapses, that is useful information. The experiment did its job.

If performance survives, you still do not have proof of a permanent edge. You have earned the right to investigate further with robustness tests, paper trading, and eventually carefully sized live exposure.

Part 2: Random Forest

In Part 2, we use a random forest to show how a nonlinear model changes the research problem. The important pieces remain the same: chronological validation, leakage control, simple baselines, realistic costs, and skepticism about any result that looks too easy.

Machine learning is most useful in trading when it is treated as a statistical tool, not a shortcut around the scientific method.

Sources and further reading

• TensorFlow.js: Set up a TensorFlow.js project
• TensorFlow.js: Node.js guide
• Grizzly Bulls: Overfitting in Algorithmic Trading

Code examples are educational illustrations. They are not trading recommendations or evidence of predictive performance.