A holdout set is a block of data intentionally withheld from model or strategy development and used later for evaluation.
In systematic investing, the value of a holdout comes from not letting its outcomes drive the choices made during research.
A holdout stops being untouched when it becomes part of iteration
If a researcher checks the holdout, changes the strategy, checks it again, and repeats that process, the holdout has effectively joined the search process.
The page on family-wise error rate explains why repeated search creates more false-positive opportunities. Label overlap addresses a different issue: data can leak across a train/test boundary even when the holdout was chosen in advance.
The Backtest Out-of-Sample Decay benchmark shows what happens to a selected zero-edge winner on a genuinely independent holdout.
Holdout evidence is useful but not permanent proof
A clean holdout can test whether a selected strategy generalizes to data that did not choose it. It cannot prove future profitability, eliminate regime risk, or compensate for incorrect historical data and execution assumptions.
Time-series work also has chronology constraints that ordinary random train/test splits can violate.
Sources: CFA Institute, Backtesting & Simulation and Grizzly Bulls Backtest Out-of-Sample Decay.
Continue Research
Continue from the concept into the Grizzly Bulls research surface that best matches the next question. These links are research continuations, not recommendations or required steps.
Test out-of-sample decay
See what happens when a selected zero-edge in-sample winner is evaluated on genuinely independent holdout data.
Explore more topics in the Financial Research Encyclopedia.