The family-wise error rate, or FWER, is the probability of making at least one false positive across a defined family of hypothesis tests.
Testing many strategy variants changes the question from "How often does one test produce a false positive?" to "How often does the whole research search produce at least one false positive?"
Repeated testing raises the chance of a lucky winner
If independent tests each use significance level alpha, the probability of at least one false positive across m tests is:
1FWER = 1 - (1 - alpha)^mBonferroni control uses a per-test threshold of approximately:
1alpha / mThe calculation is simple, but the interpretation depends on what belongs to the test family and whether tests are actually independent.
That distinction connects FWER to the effective number of trials and to holdout sets, which answer different parts of the research-validation problem.
Use Backtest Multiple Testing for the family-wise false-positive calculator and Search-Adjusted Sharpe Threshold for the related Sharpe hurdle.
FWER control does not repair a flawed backtest
Multiplicity correction cannot fix look-ahead data, unrealistic fills, survivorship bias, hidden strategy trials, or changing market regimes.
It only addresses the probability of false rejection across the test family under the stated assumptions.
Sources: Penn State STAT 555, Controlling Family-Wise Error Rate and Grizzly Bulls Backtest Multiple Testing.
Continue Research
Continue from the concept into the Grizzly Bulls research surface that best matches the next question. These links are research continuations, not recommendations or required steps.
Measure multiple-testing risk
Calculate family-wise false-positive probability and Bonferroni or Sidak thresholds across a tested strategy family.
Raise the Sharpe hurdle
See how the best result's Sharpe threshold rises when a full independent strategy search must meet one family-wise error target.
Explore more topics in the Financial Research Encyclopedia.