How Many Ideas Did It Take? Significance After a Search
A t-statistic of two means something different when it is the best of two hundred specifications.
In this note
Working definition
Multiple testing is the inflation of false discoveries that occurs when many hypotheses, parameters or variants are tried and the best one is reported. The appropriate significance threshold depends on how much was explored.
01
The search is part of the experiment
If a researcher tests one pre-specified rule and finds a t-statistic of about two, the conventional 5% threshold has its usual meaning. If the same statistic is the best of hundreds of variants with different lookbacks, filters, universes and holding periods, a result of that size can be expected by chance alone. The reported number is the same; the evidence it represents is not.
Harvey, Liu and Zhu made this argument for the cross-section of expected returns: given the extensive data mining behind hundreds of proposed factors, conventional significance criteria are too lenient, and a new factor should clear a higher hurdle. The principle carries over to individual strategy research, where the search often happens in one notebook over a few weeks.
02
Count the trials, including the abandoned ones
A credible adjustment needs an honest count. That includes variants discarded early, parameter grids, alternative universes, and the specifications a colleague tried before handing over the project. Counting only the variants in the final report understates the search. Correlated variants complicate the count further: two lookbacks that differ by one day are not two independent ideas.
In practice, the useful artifact is a research log written as the work happens. It records each specification evaluated, on which data, and with what result. The log then supports a multiple-testing adjustment such as a deflated Sharpe ratio, and it lets a reviewer see whether the final choice sits in a broad region of good results or on an isolated peak.
- Log every evaluated variant with its data window and metric, including failures.
- Group near-duplicate variants before estimating how many independent trials were run.
- Set the significance bar after counting the search, not after looking at the winner.
03
Raise the bar, then seek new evidence
A higher threshold reduces false discoveries but cannot remove them, and it can reject genuine but modest effects. Treat it as one input. Evidence that was not part of the search, such as a later period, a different market or a related mechanism, carries more weight than extra in-sample significance.
We would rather see a candidate described as a promising result from an extensive search, with the count attached, than presented as significant at a threshold meant for a single test. The first description lets the client judge how much doubt remains.
Practical takeaways
- The same t-statistic is weaker evidence when it is the best of many variants.
- Count every evaluated specification, including abandoned and near-duplicate ones.
- A research log written during the work is what makes an honest adjustment possible.
- Evidence from outside the search outweighs extra in-sample significance.
Want an independent read?
Test the claim, the data, and the implementation together.
Validraft scopes the hypothesis, checks feasibility, and delivers a descriptive validation report with visible evidence and limitations. Research and simulation only; never investment advice.