Parameter Sensitivity: Look for Plateaus, Not Peaks
A strategy that works at only one lookback is describing the sample, not the market.
In this note
Working definition
Parameter sensitivity describes how a strategy's results change when its settings move by reasonable amounts. A robust choice sits in a region of similar outcomes; a fragile one depends on a narrow setting.
01
Sweep the neighborhood, not just the winner
Consider a hypothetical moving-average rule tested with lookbacks from 20 to 200 days. If the reported 63-day lookback is surrounded by settings with similar results, the choice is a reasonable summary of a broader effect. If 62 and 64 days look much worse, the result depends on the precise alignment of the sample with that one setting. This is an illustrative example, not a result from a Validraft client strategy.
A useful sweep varies each parameter over a range justified by the mechanism, and varies interacting parameters together. Results should be shown as a grid or surface with the chosen point marked, rather than as a single row extracted from the search.
02
Smooth surfaces can still mislead
A broad plateau is necessary, not sufficient. If all nearby settings share the same few trades or the same short period, the plateau reflects one episode rather than many. Read the sweep together with the period breakdown: a region that performs well in every subperiod tells a different story from one that performs well only in aggregate.
The sweep is also part of the search. Every point evaluated adds to the number of trials that a multiple-testing adjustment should account for, so the map should not become a larger hunting ground for the best number.
- Vary each parameter over a range the mechanism can justify, and interacting parameters jointly.
- Show the full grid with the chosen setting marked, not only the best row.
- Check whether the plateau holds in each subperiod or depends on one episode.
03
Choose by rule, then hold it fixed
Decide how the parameter will be chosen before looking at the out-of-sample period: the center of the plateau, the setting implied by the mechanism, or a fixed rule applied at each walk-forward refit. Then keep it fixed for the evaluation. Reoptimizing after seeing out-of-sample results turns the holdout into another training set.
In a report, a parameter table is most informative when it shows the neighborhood alongside the chosen value, the share of nearby settings that clear the stated threshold, and how stable the choice was across refits.
Practical takeaways
- An isolated optimum describes the sample; a plateau of similar results is more likely to describe a mechanism.
- A plateau built on the same few trades or one period is still fragile.
- Every point in a sweep counts toward the multiple-testing burden.
- Fix the selection rule before the out-of-sample evaluation and keep it fixed.
Want an independent read?
Test the claim, the data, and the implementation together.
Validraft scopes the hypothesis, checks feasibility, and delivers a descriptive validation report with visible evidence and limitations. Research and simulation only; never investment advice.