Research infrastructure
Reproducible Backtests: From Notebook to Evidence Package
A result is reproducible only when another run can recover the same inputs, decisions, and outputs.
Working definition
A reproducible backtest is one whose declared inputs, code identity, environment, parameters, and random seeds can recreate the reported artifacts without manual reconstruction.
01
The notebook is a workspace, not the evidence boundary
Interactive notebooks are valuable for exploration, but their hidden state and editable execution order make them weak release artifacts. The production run should begin from a declared configuration and create outputs in a fresh directory. The report should read those outputs rather than live variables from an analyst session.
This separation prevents a common failure: the chart was generated from one state, the metric table from another, and neither can be matched to the final code revision.
02
Identity must include more than a Git hash
Code identity matters, but so do data snapshots, dependencies, feature definitions, calendar policies, parameters, and seeds. A package manifest should hash the files that support the claim and reject changes after assembly. Reproducibility is a property of the whole run, not one repository commit.
- Machine-readable run declaration and strategy brief
- Input checksums and point-in-time coverage metadata
- Locked parameters, environment, dependencies, and seeds
- Metrics, ledgers, charts, checklists, and decisions tied by hashes
03
Idempotence is a useful test
Running the same declared job twice should either create the same content-addressed package or fail with a clear explanation. Silent overwrites and mutable final folders make it impossible to know which evidence a report originally described.
Practical takeaways
- Generate release artifacts from fresh, declared runs.
- Hash data, configuration, code, and outputs together.
- Keep report rendering downstream of immutable evidence.
- Make reruns idempotent or explicitly versioned.
Want an independent read?
Test the claim, the data, and the implementation together.
Validraft scopes the hypothesis, checks feasibility, and delivers a descriptive validation report with visible evidence and limitations. Research and simulation only; never investment advice.