Data infrastructure
Market Data Lineage: Every Bar Needs a Provenance Trail
Clean data is not enough; a reviewer must be able to reconstruct where each transformation came from.
Working definition
Data lineage is the evidence chain linking a reported number to source files, vendor metadata, normalization steps, and the exact code that produced it.
01
A schema is not provenance
Column names and types explain how a dataset is shaped, not where the values came from. A lineage record should identify the vendor, dataset, delivery or snapshot, license boundary, checksums, coverage window, and the transformations applied before the backtest consumed it.
This matters when two apparently identical files differ because of a vendor correction, a timezone policy, or an adjusted-price convention. Without retained source identity, a rerun can be internally consistent and still test different evidence.
02
Transformations need observable contracts
Deduplication, session filtering, corporate-action adjustment, symbol mapping, and resampling should emit counts and diagnostics. A silent transformation is difficult to review because it leaves no boundary between expected cleaning and accidental data loss.
- Hash raw snapshots and normalized outputs.
- Store timezone, calendar, and adjustment policies.
- Report rows removed, imputed, duplicated, or quarantined.
- Bind the final report to immutable input identities.
03
Lineage shortens investigations
When a metric changes, provenance turns an open-ended debugging exercise into a comparison of named inputs and transformations. It also makes limitations concrete: unavailable point-in-time history or undocumented corrections can be described precisely instead of hidden behind a generic data-quality disclaimer.
Practical takeaways
- Retain source identity and checksums, not only normalized files.
- Make cleaning steps emit reviewable diagnostics.
- Bind reports to immutable data and code identities.
- Quarantine evidence whose provenance cannot be reconstructed.
Want an independent read?
Test the claim, the data, and the implementation together.
Validraft scopes the hypothesis, checks feasibility, and delivers a descriptive validation report with visible evidence and limitations. Research and simulation only; never investment advice.