News Sentiment Signals: The Model Is Part of the Dataset
A sentiment score depends on the headline, its timestamp, and the exact model that read it.
In this note
Working definition
A model-derived feature is a value produced by applying a specific model version to specific inputs. For news sentiment, the score's meaning depends on the text available at the time, the model and its training data, and the label scheme it outputs.
01
Start from the timestamp of the text
A headline carries several times: when the event happened, when the story was published, when the data provider ingested it, and when it reached the research database. A signal can use the story only after the latest of the times that applies to the trading decision. Stories that are updated or corrected after publication raise the same question as revised economic data: which version was visible when?
Consider an illustrative rule that buys stocks with strongly positive overnight headlines at the next open. If the stored timestamp reflects a later edit rather than first publication, or if late-arriving stories are backfilled into the overnight window, the test reads text the market had not yet seen. This is a hypothetical example, not a result from a Validraft client strategy.
02
The score depends on the model that produced it
FinBERT, introduced by Dogu Araci in 2019, adapts a pre-trained language model to financial sentiment. The widely used ProsusAI release is fine-tuned on the Financial PhraseBank and returns softmax outputs for three labels: positive, negative and neutral. A different checkpoint, tokenizer or decision threshold produces a different feature, even on identical headlines.
That makes the model version part of the data lineage. Rescoring the archive with a newer model can change the history a strategy was fitted on, and a model trained on text published after the test period can carry knowledge of later events. Record the model identifier, the scoring date, and the rule that aggregates headline scores into a daily feature.
- Store publication, ingestion and scoring times for each item.
- Pin the model, tokenizer and label mapping used to score the archive.
- Rescore only as a new, labelled version of the feature, never in place.
03
Domain shift is a monitoring question
Language changes: new companies, new jargon, new styles of headline. A sentiment model trained on an older corpus reads today's news with yesterday's vocabulary, and its errors may not be stable over time. Monitoring the distribution of scores, the share of neutral labels, and coverage by sector can reveal drift before performance does.
Sentiment can be a useful input when it is treated as a measurement with an instrument attached. The report should state which instrument, how it was applied, and where its readings are least reliable.
Practical takeaways
- Use each story only after the latest timestamp relevant to the decision.
- A sentiment score is defined by the model version, tokenizer and label mapping as well as the text.
- Version rescored archives instead of overwriting history.
- Monitor score distributions for drift as the language of news changes.
Want an independent read?
Test the claim, the data, and the implementation together.
Validraft scopes the hypothesis, checks feasibility, and delivers a descriptive validation report with visible evidence and limitations. Research and simulation only; never investment advice.