Labels are decisions too
A dataset column called fraud can combine very different evidence: a customer claim, analyst suspicion, a dispute result or an independently confirmed incident. Those categories are not interchangeable. Define what each label means before asking a model to reproduce it or a dashboard to summarise it.
Separate unknown from negative
An unreviewed transaction is not automatically a confirmed legitimate transaction. Missing outcomes can arise because a case was never selected, a customer did not report or evidence has not arrived. Treating all missing values as clean examples can teach the model the previous review process’s blind spots.
A feedback example
A team reviews only high-value transactions and labels every unreviewed low-value transaction legitimate. A model trained on that dataset may learn that low value predicts legitimacy partly because the team never looked there. High measured accuracy against those labels would not independently validate the assumption.
Keep provenance attached
Record label source, confidence, time and revision history. Distinguish provisional judgements from mature outcomes, and compare disagreements between reviewers using a shared rubric. When evidence changes, update downstream evaluation deliberately so last month’s mistaken label does not remain an invisible benchmark for every later model.