A good customer behind a bad signal.
Account for the cost of a rule, using labelled outcomes.
A fictional team tests a review rule on 1,000 historically labelled payments. The sample contains 20 confirmed fraudulent payments and 980 legitimate ones. The rule flags 15 fraudulent payments and 85 legitimate ones. These are exercise labels, not a claim about a real model.
Fictional scenario · Synthetic data · No real customer records- flagged
- 100
- fraud among flagged
- 15
- legitimate among flagged
- 85
What would
you look at?
Open at least two files before making a decision. You can read every file.
0 of 3 files opened
A useful definition01 / Labelled sampleOutcome table+
Flagged: 15 fraud and 85 legitimate. Not flagged: 5 fraud and 895 legitimate. Precision is 15/100 = 15%. Recall is 15/20 = 75%. The false-positive rate is 85/980, approximately 8.7%. These metrics answer different questions.
An alert count says little without a denominator and outcomes. Do not confuse false-positive rate with the share of alerts that are legitimate.
02 / Review operationsCapacity & impact+
The team can review forty extra payments per day. This sample represents one day and would create one hundred reviews. The team has not tested a narrower threshold, and the cost of holding a legitimate payment varies by merchant.
Capacity and customer impact belong in the evaluation. A rule can identify fraud while creating an unsustainable queue.
03 / Data noteValidation limits+
The sample is historical. Labels were reviewed after an agreed outcome window, but future traffic may differ. Amounts and operational costs are not supplied. No monetary savings can be calculated from these counts alone.
A useful metric needs a defined population and observation window. Historical labels do not guarantee future performance.
Make the call.
Leave your reasoning.
Choose what you would do under this scenario’s assumptions. Your notes stay in this browser. Do not enter real customer information.
The trade-off
Still unknown
How alternative thresholds perform, how the rule generalises, and the monetary or customer cost of each outcome.
A useful handoff
Record the confusion matrix, capacity, observation window and rollout criteria. Explain what would make you revise your recommendation.
Completion records participation, not competence or certification. You can change your answer and compare again.
Further reading
Google / Classification metricsReference checked 5 November 2026. The scenario, runbook and debrief are original Antifrog teaching material; the source does not endorse them.