Correctness is a design decision, not a review step.
Most data pipelines commission work and then ask how to check it. That order is backwards, and it is the reason so much expert data is expensive and mediocre at the same time. We invert it: the grader exists before the task does.
Enforced by the platform, not by good intentions
A gate that a deadline can talk its way past is not a gate. Each of these is a state transition in the system — work cannot skip one.
Specify
The capability target and acceptance criteria are written with your team in plain language. If we can't describe what a correct answer looks like, we are not ready to hire anyone.
Build the grader
The reward function or rubric is written and tested against known-good and known-bad work. If correctness can't be checked programmatically, the task gets redesigned until it can.
Commission
Vetted practitioners produce the work in professional tools, with the brief and its acceptance criteria visible throughout. No one is guessing at a hidden rubric.
Adjudicate
The grader runs, a second expert reviews blind, and disagreement escalates to a senior adjudicator. Rejections return to the author with a reason they can act on.
What "grader first" actually means
A real task from the finance track, reduced to its bones. Note that the grader is written and reviewed before a single practitioner sees the brief.
# gate 01 — the spec
capability Build a five-year LBO with a debt schedule and a cash sweep
inputs Audited financials, an indicative term sheet, a transaction assumptions memo
done means Every output cell traces to an input or a stated assumption.
The sweep repays in the right order of priority. The model balances.
# gate 02 — the grader, written before the task is commissioned
check_1 balance_sheet_ties(model, tolerance=0.01)
check_2 debt_waterfall_order(model) == expected_priority
check_3 every_hardcode_has_a_source_note(model)
check_4 irr_reconciles(model, method="xirr", tolerance=0.0005)
partial weighted; check_1 and check_2 are gating, 3 and 4 scale the score
# gate 03 — commissioned to a practitioner, brief and criteria visible
author VP, sponsors coverage · credential verified · under NDA
# gate 04 — adjudicated
grader check_1 pass · check_2 pass · check_3 FAIL · check_4 pass
reviewer agrees — row 14 growth rate is hardcoded with no source note
verdict rejected → returned to author with the failing check
The rejection is the point. That model was almost certainly right. It was rejected because one figure could not be traced, and a dataset that tolerates untraceable figures teaches a model that untraceable figures are acceptable. The bar is not "an expert did it." The bar is "a machine confirmed it, and a second expert agreed."
What we hold to when it's inconvenient
These cost us throughput. We think they are the only reason to buy from us rather than from someone cheaper.
- Verifiable or not shipped
- If we cannot state how a claim was checked, it does not go in the dataset. "An expert said so" is provenance, not verification.
- Rejections are informative
- A rejection that just says "no" wastes the most expensive input in the pipeline. Every one returns the failing check and the reasoning.
- Disagreement escalates
- When a grader and a reviewer disagree, that item is more interesting than the ones they agree on. It goes to an adjudicator and often changes the spec.
- Exposure is tracked
- We record which items have been seen by whom and when. Contamination is a bookkeeping problem, and pretending otherwise is how benchmarks quietly die.
- Models draft, humans author
- We use models to triage, draft and stress-test. A model is never the author of record, and anything model-generated is labelled in the manifest.
- The spec can be wrong
- When practitioners keep failing the same check, the usual cause is a bad spec, not bad experts. We fix the spec and re-run.
Send us something hard to check
The tasks worth commissioning are the ones where correctness is contested. Those are the interesting conversations.