Write the grader first

The standard way to build an expert dataset goes like this. You decide on a capability. You hire people who have it. You give them a brief, collect what comes back, and then — at the end, under deadline pressure, with a pile of expensive submissions already sitting there — you work out how to tell the good ones from the bad ones.

By then it is too late, and not for the reason people usually assume. The problem is not that reviewing is hard. The problem is that you have already paid for work produced against a target nobody could see.

Ambiguity is priced at the end, but created at the start

Give ten senior practitioners the same loosely-worded brief and you get ten defensible, mutually incompatible submissions. Each one is correct under a slightly different reading. None of them is wrong in a way you can point at. So the review turns into negotiation, the acceptance bar drifts toward whatever the majority happened to do, and the dataset ends up encoding your reviewers' tolerance rather than the domain's actual standard.

Writing the grader first collapses that ambiguity while it is still cheap to collapse. Not because a grader is smarter than a reviewer, but because a grader has to be specific. You cannot write every_hardcode_has_a_source_note(model) without first deciding what counts as a hardcode and what counts as a source note. The act of writing the check forces the argument you were going to have anyway — before it costs you a batch.

The test: can you write the check?

We use one filter at the specification gate. Someone has to be able to describe, concretely, how a machine would distinguish a passing submission from a failing one. Not "an expert would know." A procedure.

Three things happen when you apply it.

• Some tasks survive intact. Usually the ones with a ground truth hiding in them — a model that has to balance, a patch that has to pass a suite, a derivation that has to be symbolically equivalent to a known form.
• Some tasks get redesigned. "Assess this company's credit quality" is unhelpable. "Reconstruct the leverage ratio from the filings, and flag every adjustment the company made that the covenant definition does not permit" is checkable, and it turns out to be the part that carried the expertise anyway.
• Some tasks get abandoned. This is the valuable outcome and the one teams resist. If nobody can describe how to check it, you are not going to be able to verify it at scale either — you are going to ship reviewer opinion and call it expert data.

Rubrics are graders with worse error messages

The common objection is that most real work isn't programmatically checkable. Judgment, taste, the well-constructed argument. True in general, and less true than people expect in the specific.

A rubric is still a grader. It is a slower, noisier one with a human in the loop, and the same discipline applies: written before commissioning, tested against known-good and known-bad work, versioned alongside the spec. The failure mode is not "we used a rubric." The failure mode is that the rubric was written after the submissions arrived, by someone looking at the submissions, which is how you launder a reviewer's preferences into a quality standard.

The other objection is to reach for a model as judge. That is a legitimate tool and a dangerous default. A model judge with a human-calibrated reference set behind it, whose agreement rate with expert adjudicators you actually measure, is a grader. A model judge you dropped in because writing the check was tedious is a way of generating scores that feel like verification.

What it costs

Grader-first is slower at the front. It puts the argument at the beginning of the project, when the argument is uncomfortable and no work has been produced yet and everyone would rather be collecting data. Teams pushing for volume hate it.

What it buys is that rejection becomes informative. When a submission fails, it fails a named check, and the author gets told which one. They can fix it. The next submission is better, and the improvement compounds across a network instead of evaporating into a reviewer's head. That feedback loop is the only mechanism we know of that makes expert data get better at scale rather than just larger.

A rejection that says "no" wastes the most expensive input in the pipeline.

There is a version of this argument that is really about honesty. When you write the grader first, you find out early whether you understand the capability you claim to be building. Quite often you don't. Better to discover that in a specification meeting than in a training run.