Validating Structured Extraction: Code First, Judgment Second

#The scenario

An LLM extracts structured data from documents — invoices, contracts, forms — into JSON. Your code already validates the shape: required fields present, dates parse, totals are numbers. But shape validation is blind to meaning: a plausibly-formatted date that's the signature date instead of the effective date, a total that's actually the subtotal, a counterparty name pulled from the footer instead of the header. These errors parse perfectly and corrupt downstream systems silently.

#Why Jev fits

"Does this extracted value correctly reflect the source document?" is a judgment call, not a parsing problem — and it's exactly the kind of closed question (yes/no per field, or a confidence score per record) a judgment-only model answers cheaply. The division of labor is clean: code checks what code can prove, Jev judges what requires reading comprehension, and neither pretends to do the other's job.

#Judgment design example

Structured input per extracted record:

Judgments to define:

1. Noul per critical field: "Does the excerpt support effective_date = 2026-03-01?" → any high-value field failing sends the record to review.
2. Noul per record: "Is this extraction as a whole faithful to the source?" → cheap gate for the common case where you only need record-level pass/fail.
3. Score (optional): "Extraction quality 1–5" → records at 1–2 go to human review, 3 gets a second extraction attempt with a different prompt, 4–5 flows through.

#Where the LLM sits

Upstream, as the extractor — and occasionally downstream, as the fixer. The pipeline is: LLM extracts → code validates shape → Jev judges faithfulness → only failures loop back to the LLM with the specific field flagged. The expensive re-extraction happens on the small failing fraction, not the whole corpus. This is confidence gating applied to extraction instead of routing.

#Watch-outs

• Judge against excerpts, not whole documents. Context is limited (reported 64k tokens, with the longest single question capped at 32k) — pass the relevant passage plus the extracted fields, not the full contract.
• Code stays the first gate. Never spend judgment calls on records that fail schema validation; reject those in code for free.
• Judgment is not proof. A Noul "yes" means the model believes the field is faithful. For money-moving fields, sample-audit the passers too — independent early testing suggests real-world accuracy runs below official claims and varies by task.

#Sources

• jev101.com — application-scenario coverage of structured extraction (结构化抽取) pipelines as a judgment-model workload.
• jevai.wiki — API reference (state + typed questions request shape) and model pages.
• learnjev.com — Concepts: System One and the Three Primitives — the judgment-not-generation framing this pattern depends on.
• "Jev explained in 7min" — Caleb Writes Code — the "new paradigm" overview: typed decisions as a function call.

Unofficial fan-made handbook. Not affiliated with TypeSafe AI or jev.com.