Making a fabricated citation structurally impossible

Four failure modes, each needing a different layer. Conflating them is why prompt-only approaches to legal drafting stall.

Structural drift. A general model produces a differently-shaped answer every time. For a document whose value is a fixed argument in a fixed order, variance is not a quality issue, it is a correctness issue. A reviewer cannot build a scanning pattern against output that is never the same shape twice.

Citation fabrication. The expensive one. A model asked for supporting authority will produce something that looks exactly like a citation, because a plausible-looking citation is what the objective rewards. Nothing in the model distinguishes a real case from a well-formed invention.

Jurisdiction drift. A correct authority from the wrong jurisdiction is worse than no authority, because it survives a casual read.

Register drift. Informal phrasing no firm would sign, which is cheap to fix and the only one of the four that fine-tuning alone handles well.

And the tolerance is asymmetric in a way that rules out the usual accuracy framing. A system that is right 95% of the time still requires the attorney to check 100% of it, which is the work they were trying to avoid. The requirement is a constraint they can verify, not a rate they have to trust.

• Structure enforced, not requested. The five sections cannot be optional

• No path from the model to an unapproved source, rather than an instruction not to use one

• Jurisdiction as a retrieval filter, not a stylistic preference

• Quality as a measured number, so a regression is detectable before a client sees it

Four layers, composed so that each closes a failure the others cannot.

Corpus normalisation came first and mattered more than expected. Years of attorney opinions were unified into a single five-section template, tagged by mark type, jurisdiction and NICE class. Without this the model learns a dozen individual writing styles and averages them; with it, it learns one standard.

Supervised fine-tuning on that corpus teaches legal reasoning patterns, section boundaries and formal register, the shape of how a trademark opinion is built, not just how one sounds. This handles structure and register. It does not, and cannot, handle citation integrity.

Retrieval over a whitelisted index handles that. Curated case law, statutes and registry data form the entire universe of citable authority. The model has no access outside it. Jurisdiction and class filters apply at this layer, which is what makes jurisdiction discipline a property of retrieval rather than a hope about attention.

Schema-constrained decoding and validation is the last gate. Output is constrained to the five-section schema, jurisdiction tags and citation formats. Every draft then runs through validators, and anything that fails (a missing section, a malformed citation, a source that does not resolve) triggers a retry drawing only on retrieved evidence.

Design decision: citation integrity is a retrieval-and-validation property, never a model property. Fine-tuning improves how a citation is phrased, which is precisely the wrong thing to improve if the underlying authority might not exist.

Normalise the matter#

Mark type, jurisdiction and NICE class are resolved before drafting, because they determine which authorities are admissible at all. Getting this wrong upstream produces a well-formed opinion citing correct law that does not govern.

Retrieve the admissible authorities#

Retrieval runs over the curated index, filtered by jurisdiction and class. The result set defines what the draft is permitted to cite. This step, not the prompt, is where citation integrity is decided.

Draft into the schema#

Generation is constrained to the five-section structure with jurisdiction tags and citation formats. The model fills a shape rather than choosing one.

Validate and retry#

Structure, citation format and citation resolution are checked. Failures trigger a bounded retry constrained to retrieved evidence, not a free regeneration, which would reintroduce exactly the failure being corrected.

Score it#

Every draft passes the legal-integrity harness before it reaches an attorney, so quality is a number in a log rather than an impression.

Governance#

The model is not the legal authority anywhere in this system. The approved corpus, jurisdiction rules, five-part schema, citation validator and reviewing attorney define what may reach a final opinion. Authority is layered deliberately: retrieval bounds evidence, the schema bounds form, validators bound correctness, and the attorney holds judgement. No layer can be bypassed by a better prompt.

Orchestration#

Normalise, retrieve, draft, validate, retry-on-failure, present for review. Each stage has a defined failure behaviour rather than a fallback to unconstrained generation, which is the single most common way systems of this shape leak.

Observability#

A 0–10 legal-integrity evaluation across six axes: structure, reasoning depth, citation accuracy, jurisdiction discipline, edge-case handling, and tone. Production logging compares base against fine-tuned models on the same inputs, so a regression from a model change is visible as a score movement.

Per-draft, a reviewer can inspect the retrieved authorities, the generated sections, how each citation resolved, which validations failed, what was retried, and what the attorney corrected. Attorney corrections are the most valuable signal in the system. They are the only measure of the gap that automated scoring cannot see.

A citation that does not resolve is the designed-for case, and it is uneventful: rejected, retried against retrieved evidence, and if it still cannot be supported the claim does not appear.

The harder failures are upstream. If matter normalisation picks the wrong jurisdiction, retrieval returns authorities that are real, well-formed, correctly cited and wrong, and every downstream validator passes them, because each is checking form rather than applicability. That failure is caught by the attorney or not at all, which is why jurisdiction is a hard tag through the whole pipeline rather than an inference.

Retrieval returning too little is the other real case. The correct behaviour is a draft that says less, not one that fills the gap from parametric memory, which is why retry is constrained to retrieved evidence rather than being a fresh generation.

The constraint layers held, and they held outside the conditions they were built in. The system has run in production since 2023.

• No path from the model to an unapproved authority, enforced by retrieval scope rather than instruction

• Five-section structure enforced by the decoder, so structural drift is not a failure mode that exists

• A citation that does not resolve rejects the draft, with retry constrained to retrieved evidence

• Quality measured on six axes, base against fine-tuned, so regressions surface as numbers

Directly demonstrable: the whitelisted retrieval index and the absence of any path around it, schema-constrained decoding, the citation validator and its retry behaviour, and the legal-integrity harness with its production logging.

Client-confirmed: a practising trademark attorney tested the system extensively and approved its direction and output quality.

Not published: our development-time figures for structural adherence, citation-error reduction and review speed-up. They exist in our internal records, they were never independently measured, and our evidence standard does not let an unmeasured number become a published result. We would rather hand you a constraint you can verify than a percentage you have to accept.

If you are building in a domain with this tolerance profile (where being right most of the time is worth roughly nothing) the useful question is not how accurate your model is. It is which of your failure modes you have made structurally impossible, and which you are still hoping the model avoids.