When is extracted document data ready for an agent to act on? Only after it has been checked across the whole packet. Extraction reads each document and returns its fields, one document at a time. A lending, claims or compliance decision depends on whether those fields agree with each other, whether they describe the same person and period, and whether anything is missing. We put a reasoning layer between extraction and action that resolves entities, reconciles figures, checks completeness, keeps a page and span on every value, and passes a policy gate before any write.
The hard part is that extraction can be excellent and the decision still wrong. A model can read every pay stub in a loan file correctly and every bank statement correctly. The borrower's stated income can still fail to hold up, because the deposits do not match the pay, or one month of statements is missing, or the employer on the stubs is not the one on the application.
None of those failures shows up in a field-accuracy score, because each document was read correctly. The error lives in the relationship between documents, and a pipeline that stops at extraction hands that relationship to whoever reads the output next. Usually that is a person doing the comparison by hand, page after page, which is the work the system was bought to remove.
The second difficulty is that the wrong answer looks right. On our DocVerse ESG filings pipeline, retrieving a correct figure from the wrong reporting year produced output that looked exactly like success. The number was real, correctly formatted and cited. It belonged to the wrong year, and nothing downstream could tell.
This guide explains why extraction stops short of action, and how a packet workflow differs from chat over documents. It then walks through the pipeline stage by stage, covers grounding and validation, and sets out the gate an agent passes before it writes. It closes with illustrative loan, claim and compliance packets, how to evaluate the whole case, and where people stay in the loop.
Why is extraction not enough for an agent to act on?
Extraction answers a question about one document: what does this page say? An action answers a question about a case: what is true about this borrower, this claim or this supplier, given everything in the file? The second question needs the documents compared with each other, and extraction never does that comparison.
This figure is a framework: its lending documents and fields illustrate the shape of the problem rather than any client's schema, and it reports no measured result.
Four kinds of cross-document failure account for most of what we design against. Each passes a per-document check, which is why it reaches the decision.
| Failure | What it looks like | Why per-document extraction misses it |
|---|---|---|
| Entity mismatch | "J. Smith" on a pay stub, "Jane Smyth" on the ID, a different employer name on the application | Each name was read correctly; nobody asked whether they are one person |
| Contradiction | Stated income disagrees with pay stubs, or pay stubs disagree with deposits | Each figure is right for its own document |
| Period or unit mismatch | A figure from last year's filing, or thousands read as units | The number is on the page; its context is elsewhere |
| Missing document | Two months of statements where the checklist needs three | There is no page to extract from, so no error is raised |
Alberto Gimeno, writing for the Forbes Technology Council on 18 September 2026, makes the same point about the market. He describes teams that build clean extraction pipelines and assume something downstream will turn fields into insight, and says it rarely does. His test for a vendor is a useful test for your own system: hand it a complete loan file and ask whether the borrower's income holds up across every document in it.
How is a document packet different from RAG over documents?
A packet workflow and chat over documents both read documents with a model, and there the likeness ends. Our guide on why RAG systems fail in production covers the chat case, where a user asks a question and the system retrieves passages to answer it. A packet workflow starts from a case with a checklist, and has to account for every document in it.
| Chat RAG over documents | Packet workflow | |
|---|---|---|
| Starts from | A user's question | A case and its checklist |
| Unit of work | A few retrieved passages | Every document in the packet |
| Completeness | Top-k is enough if it holds the answer | A missing document is a finding |
| Output | An answer with citations | Reconciled fields, open conflicts, a routed action |
| Worst failure | A confident answer from the wrong passage | A write to a system of record from values that disagree |
| Who acts next | The person reading the answer | An agent's tool call, or a reviewer |
Of those differences, completeness changes the engineering most, because it decides which documents the system may skip. Retrieval is allowed to ignore documents that look irrelevant, and ranking decides which ones are read. A packet workflow must read everything, because the document a ranker would skip is often the one that contradicts the others. We treat the packet as a closed set with a manifest, and every step reports against the manifest.
What does the packet pipeline look like?
The pipeline has eight stages, and each one adds something the next stage checks. The model does the reading and proposes matches. Code resolves, checks and writes. We draw the stages separately because each fails in its own way, and a failure should point to one stage in the trace rather than to "the AI".
This figure is a framework showing stage order and the three routes out of the pipeline, and it does not describe the internals of any product.
| Stage | Job | Failure it prevents |
|---|---|---|
| Parse | Keep layout, tables and page structure | A number separated from the row and column that give it meaning |
| Extract | Typed fields, each with a span and a confidence | Values with no source and no measure of doubt |
| Resolve | One entity per real party, across documents | Two people treated as one, or one person as two |
| Reconcile | Compare figures that should agree, align periods and units | Contradictions passed through as facts |
| Complete | Check the packet against the case checklist | A decision made on a partial file |
| Validate | Deterministic rules on types, ranges, dates and sums | Impossible values reaching the gate |
| Policy gate | Decide who may write what, for this case | An agent writing where a person must decide |
| Act | Write with an audit record, queue for review, or request a document | Actions with no trail back to the evidence |
Parse without flattening#
Parsing decides whether the evidence survives. Converting a table to a text blob keeps the numbers and loses what each number is a figure of, and footnotes that qualify a figure end up pages away from it. On DocVerse we built layout-aware parsing that keeps tables as tables and attaches footnotes to what they qualify, because every later stage depended on it.
Parsers built for agents now return this structure directly, instead of leaving the next stage to rebuild it. LandingAI's second-generation Agentic Document Extraction, announced on 8 September 2026, returns a hierarchy of pages, blocks, lines and words, with a stable ID on each block and grounding at line or word level. Whatever parser you use, keep its structure and IDs all the way to the write, because they are what the grounding later points at.
Extract with spans and confidence#
Extraction fills a typed schema per document type, and every field carries three things beyond its value. It holds the span it came from, a confidence score, and the extractor version that produced it. The schema is a contract, and structured output with a validator on it keeps the model from inventing fields or returning prose where a date belongs.
Confidence is a routing signal that decides where a value goes next. On DocVerse every extracted field carries a score, and a low one sends the value to review instead of into a dashboard as accepted data. Without that, a reviewer faces a binary choice between trusting everything and re-checking everything, and they will sensibly choose the second.
Resolve entities across documents#
Entity resolution decides which names, accounts, addresses and employers refer to the same real party. We match on hard keys first, such as a policy number, a tax ID or an account number. The model proposes fuzzy matches only where no key exists, such as "Smith, Jane" against "Jane Smith" at the same address.
A proposed match never merges silently. Code records it as a proposed link with its evidence, and a match below the business's threshold goes to review. A false merge is the most expensive error in the pipeline, because every later check then compares one person's income with another person's deposits and finds them consistent.
Reconcile across documents#
Reconciliation compares values that should agree and records the result either way. Stated income against pay stubs, pay stubs against deposits, a claimed loss date against a police report, and a supplier's certificate expiry against the audit date are each a rule someone can write down. We keep those rules in code, versioned, with a tolerance owned by the business.
When values disagree, the pipeline records a conflict rather than choosing a winner quietly. A source precedence rule can settle some conflicts, such as a bank statement outranking a self-reported figure, and the rule is written down with its owner. Conflicts that no rule settles stay open. On FinSight, a fund-operations reconciliation platform we built and tested internally, an unresolved break holds its own state until someone resolves it. It is never netted into a total so the numbers balance.
Check the packet is complete#
Completeness is checked against a checklist for the case type, before any conflict is judged. A missing document is a finding with its own route, which is to ask for it. Judging a conflict on a partial file wastes review time, and the missing document often settles the conflict on arrival.
The checklist also has to catch the gaps that are easy to read past. Three months of statements where one month covers only half its period, a signed form with no signature on the last page, and an expired identity document all count. Each one needs a rule, and the rules belong to the team that owns the process.
How do you ground every value to a page and span?
Grounding means every value an agent might act on can be opened at the place it came from. We store it as a citation on the value itself: the document ID, the page, the block or span ID from the parser, the coordinates on the page, and the extractor version. A reviewer clicks the value and sees the highlighted span.
Derived values carry the lineage of every input they were computed from. An average monthly income over three pay stubs points to all three spans and to the rule that computed it. A figure with no path back to a page is treated as unsupported, however plausible it looks. We apply that rule at the gate rather than at display time, so an unsupported value cannot reach a write.
Grounding also decides how quickly a reviewer can judge a contradiction. A conflict shown as two numbers asks the reviewer to go and find both. The same conflict shown as two highlighted spans, side by side, takes seconds to judge. Most of the saving in a review queue comes from that one design choice.
We made lineage a property of the data on both DocVerse and FinSight, rather than a report assembled afterwards. Lineage cannot be added later to figures that were computed without it. A pipeline that drops the span at any stage has a hole in the trail, and holes are found at audit. The groundedness entry covers how to score whether an answer is supported by its sources.
What should validation check after extraction?
Validation runs deterministic rules over the extracted and reconciled values before the gate sees them. A model can read a date wrongly, and a rule can tell that a policy start date after its end date is impossible. Rules are fast, repeatable and easy to review, so everything that can be a rule should be one.
| Check | Example |
|---|---|
| Type and format | A sort code has the right shape; a date parses |
| Range | A percentage lies between 0 and 100; a claim amount is positive |
| Date order | A loss date falls inside the policy period |
| Arithmetic | Line items sum to the stated total; net plus tax equals gross |
| Cross-field | The currency on the invoice matches the currency on the order |
| Freshness | An identity document or certificate is in date on the decision date |
Model-based checks come after the rules, and they handle what rules cannot, such as whether a free-text description of damage is consistent with the claimed cause. Their output is a flag with a reason and a span. It is never a verdict. IBM's watsonx Orchestrate write-up from 9 September 2026 describes the same shape, with agents doing verification and validation tasks on extracted fields before a routing agent decides what to do.
What should the gate check before an agent writes?
The gate is the last step before an agent's tool call changes a system of record. It runs in code, in a fixed order, and the first failure decides the route: request a document, send to review, or deny. An agent that passes the gate writes through a narrow tool, and the tool records which evidence justified the write.
This figure is an illustrative model of a claims gate. The order is the one we use, and the checks are examples; your checklist and tolerances come from the team that owns the process.
Each position in the order has a reason. Completeness goes first because a missing document can settle a conflict. Grounding goes next because an unsupported value cannot be reconciled. Identity follows because every later comparison assumes the parties are resolved. Conflicts are judged only once those three pass, and policy runs last because it decides whether this agent, for this case, may write at all.
The policy check is a permissions question, and our guide on scoping AI agent permissions covers how to bind each write to a principal, a task and an expected state. The agent never gets a general write tool that can change any field. It gets typed tools such as "set claim status to approved for payment" with the case ID and the evidence set as arguments, and the tool itself refuses a write the gate has not cleared.
Soft cases can use a decision model. A conflict inside tolerance but close to its edge, or an entity match just above threshold, is a bounded judgment. Our guide on introducing Jev into agentic workflows covers how to ask it as a typed question with a probability that code thresholds. Hard failures still route in code without asking anything.
Every route writes a decision record: the case, the checks run, the evidence set, the route taken and who or what decided. Our guide on building an audit trail for AI decisions sets out that record. When a claim is challenged months later, the record should show which pages the decision rested on and which version of each rule applied.
What does this look like on a loan, claim or compliance packet?
The same pipeline serves each packet type, and what changes is the checklist, the reconciliation rules and the write. The three packets below are illustrations, built to show where the cross-document work sits. They describe no client's process, and they carry no results from any engagement.
| Packet | Typical documents | Cross-document question | Conflict that matters | The write |
|---|---|---|---|---|
| Loan file | Application, pay stubs, bank statements, tax return, ID | Does the stated income hold up across every document? | Deposits that do not match pay, an employer that differs | Move to underwriting, or request documents |
| Insurance claim | Claim form, policy, invoices, photos, third-party report | Is this loss covered, for this policyholder, on this date? | A loss date outside the policy period, invoices that do not sum | Set a claim status, or refer to an adjuster |
| Compliance packet | Supplier certificates, audit reports, attestations, registry data | Is every required control evidenced and in date? | An expired certificate, an attestation for a different entity | Mark a control evidenced, or open a finding |
On the loan file, the question that matters is income, and it spans at least three documents. The pipeline resolves the borrower and employer across them, then reconciles stated income against pay against deposits for the same months. It checks the months are all present, then gates on policy. A single missing statement month stops the case and asks for it, without anyone reading the rest.
On the claim, the question is coverage, and the dates carry most of the risk. The loss date must fall inside the policy period, the invoices must sum to the claimed amount, and the claimant on the form must resolve to the policyholder. The write is a status change, and only after every value it depends on has a span a reviewer could open.
On the compliance packet, the question is whether each required control has in-date evidence for the right entity. The failure that matters is an attestation that is valid for a parent company or a sister entity, which reads correctly and applies to nobody in scope. Entity resolution does most of the work here, and a false merge is the error to guard against.
How do you evaluate cross-document reasoning?
Evaluate the case, not the field. A field-accuracy score tells you the extractor reads documents well. It says nothing about whether the pipeline catches a contradiction, notices a missing document, or merges two people who are not the same, and those are the failures that reach a decision.
Field-level measurement still belongs at the first layer, where it tunes the extractor. IBM documents accuracy, precision, recall and F1 per field and per document for its extractor, which is the right tool for that job. A case-level eval set sits on top of it, built from real packets with the answer a senior reviewer would give.
We seed the case set with the failures we most need to catch, because real packets contain few of them. We add a packet with one statement month removed, a second with a pay figure altered, and a third with a name variant that belongs to a different person. Each seeded case has a known right answer, so the metrics mean something.
| Measure | What it tells you |
|---|---|
| Conflict recall | Of the seeded contradictions, how many were raised |
| Missing-document detection | Of the incomplete packets, how many were stopped and asked for the right item |
| False merges | How often two different parties were resolved as one |
| Grounding coverage | The share of values reaching the gate with a span that opens |
| Route agreement | How often the pipeline's route matched the senior reviewer's |
| Reviewer overturns | How often review changed what the pipeline proposed, per check |
Our guide on how to build LLM evals covers how to build the set from real failures and when a model judge is justified. For packets, most checks can be exact, because the right route for a seeded case is known in advance.
Where do people stay in the loop?
People review the cases that earn their time, and the review queue is designed around the evidence. A reviewer sees the conflict or the low-confidence value with the spans highlighted side by side, the rule that fired, and the route the pipeline proposes. Their decision is recorded with a reason, and it becomes a labelled case for the eval set.
Every rule in the pipeline has a named owner. The business owns the checklists, tolerances and source precedence rules and changes them through review like code, while engineering owns the pipeline and its gates. A tolerance changed in a spreadsheet by someone outside that process is a silent change to every future decision, and the audit record will not explain it.
Review volume is worth watching for each check separately, as well as in total. A check that sends most cases to review is either catching a real problem in the documents or set too tight, and the overturn rate tells you which. A check that never fires may be broken, or may be looking for something that no longer happens. Our autonomy guide covers how a route earns the right to write without review, and how it loses that right when its evidence drifts.
We build this kind of system through our document extraction work, where every field traces to a span a reviewer can open. The same pattern applies whether the parser is ours, a vendor's or a mix.
Common questions
What is the difference between document extraction and document intelligence?#
Extraction reads one document at a time and returns its fields. Document intelligence, in the sense this guide uses, reasons across a set of documents. It resolves the parties, reconciles figures that should agree, checks the packet is complete and routes the case to an action. An agent needs the second before it can act safely on the first.
Why can't an AI agent act directly on extracted fields?#
Because the fields can each be right and the case still wrong. Stated income may not match deposits, a name may belong to a different person, a figure may come from the wrong year, or a required document may be missing. None of those shows up in a per-document check, and each changes the decision.
How is cross-document reasoning different from RAG?#
RAG retrieves a few passages to answer a user's question and may ignore documents that look irrelevant. A packet workflow starts from a case and a checklist, must account for every document in it, and treats a missing document as a finding. Its output is a routed action with open conflicts listed, rather than an answer with citations.
What does grounding to a page and span mean?#
Every value carries a citation to where it came from: the document ID, the page, the block or span ID, the coordinates on the page and the extractor version. A reviewer can open the value and see the highlighted source. Derived values point to all their inputs, and a value with no path back to a page is treated as unsupported.
How should an agent handle contradictions between documents?#
Record the conflict with both values and both spans, and settle it only by a written source precedence rule that the business owns. Conflicts no rule settles stay open and go to review, shown side by side. The pipeline should never choose a winner quietly or net a difference away so totals balance.
What should happen when a document is missing from the packet?#
The pipeline should stop the case and request the specific missing item before it judges any conflict. A missing document often settles the conflict when it arrives, and judging a partial file wastes review time. The checklist that defines completeness belongs to the team that owns the process.
How do you evaluate a document intelligence pipeline?#
Measure the case rather than the field, starting from a set of real packets with the route a senior reviewer would choose. Then seed the set with removed documents, altered figures and name variants. Track conflict recall, missing-document detection, false merges, grounding coverage, route agreement and reviewer overturns, alongside field accuracy for the extractor.
Where should people review document-driven decisions?#
On conflicts, low-confidence values, uncertain entity matches and any write the policy gate reserves for a person. The review screen should show the rule that fired, the spans side by side and the proposed route. Each decision is recorded with a reason and added to the eval set.
Further reading
- Why RAG systems fail in production. The chat-over-documents case this guide is distinct from.
- How to scope AI agent permissions. Binding each write to a principal, task and expected state.
- How to build an audit trail for AI decisions. The decision record each route writes.
- How to roll out AI agent autonomy in five levels. How a route earns the right to write without review.
- How to build LLM evals. Building the case set from real failures.
- Introducing Jev into agentic workflows. Asking soft gate questions as typed judgments with probabilities.
- DocVerse: per-field confidence and the claim-to-evidence path. Our ESG filings pipeline, with layout-aware parsing and page-level lineage.
- FinSight: lineage as a data property. Reconciliation where an unresolved break holds its own state.
- Alberto Gimeno, Forbes Technology Council, Document AI's shift from reading pages to reasoning across them, 18 September 2026. The case for a reasoning layer, and the complete-loan-file test.
- LandingAI, Introducing Agentic Document Extraction, 2nd Generation, 8 September 2026. Hierarchical parse output with stable IDs and line or word grounding.
- IBM, Turn document processing into an agentic workflow with watsonx Orchestrate, 9 September 2026. Verification and validation by agents after extraction.
- IBM, What's new in watsonx Orchestrate. Field-level and document-level accuracy evaluation for the extractor.