Self-Evolving Financial Ontologies: How Multi-Agent AI Learns to Discover Enterprise Schemas from Financial Documents

DataOps Labs · Education and Knowledge Democratised
Disclaimer: These are my personal views. This is a reference architecture for discussion, not a description of any employer's or vendor's production system.
TL;DR
- A 450-page credit agreement isn't a pile of paragraphs. It's a web of defined terms, cross-references, schedules and conditions. RAG can answer questions about it, but it can't turn it into governed, comparable data.
- Every financial document is a little different, so you can't fix a schema in advance. You also can't let an LLM invent a new one for every document. The schema has to evolve, under control.
- The architecture has three building blocks:
- An ontology. FIBO, pinned to one release, gives every concept a stable meaning. Agents read it through Open Knowledge Format (OKF) bundles.
- A schema registry. It versions what you capture, the way a package registry versions code.
- Two learning loops that are never merged. Loop 1 changes what you capture. Loop 2 improves how well you capture it.
- "Self-evolving" must never mean "self-modifying". Agents propose. Validators check. Humans approve. Everything is versioned.
How to Read This Article
The article is long because the problem has several layers. Different architects will care about different parts, so here's a map:
| If you are a… | Start with | You'll get |
|---|---|---|
| Enterprise Architect | Sections 1, 2, 5, 13, 14 | Why this needs to be a platform capability, how the governance works, how it scales across business lines |
| Solution Architect | Sections 6–9, 15, 16 | The end-to-end flow, the agent responsibilities, a reference implementation, a delivery roadmap |
| Data Architect | Sections 3, 4, 10 | The three-layer semantic model, schema versioning, lineage, memory design |
| AI / ML Engineer | Sections 7, 11, 12 | Agent contracts, GEPA optimization, the evaluation framework |
| Ontology Engineer | Sections 3, 8 | FIBO alignment, the OWL → OKF compilation approach, SHACL validation |
| Platform Engineer | Sections 6, 14, 15 | Ingestion, the harness, multi-tenancy, deployment |
A small glossary before we start:
- Ontology: a formal dictionary of concepts and how they relate. For example, "a Guarantor is a kind of Obligor".
- FIBO: the Financial Industry Business Ontology, the industry's shared ontology for finance.
- OKF: Open Knowledge Format, an open format from Google that stores knowledge as Markdown files with YAML metadata, including who wrote each concept, who verified it, and when.
- Schema: the list of fields you extract, with their types and rules. For example,
max_total_leverage_ratio: ratio, required, evidence needed. - Schema registry: the versioned, authoritative catalog of schemas.
- SHACL: a W3C standard for validating graph data, e.g. "every facility must have exactly one currency".
- GEPA: a method that improves agent prompts by reading what went wrong in past runs.
- HITL: human-in-the-loop review.
1. A Monday Morning Problem
Picture a credit portfolio team on a Monday morning. Over the weekend, 40 new and amended credit agreements arrived. Each one is between 200 and 600 pages. The head of credit risk asks three questions:
- "What is the maximum leverage ratio in each agreement, with its step-downs and test dates?"
- "Which agreements allow incremental debt without lender consent, and up to how much?"
- "From this quarter onwards, I also want to track sustainability-linked margin adjustments."
The first question looks like a lookup. The second needs reasoning across clauses. The third isn't a question at all: it's a change to what the organization captures.
Most GenAI document programs are built for the first kind of question. This article is about building for all three.
2. Why RAG Is Not Enough
The standard pipeline looks like this:
PDF → text → fixed-size chunks → embeddings → vector database → top-k zretrieval → LLM answer
It's a good pipeline for question answering. It breaks in four places when the goal is governed data.
Figure 1 — Left: RAG splits the agreement into chunks and loses the links between them. Right: a structure-aware model keeps definitions, cross-references and schedules connected, so every value can be traced to its evidence.
2.1 Definitions live far from where they're used
In a typical credit agreement, the definitions article can run to dozens of pages. "Consolidated EBITDA" might be defined on page 19, adjusted by a schedule on page 590, and tested in a covenant on page 233. A chunk containing "the Consolidated Total Leverage Ratio shall not exceed 4.50:1.00" means little without the chain of definitions behind it. Similarity search doesn't reliably bring that chain back.
2.2 Cross-references are the structure
Phrases like "subject to Section 2.14(d)", "as set forth in Schedule 2.01" and "except as permitted under Section 7.02(k)" aren't noise. They are the meaning of the document. Fixed-size chunks cut exactly at those joins.
2.3 An answer isn't a record
RAG returns a paragraph. A portfolio needs the same record shape for every agreement, such as:
{
"leverage_covenant": {
"max_ratio_schedule": [
{"test_period_end": "2027-03-31", "max_ratio": 4.50},
{"test_period_end": "2028-03-31", "max_ratio": 4.00}
],
"test_frequency": "quarterly",
"equity_cure": true
}
}
2.4 Nothing learns and nothing is versioned
When an analyst corrects a RAG answer, the correction disappears with the chat session. When the business asks for a new data point, there's no schema to extend, only a prompt to edit. And without a schema, there's nothing to audit.
Key idea: The hard problem isn't extraction. It's knowledge engineering: deciding what the information model is, keeping it consistent across documents that are all different, and changing it safely as needs grow.
3. The Heterogeneity Problem: Same Family, Different Documents
All the documents in our scenario are financial agreements, and they share a skeleton: parties, definitions, facilities, pricing, repayment, covenants, collateral, events of default, schedules. But the details vary a lot:
| Variant | What's special about it |
|---|---|
| Syndicated revolving facility | Per-lender commitments, letter-of-credit sublimits, swingline loans, commitment fees |
| Term Loan B (leveraged) | Often covenant-lite; incremental capacity, MFN protection, excess cash flow sweep |
| Asset-based lending (ABL) | Borrowing base, eligibility criteria, advance rates, reserves |
| Sustainability-linked loan | KPIs, sustainability performance targets, margin ratchets, verification |
| Project finance | Cash waterfalls, reserve accounts, completion tests |
| Amendment and restatement | Changes an existing agreement, so you must know what it amends |
On top of that comes the long tail: bespoke definitions, jurisdiction-specific clauses, deal-specific carve-outs.
That leaves three bad options and one good one:
- ❌ One giant fixed schema: hundreds of mostly empty fields, still incomplete, slow to change.
- ❌ An LLM-generated schema per document: nothing is comparable and field names drift.
- ❌ No schema, just RAG: see Section 2.
- ✅ A governed, layered, versioned schema that evolves: a stable core, variant overlays, a controlled path for new concepts, and one registry that says what "the schema" is at any point in time.
The rest of the article builds option 4.
4. Building Block 1: An Ontology as the Semantic Anchor
4.1 Why an ontology at all?
Take six words from the parties section of a credit agreement: Borrower, Obligor, Loan Party, Guarantor, Administrative Agent, Lender.
A naive extractor creates six independent fields. But these are roles that parties play in a contract, and they relate to each other. A Guarantor is an Obligor but not a Borrower. "Loan Parties" is usually a defined term that groups the Borrower and the Guarantors. An ontology lets you model this once, correctly and reuse it across every agreement.
4.2 Why FIBO
The Financial Industry Business Ontology (FIBO) is the industry's shared, formal model of financial instruments, parties and contracts. Some facts architects should know:
- Stewardship: FIBO is developed by the EDM Council, now EDM Association following its combination with the Object Management Group (OMG). The acquisition of OMG's assets completed on 1 October 2025.
- Format and license: FIBO is published as OWL 2 ontologies, developed openly on GitHub under the MIT license, and browsable in the FIBO Viewer at spec.edmcouncil.org.
- Cadence: releases ship quarterly, and they're not cosmetic. Recent releases removed long-deprecated concepts, and the release notes warn of more removals to come.
Some FIBO concepts that matter for credit agreements:
| What you need | FIBO concept or module |
|---|---|
| The agreement | CreditAgreement: https://spec.edmcouncil.org/fibo/ontology/FBC/DebtAndEquities/Debt/CreditAgreement |
| Secured facilities | CollateralizedLoan: https://spec.edmcouncil.org/fibo/ontology/LOAN/LoansGeneral/Loans/CollateralizedLoan |
| Events of default, covenant breaches | Credit Events ontology (FBC / Debt and Equities) |
| Prepayment, default proceedings | Loan Events ontology (maturity: provisional) |
| Letters of credit, guarantees | Guaranty ontology (FBC / Debt and Equities) |
FIBO's CreditAgreement already says what we need: a contract in which a debtor receives something of value and agrees to repay a creditor, with restrictions such as has contract party some creditor and has contract party some debtor. Your schema should inherit that meaning rather than reinvent it.
4.3 The three-layer model
Don't copy FIBO as your extraction schema. FIBO is a conceptual model. It doesn't say "read the applicable margin from the pricing grid in Schedule 1.01(a)". Keep three layers with explicit, versioned mappings between them:
| Layer | What it holds | Owner | Change rate |
|---|---|---|---|
| 1. Industry ontology | FIBO, pinned release | EDM Association upstream; you decide when to upgrade | Quarterly upstream |
| 2. Enterprise schema | Your concepts and attributes, including extensions FIBO doesn't have | Business data owner | When the business needs new data |
| 3. Extraction profiles | How to find each attribute in a given document type | Platform team + domain experts | Whenever quality or coverage improves |
4.4 Delivering FIBO to agents: OKF as the knowledge layer
Agents shouldn't parse OWL or load thousands of FIBO classes into a prompt. They need knowledge they can read, with provenance attached. Google's Open Knowledge Format (OKF) v0.2 fits this well. It represents knowledge as a folder of Markdown "concept" files with YAML frontmatter, and v0.2 adds trust and lifecycle signals:
| OKF v0.2 field | How this architecture uses it |
|---|---|
type, title, resource |
The kind of concept and its canonical IRI |
sources |
What the concept came from: a FIBO release, a clause, a customer decision |
generated: {by, at} |
Who wrote it: a compiler, an agent version or a person |
verified: [{by, at}] |
Confirmation events. Trust tiers come from these: unverified, machine-confirmed, human-reviewed (human: actor). This is your human-in-the-loop record. |
status |
draft → stable → deprecated |
stale_after |
When the concept must be reviewed again, e.g. at the next FIBO release |
Attested Computation |
An approved way to compute a value (leverage ratio, borrowing base). Agents supply parameters but never write the formula. |
log.md |
A dated change history for the bundle |
One critical caveat. In OKF v0.2, links between concepts are untyped. A link says a relationship exists, and the prose says what kind. FIBO's value is in its typed semantics: subclasses, restrictions, property domains. If you hand-convert FIBO into plain OKF files, those semantics become prose that agents can read but no validator can enforce.
Key idea: Compile OKF from OWL, never the reverse. OWL stays the source of truth for meaning. OKF is the agent-friendly view generated from it.
Figure 2 — FIBO (pinned) and your enterprise extensions live in OWL/SHACL. A deterministic compiler produces OKF bundles per domain. Agents read the bundles, and validators enforce the OWL/SHACL source. Mappings between layers carry an explicit state: exact, close, extension, unmapped or rejected.
The compiler writes typed relationships into custom frontmatter keys. OKF requires consumers to preserve unknown keys, so this stays conformant:
---
type: Ontology Class
title: Credit Agreement
resource: https://spec.edmcouncil.org/fibo/ontology/FBC/DebtAndEquities/Debt/CreditAgreement
fibo_release: "2026Q3"
fibo_restrictions:
- "hasContractParty some Creditor"
- "hasContractParty some Debtor"
status: stable
generated: { by: fibo-okf-compiler/1.0, at: "2026-10-01T00:00:00+00:00" }
verified:
- { by: process:owl-consistency-check, at: "2026-10-01T02:00:00+00:00" }
- { by: human:ontology-steward-07, at: "2026-10-03T04:00:00+00:00" }
stale_after: "2027-01-15T00:00:00+00:00"
---
A credit agreement is a contract in which a debtor receives something of value
and agrees to repay a creditor, generally with interest…
Two practical notes:
- Use full date-times with a UTC offset in
stale_afterand similar keys. A spec revision in August 2026 made date-only values non-conforming, even though the version number stayed at 0.2. - OKF's version number is the format's version, not your content's. Version your bundles yourself (git tags + semver), and pin the OKF spec revision you build against, just as you pin FIBO.
4.5 Three rules I never break
- Never invent an official FIBO IRI. Every IRI must resolve in the pinned release. Research on ontology grounding (SPIRES/OntoGPT) shows that grounding by lookup is far more reliable than letting an LLM generate identifiers. In those tests, raw LLMs sometimes returned identifiers from the wrong ontology.
- Unmappable attributes stay enterprise extensions. "Relationship Manager Review Status" has no FIBO equivalent, and that's fine. Mark it
extension. - Pin the release and plan for drift. J.P. Morgan AI Research ("Ontology drift is a challenge for explainable data governance", arXiv 2108.05401) found that up to a third of FIBO could change in a quarter. A FIBO upgrade is a planned compatibility exercise, not a silent runtime change.
5. Building Block 2: A Schema Registry That Can Evolve
If you remember one component from this article, make it this one. The schema registry is the single source of truth for what we capture, and it's what most RAG programs are missing.
A useful analogy for platform engineers: treat schemas the way you treat software packages. They have versions, dependencies, release notes, deprecation policies and consumers who need to know when something breaks.
5.1 Core + overlays
Instead of one monolithic schema, use a core plus variant overlays, and resolve them for each document:
| Schema module | Example attributes |
|---|---|
credit_agreement.core |
Parties and roles, facilities, pricing, maturity, events of default, governing law |
overlay.revolver |
Per-lender commitments, LC sublimit, swingline, commitment fee |
overlay.term_loan_b |
Incremental capacity, MFN, ECF sweep, springing covenant |
overlay.abl |
Borrowing base, advance rates, eligibility, reserves |
overlay.sustainability_linked |
KPIs, targets, margin ratchet, verification |
overlay.amendment |
Amended agreement reference, changed provisions |
One agreement can activate several overlays. Revolver + term loan + sustainability-linked pricing is common. The registry resolves an effective schema for each document:
{
"document_id": "doc-ca-0042",
"effective_schema": {
"core": "credit_agreement.core@2.3.0",
"overlays": ["overlay.revolver@1.4.0", "overlay.sustainability_linked@1.1.0"],
"fibo_release": "2026Q3"
},
"overlay_evidence": [
{"overlay": "overlay.revolver", "section": "2.01(b)", "page": 41},
{"overlay": "overlay.sustainability_linked", "section": "2.12", "page": 77}
]
}
You get comparability (every agreement shares the core), specificity (overlays carry the variant detail) and governability (each module versions on its own).
5.2 The attribute contract
Every attribute follows the same contract, independent of how any LLM happens to phrase things:
{
"attribute_id": "attr-fin-cov-max-leverage",
"canonical_name": "max_total_leverage_ratio",
"concept": "FinancialCovenant.LeverageCovenant",
"business_definition": "Maximum permitted ratio of Consolidated Total Debt to Consolidated EBITDA, as defined in the agreement",
"aliases": ["Total Leverage Ratio", "Consolidated Leverage Ratio"],
"data_type": "ratio_schedule",
"cardinality": "0..* (one per test period)",
"depends_on_definitions": ["Consolidated Total Debt", "Consolidated EBITDA"],
"semantic_mapping": {
"ontology": "FIBO", "release": "2026Q3",
"target_iri": null, "mapping_status": "close",
"rationale": "Related to FIBO credit-event concepts; no exact property for the threshold"
},
"validation_rules": ["ratio > 0", "test periods ordered", "step-downs non-increasing unless stated"],
"evidence_requirement": "page + clause + bounding box",
"lifecycle_status": "stable",
"introduced_in": "credit_agreement.core@2.0.0"
}
Two fields do most of the work with heterogeneous documents. aliases absorbs terminology differences between law firms. depends_on_definitions tells extraction to resolve the definition chain first.
Watch out: aliases that only look equivalent. "Total Net Leverage Ratio" and "Senior Secured Leverage Ratio" are different metrics. Aliases tell the system where to look. Equivalence is a decision a human approves, never something string similarity decides.
5.3 Semantic versioning for schemas
Figure 3 — How the schema grows safely. Overlays absorb document variety. Uncovered content becomes candidate proposals only when it recurs. Breaking changes go through a deprecation window: v2.3 → v2.4 (add new attributes and deprecate the old one) → v3.0 (remove it).
| Change | Bump | Examples | Reprocessing | Consumer impact |
|---|---|---|---|---|
| Patch | x.y.Z | Better definition, new alias, refined instruction | None required | None |
| Minor | x.Y.0 | New optional attribute, new overlay, new enum value | Targeted: only relevant sections | Additive |
| Major | X.0.0 | Split/merge/rename, type or cardinality change, optional → required | Targeted or full + migration mapping | Breaking; notify and migrate |
Breaking changes always get a deprecation window. Say the business wants to split termination_condition into trigger, notice period and approval authority:
- v2.4.0 (minor): add the three new attributes, mark the old one
deprecatedbut keep populating it, and publish a migration mapping. - Consumers migrate within the agreed window.
- v3.0.0 (major): remove the old attribute.
No dashboard breaks overnight.
5.4 Every value knows where it came from
Each extracted value carries a lineage stamp:
{
"document_id": "doc-ca-0042",
"document_sha256": "9f2c…",
"attribute_id": "attr-fin-cov-max-leverage",
"schema_version": "credit_agreement.core@2.3.0",
"value": [{"test_period_end": "2027-03-31", "max_ratio": 4.50}],
"status": "extracted",
"evidence": [
{"page": 233, "section": "7.11(a)", "bbox": [72, 188, 540, 402]},
{"page": 19, "section": "1.01", "term": "Consolidated Total Debt"}
],
"lineage": {
"parser_version": "layout-parser@3.2.1",
"agent_prompt_version": "value-extractor@1.7.0",
"model_id": "<governed-llm-endpoint>",
"fibo_release": "2026Q3",
"okf_bundle": "corporate-credit@2026.11.0"
},
"review": {"status": "pending", "reviewer": null}
}
This answers the auditor's question ("why does the system say 4.50, and would it still say that today?") and the engineer's question ("which records do I reprocess after this change?").
Key idea: "Not found" is a valid answer. Covenant-lite deals often have no maintenance leverage covenant. Record
status: "not_found"with the sections searched. Never let a model fill in a required field just because the schema expects one.
6. The End-to-End Flow
Here's the whole pipeline in one view before we go into each part.
In words:
| Step | Component | What happens | Output |
|---|---|---|---|
| 0 | Ingestion | Parse layout, tables and hierarchy; resolve definitions and cross-references | Canonical document model + evidence index |
| 1 | Agent 1: Classify | Identify document family, section types and overlays, using OKF concept lookups | Classification profile |
| 2 | Agent 2: Bind schema | Resolve the registered effective schema; propose changes only for real, recurring gaps | Effective schema + draft proposals |
| 3 | Agent 3: Extract | Extract values from routed sections with evidence; call attested computations for maths | Evidence-backed values |
| 4 | Validation gates | OWL, SHACL, JSON Schema, business rules, IRI-exists checks | Pass / fail reports |
| 5 | Risk-based HITL | Humans review exceptions, material items and samples | verified events |
| 6 | Registry + lineage store | Publish versions; store values with lineage | Auditable data |
| 7 | Chat on the harness | Answer questions; route feedback to the right loop | Answers or proposals |
The rest of this section covers ingestion. Section 7 covers the agents.
6.1 Never send 450 pages to an LLM
Long context windows are impressive, but governed extraction needs every claim to point to a page, section, clause and bounding box, and that pointer has to exist before any agent runs.
The ingestion stage produces a canonical document model:
- Stable, human-readable node IDs such as
doc-ca-0042/art7/s7.11/a/para2. - Page, bounding box and parser confidence for each node.
- Parent/child links: article → section → clause → paragraph.
- Outgoing references to definitions, sections and schedules.
6.2 Build the definitions graph first
For credit agreements, the definitions graph is the most valuable structure you can build. Parse the definitions article into a term → definition map, then link every capitalized defined term elsewhere in the document back to it. When the extractor reads Section 7.11, it gets the covenant clause plus the resolved definitions of every term in it, and nothing else.
6.3 Route sections; don't scan pages
With 450 pages and perhaps 120 attributes, extracting every attribute from the whole document is wasteful and inaccurate. Instead:
- Agent 1 labels each section with a section type (
financial_covenants,pricing,events_of_default…). - The registry keeps an attribute → section-type index.
- Agent 3 sees only the routed sections plus their resolved definitions.
This is also what makes targeted reprocessing cheap. A new events_of_default attribute touches only those sections across the portfolio.
6.4 Tooling
Benchmark on your documents. Vendor claims don't tell you how a parser handles a scanned 1990s amendment.
- Amazon Textract: OCR, forms, tables.
- Amazon Bedrock Data Automation: blueprint extraction with confidence scores and bounding boxes, plus blueprint optimization from a few labeled documents.
- Docling (IBM Research, MIT license): layout analysis, table structure and a unified document model. A good fit for self-hosted or air-gapped environments.
- Azure Document Intelligence, LlamaParse and Unstructured fill the same slot.
7. The Three Agents
Start with logical agents inside one governed workflow, not ten microservices on day one. Split deployment units only when scale, security or team ownership requires it.
Key idea: No agent is a source of truth. Agents propose; the schema registry and the ontology service decide.
7.1 Agent 1: Classification
Agent 1 classifies at three levels: document (credit agreement, amendment, guaranty, intercreditor), section (covenants, events of default…) and overlay (revolver, TLB, ABL, SLL…). It looks up concepts in the OKF bundle through an index or search tool and may only cite concepts that exist.
{
"document_id": "doc-ca-0042",
"node_id": "doc-ca-0042/art7/s7.11",
"level": "section",
"label": "financial_covenants",
"okf_concept": "/corporate-credit/section-types/financial-covenants.md",
"confidence": 0.82,
"confidence_type": "model_self_reported",
"evidence": [{"page": 233, "section": "7.11"}],
"review_status": "auto_accepted"
}
Note confidence_type. A model's self-reported confidence isn't a probability until you've calibrated it against reviewer outcomes. Base review thresholds on calibrated scores.
7.2 Agent 2: Schema binding and gap discovery
This is the agent where most teams go wrong. Agent 2 should not write a "version 1 schema" for every document. Do that and you'll end up with thousands of incomparable schemas. Agent 2 has two jobs:
- Bind: resolve the registered
stableschema for this document family (core + overlays). - Discover gaps: find substantive content the schema doesn't explain, and propose changes.
A brand-new v1 schema happens once per document family, not once per document.
Gap discovery uses a measure I call schema coverage. After extraction, every section is one of:
- Covered: at least one attribute drew evidence from it.
- Boilerplate: non-substantive (notices, counterparts, severability).
- Uncovered: substantive, but explained by nothing in the schema. These are the candidate gaps.
Two guardrails prevent schema explosion:
- Deduplicate against the registry first. Most "new" concepts are aliases of existing ones. Agent 2 must show novelty through definitions and context, not word matching. When it's ambiguous, it proposes an alias for a human to decide.
- A recurrence threshold. A concept seen in one deal is long tail. It goes into a structured document-specific notes record: searchable, but not part of the schema. Only concepts that recur across N documents, or that the business explicitly requests, become schema proposals.
Proposals are written as OKF Schema Attribute concepts with status: draft. They become stable only after validation and data-owner approval.
7.3 Agent 3: Extraction
Agent 3 reads the routed sections, the resolved definitions and the effective schema, and returns evidence-backed values. Three rules:
- Evidence or nothing. Every value needs a span that a deterministic checker can verify.
- "Not found" is allowed. It's often the correct answer.
- Never do the maths. For derived values (commitment totals, ratios), Agent 3 gathers the inputs and calls an attested computation. The LLM supplies parameters; approved code computes.
7.4 The Ontology Alignment step
Between Agent 2 and approval, every proposal is aligned to the pinned ontology:
| Mapping state | Meaning | Example |
|---|---|---|
| Exact | Same meaning as a concept in the pinned release | The agreement ↔ CreditAgreement |
| Close | Related but narrower or broader; difference documented | Leverage threshold vs. FIBO covenant-breach concepts |
| Extension | Valid enterprise concept with no FIBO equivalent | relationship_manager_review_status |
| Unmapped | Not yet analyzed | New request |
| Rejected | Mapping failed review | Words matched, meaning didn't |
What the research says about LLMs here: they work best as judges of uncertain candidates, not as unsupervised matchers. In OAEI 2025, a classical matcher that consulted an LLM only on high-uncertainty mappings ranked second overall in the Bio-ML track, while in the Conference track an LLM-agent matcher didn't beat the classical baseline. So the pattern is: deterministic candidate generation → LLM adjudication on ambiguous cases → human approval.
7.5 Human review by risk, not by layer
A human checkpoint after every agent works for 50 agreements. It doesn't work for millions of loan files, and reviewers who see too much start rubber-stamping. Set review intensity by impact and uncertainty:
| Layer | Volume | Review policy |
|---|---|---|
| Schema changes | Low | Always reviewed and approved by the data owner |
| Ontology mappings | Low–medium | Every close and extension; exact is machine-verified |
| Classification | High | By exception: low calibrated confidence, conflicting overlays, unknown family |
| Extraction | Very high | By exception: validation failure, material attributes, low calibrated confidence, plus random sampling |
Every review is written as a verified event with a human: actor. Your HITL record and your knowledge layer are the same thing.
8. Deterministic Validation: The Gate Every Handoff Passes
LLMs propose; deterministic systems validate. These four layers complement each other. None replaces another.
| Layer | What it checks | Worldview |
|---|---|---|
| OWL 2 reasoning | Ontology + extensions are logically consistent | Open world: a missing fact is unknown |
| SHACL | Extracted data has the required shape | Closed world: a missing required property is a violation |
| JSON Schema | API payloads between services | Structural |
| Business rules | Maturity after closing date, ISO 4217 currency, non-increasing step-downs | Domain logic |
A common mistake is expecting an OWL reasoner to flag "every facility must have a currency". Under the open-world assumption it won't. That's SHACL's job:
@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix ex: <https://example.org/credit#> .
ex:LoanFacilityShape a sh:NodeShape ;
sh:targetClass ex:LoanFacility ;
sh:property [ sh:path ex:commitmentAmount ; sh:minCount 1 ; sh:maxCount 1 ] ;
sh:property [ sh:path ex:currency ; sh:minCount 1 ; sh:pattern "^[A-Z]{3}$" ] ;
sh:property [ sh:path ex:maturityDate ; sh:maxCount 1 ; sh:datatype xsd:date ] .
One more check deserves its own line: "Does this FIBO IRI exist in the pinned release?" It's a simple lookup, and it eliminates an entire class of hallucination.
9. Building Block 3: Two Learning Loops, Never Merged
This is the core idea of the architecture.
Figure 5 — Loop 1 (left) turns business requests and coverage gaps into approved schema versions. Loop 2 (right) turns reviewer corrections into better agent instructions through GEPA, offline evaluation and release gates. Dashed lines show the loops informing each other; they never write to each other's artifacts.
| Loop 1: Schema Evolution | Loop 2: Extraction Quality | |
|---|---|---|
| Question | What should we capture? | How well do we capture it? |
| Triggers | Business requests; coverage gaps; recurring uncovered content | Reviewer corrections; golden-set regressions; validation failures |
| Changes | Enterprise schema, overlays, mappings | Agent instructions, few-shot policy |
| Never changes | Agent prompts | Schema, ontology, validation rules |
| Approver | Business data owner (+ ontology steward) | AI engineering + independent reviewer |
| Artifact | credit_agreement.core@2.4.0 |
value-extractor@1.8.0 |
| Rollback | Previous schema + migration mapping | Previous prompt release |
Why keep them apart? Attribution and blast radius. If one correction could change both the schema and the prompts, you'd never know which change caused the next regression, and you couldn't roll one back without the other.
10. Loop 1 in Practice: Conversational Schema Evolution
Business users shouldn't need to know OWL, SHACL or JSON Schema. They should be able to talk to the system the way they'd talk to a good credit analyst.
10.1 Two kinds of request
- Information requests run against the current schema and data: "What's the leverage covenant in the Acme agreement?"
- Schema evolution requests start Loop 1: "From now on, track sustainability-linked margin adjustments."
Asking a question about an attribute must never change the schema by itself.
10.2 What the conversation looks like
Analyst: We need to track early termination properly: what triggers it, the notice required and who has to approve it.
Assistant: Your schema (
credit_agreement.core@2.3.0) has one attribute,termination_condition, that combines all three. I propose:
Proposed attribute Type Mapping termination_triggerList of structured conditions Close (FIBO credit events), pending validation termination_notice_periodDuration + business-day convention Extension termination_approval_authorityList of typed approvals Extension I'd deprecate
termination_conditionrather than delete it, so existing reports keep working. That makes this v2.4.0 (minor). Reprocessing scope: sections typedterminationandprepaymentacross 312 agreements, about 4% of total pages. I've sent this to the data owner for approval.Data owner: Approved. Keep board approval and regulatory approval separate.
Assistant: Done.
approval_authority.typeis an enum (board,regulatory,lender_majority,other), each with evidence. I've recorded the decision so future proposals follow it. v2.4.0 is published, and targeted reprocessing has started.
The conversation isn't what matters. What matters is the structured, versioned, validated, approved change it produces.
10.3 Route every piece of feedback to the right place
The chat is a client of the harness, not the decision-maker. Feedback comes in four kinds:
| Feedback | Example | Goes to |
|---|---|---|
| Value correction | "The margin is 2.25%, not 2.50%" | Golden set → Loop 2 |
| Schema change | "Also track the excess cash flow sweep" | Draft attribute → data-owner approval → Loop 1 |
| Mapping error | "That party is a sponsor, not a guarantor" | Ontology steward |
| Classification error | "This is an amendment, not a new agreement" | Classifier golden set → Loop 2 |
Also separate who can propose from who can approve. Any analyst can propose; only the domain data owner approves.
10.4 Memory: structured, not stuffed
Pasting old chats into a prompt isn't memory. Use three kinds:
| Memory | Contents | Authority |
|---|---|---|
| Schema memory | Approved schemas, mappings, versions | Authoritative (it is the registry) |
| Interaction memory | Decisions, preferred terminology, rejected proposals | Tenant-scoped context |
| Learning memory | Corrections, golden examples, curated traces | Tenant-scoped; feeds Loop 2 only |
Multi-tenant isolation is non-negotiable. One client's confidential agreements must never become another client's training signal. On AWS, Amazon Bedrock AgentCore Memory supports namespaces with IAM conditions, and AWS's own guidance warns against relying on a tenant_id metadata filter alone. For the strictest needs, give each tenant its own memory store.
11. Loop 2 in Practice: GEPA for Extraction Quality
11.1 What GEPA is
GEPA (Genetic-Pareto) is a reflective prompt optimizer (Agrawal et al., arXiv 2507.19457, ICLR 2026 Oral). Think of it as an experienced reviewer who reads the logs of failed runs, works out why they failed, and rewrites the instructions. It runs the system, collects execution traces, has a reflection model read them in plain language, and proposes better instructions. Instead of keeping one "best" prompt, it keeps a Pareto frontier of candidates that each do best on different examples.
In the paper's final version, GEPA outperforms GRPO by about 6% on average (up to 20%) with up to 35x fewer rollouts, and outperforms MIPROv2 by over 10%. Two cautions: gains vary by task, and none of the benchmarks were financial documents. Treat the paper as evidence that the mechanism works, then prove it on your own data.
11.2 Why GEPA fits this architecture
GEPA learns from textual feedback, and this platform produces a lot of it:
- Reviewer notes: "This is the springing covenant on the revolver, not a maintenance covenant on the term loan."
- SHACL reports: "LoanFacility lacks currency."
- Evidence failures: "Value 4.50 has no supporting span in the cited clause."
- IRI checks: "Proposed IRI does not resolve in release 2026Q3."
A single accuracy score throws all of this away. GEPA can use it.
11.3 Boundaries
| GEPA may change | GEPA may never change |
|---|---|
| Agent 1/2/3 instructions | FIBO, the pinned release or OKF ontology concepts |
| Gap-discovery guidance (alias vs. new) | The authoritative schema |
| Few-shot selection policy | SHACL shapes, business rules, approval workflows |
| Output formatting guidance | Tenant isolation boundaries |
11.4 From correction to golden example
A reviewer sees that Agent 3 extracted one leverage ratio (4.50), but the agreement has a step-down schedule and a temporary holiday after acquisitions. The reviewer fixes the value and writes a note, and the platform turns it into a golden example:
{
"example_id": "gold-lev-0193",
"tenant": "tenant-a",
"section_path": "doc-ca-0042/art7/s7.11",
"attribute_id": "attr-fin-cov-max-leverage",
"expected": [
{"test_period_end": "2027-03-31", "max_ratio": 4.50},
{"test_period_end": "2027-09-30", "max_ratio": 4.25},
{"test_period_end": "2028-03-31", "max_ratio": 4.00}
],
"reviewer_note": "Step-down table in 7.11(a); acquisition holiday in 7.11(c) raises the ratio by 0.50x for four quarters — capture as a conditional adjustment, not a separate threshold.",
"schema_version": "credit_agreement.core@2.3.0"
}
The reviewer_note is what GEPA's reflection step learns from.
11.5 Implementation sketch (DSPy)
import dspy
class ExtractAttribute(dspy.Signature):
"""Extract one schema attribute from a routed credit-agreement section.
Use resolved definitions. Cite clause IDs for every value. Return status
'not_found' if absent. Never compute ratios yourself; return inputs instead."""
section_text: str = dspy.InputField()
resolved_definitions: str = dspy.InputField()
attribute_contract: str = dspy.InputField(desc="registry contract, JSON")
value: list[dict] = dspy.OutputField()
evidence: list[dict] = dspy.OutputField()
status: str = dspy.OutputField(desc="extracted | derived | not_found")
program = dspy.ChainOfThought(ExtractAttribute)
def extraction_metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
feedback = []
# 1) Correctness vs. reviewer-approved value
value_score = compare_normalized(pred.value, gold.expected) # 0..1
if value_score < 1.0:
feedback.append(f"Value mismatch. Reviewer note: {gold.reviewer_note}")
# 2) Evidence faithfulness — deterministic check against the evidence index
unsupported = [e for e in (pred.evidence or []) if not span_supports(e, pred.value)]
if unsupported or not pred.evidence:
feedback.append("Missing or unsupported evidence. Cite the exact clause and table row.")
# 3) Fabrication guard
fabricated = gold.expected == [] and pred.status != "not_found"
if fabricated:
feedback.append("Attribute is absent; return 'not_found' instead of inferring a value.")
# 4) Deterministic validators (SHACL + business rules)
report = validate_against_shapes(pred.value, gold.attribute_id)
feedback += [f"Validation: {v.message}" for v in report.violations]
penalty = 0.3 * bool(unsupported) + 0.5 * fabricated + 0.1 * len(report.violations)
return dspy.Prediction(score=max(0.0, value_score - penalty),
feedback="\n".join(feedback) or "Correct and well-evidenced.")
optimizer = dspy.GEPA(
metric=extraction_metric,
auto="light",
reflection_lm=dspy.LM("<strong-reflection-model>", temperature=1.0),
candidate_selection_strategy="pareto",
track_stats=True,
)
optimized = optimizer.compile(program, trainset=tenant_a_train, valset=tenant_a_val)
Design notes:
- Validators are code. The LLM never grades its own evidence or ontology compliance.
- Tenant scoping. Build datasets per tenant. An optimized prompt can carry patterns from its training data, so govern it like a model artifact.
- Stratify by overlay. A prompt that improves revolvers but quietly breaks ABL borrowing-base extraction must fail the gate, even if the average improves.
11.6 Where to start
Start with Agent 1 and Agent 3 on the core schema. They have the most labeled feedback and the smallest blast radius. Expand to Agent 2 and alignment only after the golden set covers every overlay, prompts have survived a FIBO upgrade and a major schema version, and monitoring can attribute changes to a specific prompt version.
12. Evaluation: What to Measure From Day One
| Dimension | Metric | How |
|---|---|---|
| Extraction | Exact and normalized match | vs. golden values, per overlay |
| Evidence faithfulness | % of values with a verifiable span | Deterministic check + sampled audit |
| Fabrication | % of not_found cases where a value was returned |
Golden negatives |
| Schema coverage | % of substantive sections covered | Coverage analysis |
| Gap discovery | Proposal acceptance rate; alias-vs-new accuracy | Approval outcomes |
| Mapping safety | Invented-IRI rate (target: 0) | Resolution against pinned release |
| Validation health | SHACL violations per 100 documents | Validator reports |
| Schema churn | Changes per version by type; % of pages reprocessed | Registry diff |
| Ontology drift | Live mappings affected by a candidate FIBO release | Release diff |
| Calibration | Confidence vs. reviewer outcome | Reliability analysis |
| Human effort | Review minutes per document; % auto-accepted | Workflow telemetry |
Release gate: every release, whether schema or prompt, must show no regression in evidence faithfulness, fabrication or invented-IRI rate, and an improvement on its target metric.
13. Governance and Model Risk
Many articles still cite SR 11-7. On 17 April 2026, the US Federal Reserve, OCC and FDIC issued revised model risk management guidance (SR 26-2 / OCC Bulletin 2026-13), which supersedes it. Two points matter here:
- The revised guidance says generative and agentic AI models are not within its scope, while a bank's broader risk management and governance practices should still apply.
- It excludes deterministic rule-based processes from the definition of a model. Your SHACL shapes and business rules are software controls, not models.
My view: because LLM agents currently sit outside the formal MRM perimeter, architects have to supply the discipline themselves. This architecture does that:
- Inventory and tiering of every agent, prompt version and optimizer run, by downstream materiality.
- Effective challenge: golden sets, metrics and release decisions get independent review.
- Attribute-level lineage from document → clause → value → schema version → prompt version → consumer, consistent with the BCBS 239 emphasis on data lineage.
- Change control per loop: data owners approve schema changes; gates plus humans approve prompt releases; FIBO upgrades need an impact analysis.
- HITL thresholds based on calibrated confidence and materiality.
- Jurisdiction checks: under the EU AI Act as amended by the Digital Omnibus, high-risk obligations for creditworthiness assessment of natural persons now apply from December 2027. Corporate credit-agreement extraction isn't credit scoring in itself, but if outputs feed decisions about individuals, review the classification with counsel.
14. Scaling Across Business Lines: One Platform, Many Domain Packs
Credit agreements are the hardest case, but large banks face the same problem in home lending and wealth management, with very different profiles:
| Home lending | Corporate finance | Wealth management | |
|---|---|---|---|
| Documents | Loan files with 100+ document types | 200–600 page credit agreements, amendments | Account opening, KYC, trusts, investment policy statements |
| Shape | Very high volume, mostly standard forms | Low volume, highly bespoke | Medium volume, mixed |
| Dominant loop | Loop 2: quality at volume | Loop 1: schema evolution | Both |
| Best-fit standard | MISMO, mapped to FIBO | FIBO loan and debt modules | FIBO securities and party modules; ISO 20022 where relevant |
| Hardest part | Sorting documents within a loan file, PII, fair-lending controls | Definitions, cross-references, covenant logic | Ownership structures, KYC/AML evidence |
Don't build three platforms. Build one shared platform with a domain pack per business line:
SHARED PLATFORM (built once)
ingestion + canonical document model · ontology compiler + OKF bundles ·
schema registry · deterministic validation · harness (orchestration, traces,
replay, evaluation, release gates) · feedback router · governance + isolation
DOMAIN PACK (one per business line)
ontology subset + standard mappings · core schema + overlays ·
golden sets per document type · review policy · data owners · prompt versions
Two lessons:
- Use the domain's own standard where one exists. A FIBO-first mortgage schema would reinvent what MISMO already defines. FIBO's job is to be the bridge between domains.
- The bridge is the strategic payoff. Once all three business lines share one party and contract model, you can recognize one client across them: a mortgage borrower who guarantees their company's credit facility and is the grantor of a family trust. That connected client view is worth more than better extraction in any one business line.
15. Reference Implementation on AWS (One Option)
| Capability | AWS option | Vendor-neutral equivalent |
|---|---|---|
| OCR, layout, tables | Amazon Textract; Bedrock Data Automation | Docling, Azure Document Intelligence |
| Agent reasoning | Amazon Bedrock foundation models | Any governed LLM endpoint |
| Governed backbone | AWS Step Functions | Temporal, Airflow |
| Agentic steps | Bedrock AgentCore Runtime + Strands Agents | LangGraph |
| Ontology graph | Amazon Neptune (RDF/SPARQL) with pinned FIBO + extensions | GraphDB, Stardog, Apache Jena |
| Agent knowledge layer | OKF bundles compiled from OWL, in Git, served from S3 with a concept index | Same |
| Validation | pySHACL + OWL reasoner in containers | Same |
| Schema registry | Aurora PostgreSQL + Git for schema artifacts | Postgres + Git |
| Extracted values | Aurora / DynamoDB with lineage columns | Any relational store |
| Memory | Bedrock AgentCore Memory with tenant namespaces | Letta, Zep, custom |
| Optimization | DSPy + GEPA as batch jobs; prompt registry | Same |
| Evidence store | Amazon S3 | Object storage |
| Audit | CloudTrail + append-only audit table | Append-only log |
Run the governed backbone (ingest → classify → bind schema → extract → validate → review → publish) as a deterministic workflow, and keep agentic autonomy inside individual steps. The audit trail stays linear even when the reasoning isn't.
16. A Phased Roadmap
| Phase | Weeks | Scope | Exit criteria |
|---|---|---|---|
| 0: Foundations | 0–4 | Canonical document model, definitions graph, evidence index; pinned FIBO compiled to OKF; registry with core@1.0.0; golden set of 20 agreements across 2–3 variants |
Every value traceable to a bounding box |
| 1: Extract + Validate | 5–10 | Agents 1 and 3 on the core schema; SHACL; review UI; not_found handling |
Baseline accuracy per overlay; fabrication and invented-IRI rates measured |
| 2: Loop 1 | 11–16 | Overlays; coverage analysis; Agent 2 gap discovery; chat proposals; semver + deprecation; targeted reprocessing | A minor schema change completes end-to-end, with audit, in under a day |
| 3: Loop 2 | 17–24 | GEPA on Agents 1 and 3; stratified gates; shadow releases; FIBO upgrade drill | Held-out improvement with zero regression on faithfulness and fabrication |
Key idea: Build Loop 1 before Loop 2. Optimizing prompts against a schema that changes every week means optimizing against a moving target.
17. Anti-Patterns to Avoid
- "Self-evolving" as self-modifying. Agents writing directly to the schema or ontology.
- A v1 schema per document. Thousands of incomparable schemas.
- One monolithic schema instead of a core + overlays.
- Schema explosion. Every one-off clause becomes an attribute.
- Copying FIBO wholesale as your extraction schema.
- Hand-converting FIBO to Markdown, so typed semantics become untyped links.
- LLM-generated IRIs that look official but don't resolve.
- An unpinned ontology that silently picks up a release with removed concepts.
- Breaking changes without deprecation.
- Confidence treated as probability without calibration.
- Human review on every layer, which can't scale and invites rubber-stamping.
- One loop for everything, so nothing can be attributed.
- Averages across variants that hide regressions on rare overlays.
- Tenant isolation by metadata filter only.
- Ten microservices on day one.
18. What Each Architect Should Take Away
| Role | Your three takeaways |
|---|---|
| Enterprise Architect | Make it a shared platform with domain packs · keep two loops with separate approvers · the payoff is a connected client view across business lines |
| Solution Architect | Deterministic backbone with agentic steps inside · route sections instead of scanning pages · build the canonical document model before any agent |
| Data Architect | Core + overlays with semver and deprecation · a lineage stamp on every value · "not found" is a valid state |
| AI / ML Engineer | Metrics that return textual feedback · stratify evaluation by overlay · validators are code, never the LLM |
| Ontology Engineer | Pin FIBO; compile OKF from OWL · explicit mapping states · LLMs adjudicate, humans approve |
| Platform Engineer | Tenant isolation by namespace or silo · version everything (parser, prompt, model, schema, ontology, bundle) · targeted reprocessing as a first-class job |
Conclusion
Financial document AI is moving from answering questions about documents to maintaining governed knowledge derived from them. For long, heterogeneous agreements, that takes three things RAG doesn't provide:
- An ontology that gives data stable meaning: FIBO, pinned, mapped, never invented, and delivered to agents as provenance-carrying OKF bundles.
- A schema registry that turns what we capture into a versioned, layered contract that grows with the portfolio without breaking consumers.
- Two learning loops, kept apart: one that evolves what we capture through approved proposals, and one that improves how well we capture it through gated, reflective optimization.
The teams that do this well won't be the ones with the longest context windows. They'll be the ones whose schemas can change quickly because every change is proposed, aligned, validated, approved and traceable.
Where to start on Monday morning: pick one agreement family. Pin one FIBO release. Build a golden set of 20 documents across two or three variants. Ship the canonical document model, then Loop 1, then Loop 2. Measure evidence faithfulness, fabrication rate and invented-IRI rate from day one.
If you're building something similar, I'd like to compare notes. Reply here or connect on LinkedIn. Next in the DataOps Labs series: evidence-first ingestion for 450-page agreements, covering definitions graphs, cross-reference resolution and a parser bake-off.
DataOps Labs · Education and Knowledge Democratised. If this was useful, share it with an architect who's fighting a 450-page PDF this week.
About the author
- I'm Ayyanar Jeyakrishnan ; aka AJ. With over 21 years in IT, I'm a passionate Multi-Cloud Architect specialising in crafting scalable and efficient cloud solutions. I've successfully designed and implemented multi-cloud architectures for diverse organisations, harnessing AWS, Azure, and GCP. My track record includes delivering Machine Learning and Data Platform projects with a focus on high availability, security, and scalability. I'm a proponent of DevOps and MLOps methodologies, accelerating development and deployment. I actively engage with the tech community, sharing knowledge in sessions, conferences, and mentoring programs. Constantly learning and pursuing certifications, I provide cutting-edge solutions to drive success in the evolving cloud and AI/ML landscape.
Comments
Join the discussionNo comments yet. Be the first to comment.





