Skip to main content

Self-Evolving Financial Ontologies: How Multi-Agent AI Learns to Discover Enterprise Schemas from Financial Documents

Updated 40 min readMarkdown
Self-Evolving Financial Ontologies: How Multi-Agent AI Learns to Discover Enterprise Schemas from Financial Documents

DataOps Labs · Education and Knowledge Democratised

Disclaimer: These are my personal views. This is a reference architecture for discussion, not a description of any employer's or vendor's production system.

TL;DR

  • A 450-page credit agreement isn't a pile of paragraphs. It's a web of defined terms, cross-references, schedules and conditions. RAG can answer questions about it, but it can't turn it into governed, comparable data.
  • Every financial document is a little different, so you can't fix a schema in advance. You also can't let an LLM invent a new one for every document. The schema has to evolve, under control.
  • The architecture has three building blocks:
    1. An ontology. FIBO, pinned to one release, gives every concept a stable meaning. Agents read it through Open Knowledge Format (OKF) bundles.
    2. A schema registry. It versions what you capture, the way a package registry versions code.
    3. Two learning loops that are never merged. Loop 1 changes what you capture. Loop 2 improves how well you capture it.
  • "Self-evolving" must never mean "self-modifying". Agents propose. Validators check. Humans approve. Everything is versioned.

How to Read This Article

The article is long because the problem has several layers. Different architects will care about different parts, so here's a map:

If you are a… Start with You'll get
Enterprise Architect Sections 1, 2, 5, 13, 14 Why this needs to be a platform capability, how the governance works, how it scales across business lines
Solution Architect Sections 6–9, 15, 16 The end-to-end flow, the agent responsibilities, a reference implementation, a delivery roadmap
Data Architect Sections 3, 4, 10 The three-layer semantic model, schema versioning, lineage, memory design
AI / ML Engineer Sections 7, 11, 12 Agent contracts, GEPA optimization, the evaluation framework
Ontology Engineer Sections 3, 8 FIBO alignment, the OWL → OKF compilation approach, SHACL validation
Platform Engineer Sections 6, 14, 15 Ingestion, the harness, multi-tenancy, deployment

A small glossary before we start:

  • Ontology: a formal dictionary of concepts and how they relate. For example, "a Guarantor is a kind of Obligor".
  • FIBO: the Financial Industry Business Ontology, the industry's shared ontology for finance.
  • OKF: Open Knowledge Format, an open format from Google that stores knowledge as Markdown files with YAML metadata, including who wrote each concept, who verified it, and when.
  • Schema: the list of fields you extract, with their types and rules. For example, max_total_leverage_ratio: ratio, required, evidence needed.
  • Schema registry: the versioned, authoritative catalog of schemas.
  • SHACL: a W3C standard for validating graph data, e.g. "every facility must have exactly one currency".
  • GEPA: a method that improves agent prompts by reading what went wrong in past runs.
  • HITL: human-in-the-loop review.

1. A Monday Morning Problem

Picture a credit portfolio team on a Monday morning. Over the weekend, 40 new and amended credit agreements arrived. Each one is between 200 and 600 pages. The head of credit risk asks three questions:

  1. "What is the maximum leverage ratio in each agreement, with its step-downs and test dates?"
  2. "Which agreements allow incremental debt without lender consent, and up to how much?"
  3. "From this quarter onwards, I also want to track sustainability-linked margin adjustments."

The first question looks like a lookup. The second needs reasoning across clauses. The third isn't a question at all: it's a change to what the organization captures.

Most GenAI document programs are built for the first kind of question. This article is about building for all three.


2. Why RAG Is Not Enough

The standard pipeline looks like this:

PDF → text → fixed-size chunks → embeddings → vector database → top-k zretrieval → LLM answer

It's a good pipeline for question answering. It breaks in four places when the goal is governed data.

Figure 1 — Left: RAG splits the agreement into chunks and loses the links between them. Right: a structure-aware model keeps definitions, cross-references and schedules connected, so every value can be traced to its evidence.

2.1 Definitions live far from where they're used

In a typical credit agreement, the definitions article can run to dozens of pages. "Consolidated EBITDA" might be defined on page 19, adjusted by a schedule on page 590, and tested in a covenant on page 233. A chunk containing "the Consolidated Total Leverage Ratio shall not exceed 4.50:1.00" means little without the chain of definitions behind it. Similarity search doesn't reliably bring that chain back.

2.2 Cross-references are the structure

Phrases like "subject to Section 2.14(d)", "as set forth in Schedule 2.01" and "except as permitted under Section 7.02(k)" aren't noise. They are the meaning of the document. Fixed-size chunks cut exactly at those joins.

2.3 An answer isn't a record

RAG returns a paragraph. A portfolio needs the same record shape for every agreement, such as:

{
  "leverage_covenant": {
    "max_ratio_schedule": [
      {"test_period_end": "2027-03-31", "max_ratio": 4.50},
      {"test_period_end": "2028-03-31", "max_ratio": 4.00}
    ],
    "test_frequency": "quarterly",
    "equity_cure": true
  }
}

2.4 Nothing learns and nothing is versioned

When an analyst corrects a RAG answer, the correction disappears with the chat session. When the business asks for a new data point, there's no schema to extend, only a prompt to edit. And without a schema, there's nothing to audit.

Key idea: The hard problem isn't extraction. It's knowledge engineering: deciding what the information model is, keeping it consistent across documents that are all different, and changing it safely as needs grow.


3. The Heterogeneity Problem: Same Family, Different Documents

All the documents in our scenario are financial agreements, and they share a skeleton: parties, definitions, facilities, pricing, repayment, covenants, collateral, events of default, schedules. But the details vary a lot:

Variant What's special about it
Syndicated revolving facility Per-lender commitments, letter-of-credit sublimits, swingline loans, commitment fees
Term Loan B (leveraged) Often covenant-lite; incremental capacity, MFN protection, excess cash flow sweep
Asset-based lending (ABL) Borrowing base, eligibility criteria, advance rates, reserves
Sustainability-linked loan KPIs, sustainability performance targets, margin ratchets, verification
Project finance Cash waterfalls, reserve accounts, completion tests
Amendment and restatement Changes an existing agreement, so you must know what it amends

On top of that comes the long tail: bespoke definitions, jurisdiction-specific clauses, deal-specific carve-outs.

That leaves three bad options and one good one:

  1. ❌ One giant fixed schema: hundreds of mostly empty fields, still incomplete, slow to change.
  2. ❌ An LLM-generated schema per document: nothing is comparable and field names drift.
  3. ❌ No schema, just RAG: see Section 2.
  4. ✅ A governed, layered, versioned schema that evolves: a stable core, variant overlays, a controlled path for new concepts, and one registry that says what "the schema" is at any point in time.

The rest of the article builds option 4.


4. Building Block 1: An Ontology as the Semantic Anchor

4.1 Why an ontology at all?

Take six words from the parties section of a credit agreement: Borrower, Obligor, Loan Party, Guarantor, Administrative Agent, Lender.

A naive extractor creates six independent fields. But these are roles that parties play in a contract, and they relate to each other. A Guarantor is an Obligor but not a Borrower. "Loan Parties" is usually a defined term that groups the Borrower and the Guarantors. An ontology lets you model this once, correctly and reuse it across every agreement.

4.2 Why FIBO

The Financial Industry Business Ontology (FIBO) is the industry's shared, formal model of financial instruments, parties and contracts. Some facts architects should know:

  • Stewardship: FIBO is developed by the EDM Council, now EDM Association following its combination with the Object Management Group (OMG). The acquisition of OMG's assets completed on 1 October 2025.
  • Format and license: FIBO is published as OWL 2 ontologies, developed openly on GitHub under the MIT license, and browsable in the FIBO Viewer at spec.edmcouncil.org.
  • Cadence: releases ship quarterly, and they're not cosmetic. Recent releases removed long-deprecated concepts, and the release notes warn of more removals to come.

Some FIBO concepts that matter for credit agreements:

What you need FIBO concept or module
The agreement CreditAgreement: https://spec.edmcouncil.org/fibo/ontology/FBC/DebtAndEquities/Debt/CreditAgreement
Secured facilities CollateralizedLoan: https://spec.edmcouncil.org/fibo/ontology/LOAN/LoansGeneral/Loans/CollateralizedLoan
Events of default, covenant breaches Credit Events ontology (FBC / Debt and Equities)
Prepayment, default proceedings Loan Events ontology (maturity: provisional)
Letters of credit, guarantees Guaranty ontology (FBC / Debt and Equities)

FIBO's CreditAgreement already says what we need: a contract in which a debtor receives something of value and agrees to repay a creditor, with restrictions such as has contract party some creditor and has contract party some debtor. Your schema should inherit that meaning rather than reinvent it.

4.3 The three-layer model

Don't copy FIBO as your extraction schema. FIBO is a conceptual model. It doesn't say "read the applicable margin from the pricing grid in Schedule 1.01(a)". Keep three layers with explicit, versioned mappings between them:

Layer What it holds Owner Change rate
1. Industry ontology FIBO, pinned release EDM Association upstream; you decide when to upgrade Quarterly upstream
2. Enterprise schema Your concepts and attributes, including extensions FIBO doesn't have Business data owner When the business needs new data
3. Extraction profiles How to find each attribute in a given document type Platform team + domain experts Whenever quality or coverage improves

4.4 Delivering FIBO to agents: OKF as the knowledge layer

Agents shouldn't parse OWL or load thousands of FIBO classes into a prompt. They need knowledge they can read, with provenance attached. Google's Open Knowledge Format (OKF) v0.2 fits this well. It represents knowledge as a folder of Markdown "concept" files with YAML frontmatter, and v0.2 adds trust and lifecycle signals:

OKF v0.2 field How this architecture uses it
type, title, resource The kind of concept and its canonical IRI
sources What the concept came from: a FIBO release, a clause, a customer decision
generated: {by, at} Who wrote it: a compiler, an agent version or a person
verified: [{by, at}] Confirmation events. Trust tiers come from these: unverified, machine-confirmed, human-reviewed (human: actor). This is your human-in-the-loop record.
status draft → stable → deprecated
stale_after When the concept must be reviewed again, e.g. at the next FIBO release
Attested Computation An approved way to compute a value (leverage ratio, borrowing base). Agents supply parameters but never write the formula.
log.md A dated change history for the bundle

One critical caveat. In OKF v0.2, links between concepts are untyped. A link says a relationship exists, and the prose says what kind. FIBO's value is in its typed semantics: subclasses, restrictions, property domains. If you hand-convert FIBO into plain OKF files, those semantics become prose that agents can read but no validator can enforce.

Key idea: Compile OKF from OWL, never the reverse. OWL stays the source of truth for meaning. OKF is the agent-friendly view generated from it.

Figure 2: The three-layer semantic model — FIBO OWL is compiled into OKF bundles that agents read, while the enterprise schema and extraction profiles map to it with explicit mapping states

Figure 2 — FIBO (pinned) and your enterprise extensions live in OWL/SHACL. A deterministic compiler produces OKF bundles per domain. Agents read the bundles, and validators enforce the OWL/SHACL source. Mappings between layers carry an explicit state: exact, close, extension, unmapped or rejected.

The compiler writes typed relationships into custom frontmatter keys. OKF requires consumers to preserve unknown keys, so this stays conformant:

---
type: Ontology Class
title: Credit Agreement
resource: https://spec.edmcouncil.org/fibo/ontology/FBC/DebtAndEquities/Debt/CreditAgreement
fibo_release: "2026Q3"
fibo_restrictions:
  - "hasContractParty some Creditor"
  - "hasContractParty some Debtor"
status: stable
generated: { by: fibo-okf-compiler/1.0, at: "2026-10-01T00:00:00+00:00" }
verified:
  - { by: process:owl-consistency-check, at: "2026-10-01T02:00:00+00:00" }
  - { by: human:ontology-steward-07, at: "2026-10-03T04:00:00+00:00" }
stale_after: "2027-01-15T00:00:00+00:00"
---

A credit agreement is a contract in which a debtor receives something of value
and agrees to repay a creditor, generally with interest…

Two practical notes:

  • Use full date-times with a UTC offset in stale_after and similar keys. A spec revision in August 2026 made date-only values non-conforming, even though the version number stayed at 0.2.
  • OKF's version number is the format's version, not your content's. Version your bundles yourself (git tags + semver), and pin the OKF spec revision you build against, just as you pin FIBO.

4.5 Three rules I never break

  1. Never invent an official FIBO IRI. Every IRI must resolve in the pinned release. Research on ontology grounding (SPIRES/OntoGPT) shows that grounding by lookup is far more reliable than letting an LLM generate identifiers. In those tests, raw LLMs sometimes returned identifiers from the wrong ontology.
  2. Unmappable attributes stay enterprise extensions. "Relationship Manager Review Status" has no FIBO equivalent, and that's fine. Mark it extension.
  3. Pin the release and plan for drift. J.P. Morgan AI Research ("Ontology drift is a challenge for explainable data governance", arXiv 2108.05401) found that up to a third of FIBO could change in a quarter. A FIBO upgrade is a planned compatibility exercise, not a silent runtime change.

5. Building Block 2: A Schema Registry That Can Evolve

If you remember one component from this article, make it this one. The schema registry is the single source of truth for what we capture, and it's what most RAG programs are missing.

A useful analogy for platform engineers: treat schemas the way you treat software packages. They have versions, dependencies, release notes, deprecation policies and consumers who need to know when something breaks.

5.1 Core + overlays

Instead of one monolithic schema, use a core plus variant overlays, and resolve them for each document:

Schema module Example attributes
credit_agreement.core Parties and roles, facilities, pricing, maturity, events of default, governing law
overlay.revolver Per-lender commitments, LC sublimit, swingline, commitment fee
overlay.term_loan_b Incremental capacity, MFN, ECF sweep, springing covenant
overlay.abl Borrowing base, advance rates, eligibility, reserves
overlay.sustainability_linked KPIs, targets, margin ratchet, verification
overlay.amendment Amended agreement reference, changed provisions

One agreement can activate several overlays. Revolver + term loan + sustainability-linked pricing is common. The registry resolves an effective schema for each document:

{
  "document_id": "doc-ca-0042",
  "effective_schema": {
    "core": "credit_agreement.core@2.3.0",
    "overlays": ["overlay.revolver@1.4.0", "overlay.sustainability_linked@1.1.0"],
    "fibo_release": "2026Q3"
  },
  "overlay_evidence": [
    {"overlay": "overlay.revolver", "section": "2.01(b)", "page": 41},
    {"overlay": "overlay.sustainability_linked", "section": "2.12", "page": 77}
  ]
}

You get comparability (every agreement shares the core), specificity (overlays carry the variant detail) and governability (each module versions on its own).

5.2 The attribute contract

Every attribute follows the same contract, independent of how any LLM happens to phrase things:

{
  "attribute_id": "attr-fin-cov-max-leverage",
  "canonical_name": "max_total_leverage_ratio",
  "concept": "FinancialCovenant.LeverageCovenant",
  "business_definition": "Maximum permitted ratio of Consolidated Total Debt to Consolidated EBITDA, as defined in the agreement",
  "aliases": ["Total Leverage Ratio", "Consolidated Leverage Ratio"],
  "data_type": "ratio_schedule",
  "cardinality": "0..* (one per test period)",
  "depends_on_definitions": ["Consolidated Total Debt", "Consolidated EBITDA"],
  "semantic_mapping": {
    "ontology": "FIBO", "release": "2026Q3",
    "target_iri": null, "mapping_status": "close",
    "rationale": "Related to FIBO credit-event concepts; no exact property for the threshold"
  },
  "validation_rules": ["ratio > 0", "test periods ordered", "step-downs non-increasing unless stated"],
  "evidence_requirement": "page + clause + bounding box",
  "lifecycle_status": "stable",
  "introduced_in": "credit_agreement.core@2.0.0"
}

Two fields do most of the work with heterogeneous documents. aliases absorbs terminology differences between law firms. depends_on_definitions tells extraction to resolve the definition chain first.

Watch out: aliases that only look equivalent. "Total Net Leverage Ratio" and "Senior Secured Leverage Ratio" are different metrics. Aliases tell the system where to look. Equivalence is a decision a human approves, never something string similarity decides.

5.3 Semantic versioning for schemas

Figure 3 — How the schema grows safely. Overlays absorb document variety. Uncovered content becomes candidate proposals only when it recurs. Breaking changes go through a deprecation window: v2.3 → v2.4 (add new attributes and deprecate the old one) → v3.0 (remove it).

Change Bump Examples Reprocessing Consumer impact
Patch x.y.Z Better definition, new alias, refined instruction None required None
Minor x.Y.0 New optional attribute, new overlay, new enum value Targeted: only relevant sections Additive
Major X.0.0 Split/merge/rename, type or cardinality change, optional → required Targeted or full + migration mapping Breaking; notify and migrate

Breaking changes always get a deprecation window. Say the business wants to split termination_condition into trigger, notice period and approval authority:

  1. v2.4.0 (minor): add the three new attributes, mark the old one deprecated but keep populating it, and publish a migration mapping.
  2. Consumers migrate within the agreed window.
  3. v3.0.0 (major): remove the old attribute.

No dashboard breaks overnight.

5.4 Every value knows where it came from

Each extracted value carries a lineage stamp:

{
  "document_id": "doc-ca-0042",
  "document_sha256": "9f2c…",
  "attribute_id": "attr-fin-cov-max-leverage",
  "schema_version": "credit_agreement.core@2.3.0",
  "value": [{"test_period_end": "2027-03-31", "max_ratio": 4.50}],
  "status": "extracted",
  "evidence": [
    {"page": 233, "section": "7.11(a)", "bbox": [72, 188, 540, 402]},
    {"page": 19, "section": "1.01", "term": "Consolidated Total Debt"}
  ],
  "lineage": {
    "parser_version": "layout-parser@3.2.1",
    "agent_prompt_version": "value-extractor@1.7.0",
    "model_id": "<governed-llm-endpoint>",
    "fibo_release": "2026Q3",
    "okf_bundle": "corporate-credit@2026.11.0"
  },
  "review": {"status": "pending", "reviewer": null}
}

This answers the auditor's question ("why does the system say 4.50, and would it still say that today?") and the engineer's question ("which records do I reprocess after this change?").

Key idea: "Not found" is a valid answer. Covenant-lite deals often have no maintenance leverage covenant. Record status: "not_found" with the sections searched. Never let a model fill in a required field just because the schema expects one.


6. The End-to-End Flow

Here's the whole pipeline in one view before we go into each part.

In words:

Step Component What happens Output
0 Ingestion Parse layout, tables and hierarchy; resolve definitions and cross-references Canonical document model + evidence index
1 Agent 1: Classify Identify document family, section types and overlays, using OKF concept lookups Classification profile
2 Agent 2: Bind schema Resolve the registered effective schema; propose changes only for real, recurring gaps Effective schema + draft proposals
3 Agent 3: Extract Extract values from routed sections with evidence; call attested computations for maths Evidence-backed values
4 Validation gates OWL, SHACL, JSON Schema, business rules, IRI-exists checks Pass / fail reports
5 Risk-based HITL Humans review exceptions, material items and samples verified events
6 Registry + lineage store Publish versions; store values with lineage Auditable data
7 Chat on the harness Answer questions; route feedback to the right loop Answers or proposals

The rest of this section covers ingestion. Section 7 covers the agents.

6.1 Never send 450 pages to an LLM

Long context windows are impressive, but governed extraction needs every claim to point to a page, section, clause and bounding box, and that pointer has to exist before any agent runs.

The ingestion stage produces a canonical document model:

  • Stable, human-readable node IDs such as doc-ca-0042/art7/s7.11/a/para2.
  • Page, bounding box and parser confidence for each node.
  • Parent/child links: article → section → clause → paragraph.
  • Outgoing references to definitions, sections and schedules.

6.2 Build the definitions graph first

For credit agreements, the definitions graph is the most valuable structure you can build. Parse the definitions article into a term → definition map, then link every capitalized defined term elsewhere in the document back to it. When the extractor reads Section 7.11, it gets the covenant clause plus the resolved definitions of every term in it, and nothing else.

6.3 Route sections; don't scan pages

With 450 pages and perhaps 120 attributes, extracting every attribute from the whole document is wasteful and inaccurate. Instead:

  1. Agent 1 labels each section with a section type (financial_covenants, pricing, events_of_default…).
  2. The registry keeps an attribute → section-type index.
  3. Agent 3 sees only the routed sections plus their resolved definitions.

This is also what makes targeted reprocessing cheap. A new events_of_default attribute touches only those sections across the portfolio.

6.4 Tooling

Benchmark on your documents. Vendor claims don't tell you how a parser handles a scanned 1990s amendment.

  • Amazon Textract: OCR, forms, tables.
  • Amazon Bedrock Data Automation: blueprint extraction with confidence scores and bounding boxes, plus blueprint optimization from a few labeled documents.
  • Docling (IBM Research, MIT license): layout analysis, table structure and a unified document model. A good fit for self-hosted or air-gapped environments.
  • Azure Document Intelligence, LlamaParse and Unstructured fill the same slot.

7. The Three Agents

Start with logical agents inside one governed workflow, not ten microservices on day one. Split deployment units only when scale, security or team ownership requires it.

Key idea: No agent is a source of truth. Agents propose; the schema registry and the ontology service decide.

7.1 Agent 1: Classification

Agent 1 classifies at three levels: document (credit agreement, amendment, guaranty, intercreditor), section (covenants, events of default…) and overlay (revolver, TLB, ABL, SLL…). It looks up concepts in the OKF bundle through an index or search tool and may only cite concepts that exist.

{
  "document_id": "doc-ca-0042",
  "node_id": "doc-ca-0042/art7/s7.11",
  "level": "section",
  "label": "financial_covenants",
  "okf_concept": "/corporate-credit/section-types/financial-covenants.md",
  "confidence": 0.82,
  "confidence_type": "model_self_reported",
  "evidence": [{"page": 233, "section": "7.11"}],
  "review_status": "auto_accepted"
}

Note confidence_type. A model's self-reported confidence isn't a probability until you've calibrated it against reviewer outcomes. Base review thresholds on calibrated scores.

7.2 Agent 2: Schema binding and gap discovery

This is the agent where most teams go wrong. Agent 2 should not write a "version 1 schema" for every document. Do that and you'll end up with thousands of incomparable schemas. Agent 2 has two jobs:

  1. Bind: resolve the registered stable schema for this document family (core + overlays).
  2. Discover gaps: find substantive content the schema doesn't explain, and propose changes.

A brand-new v1 schema happens once per document family, not once per document.

Gap discovery uses a measure I call schema coverage. After extraction, every section is one of:

  • Covered: at least one attribute drew evidence from it.
  • Boilerplate: non-substantive (notices, counterparts, severability).
  • Uncovered: substantive, but explained by nothing in the schema. These are the candidate gaps.

Two guardrails prevent schema explosion:

  1. Deduplicate against the registry first. Most "new" concepts are aliases of existing ones. Agent 2 must show novelty through definitions and context, not word matching. When it's ambiguous, it proposes an alias for a human to decide.
  2. A recurrence threshold. A concept seen in one deal is long tail. It goes into a structured document-specific notes record: searchable, but not part of the schema. Only concepts that recur across N documents, or that the business explicitly requests, become schema proposals.

Proposals are written as OKF Schema Attribute concepts with status: draft. They become stable only after validation and data-owner approval.

7.3 Agent 3: Extraction

Agent 3 reads the routed sections, the resolved definitions and the effective schema, and returns evidence-backed values. Three rules:

  • Evidence or nothing. Every value needs a span that a deterministic checker can verify.
  • "Not found" is allowed. It's often the correct answer.
  • Never do the maths. For derived values (commitment totals, ratios), Agent 3 gathers the inputs and calls an attested computation. The LLM supplies parameters; approved code computes.

7.4 The Ontology Alignment step

Between Agent 2 and approval, every proposal is aligned to the pinned ontology:

Mapping state Meaning Example
Exact Same meaning as a concept in the pinned release The agreement ↔ CreditAgreement
Close Related but narrower or broader; difference documented Leverage threshold vs. FIBO covenant-breach concepts
Extension Valid enterprise concept with no FIBO equivalent relationship_manager_review_status
Unmapped Not yet analyzed New request
Rejected Mapping failed review Words matched, meaning didn't

What the research says about LLMs here: they work best as judges of uncertain candidates, not as unsupervised matchers. In OAEI 2025, a classical matcher that consulted an LLM only on high-uncertainty mappings ranked second overall in the Bio-ML track, while in the Conference track an LLM-agent matcher didn't beat the classical baseline. So the pattern is: deterministic candidate generation → LLM adjudication on ambiguous cases → human approval.

7.5 Human review by risk, not by layer

A human checkpoint after every agent works for 50 agreements. It doesn't work for millions of loan files, and reviewers who see too much start rubber-stamping. Set review intensity by impact and uncertainty:

Layer Volume Review policy
Schema changes Low Always reviewed and approved by the data owner
Ontology mappings Low–medium Every close and extension; exact is machine-verified
Classification High By exception: low calibrated confidence, conflicting overlays, unknown family
Extraction Very high By exception: validation failure, material attributes, low calibrated confidence, plus random sampling

Every review is written as a verified event with a human: actor. Your HITL record and your knowledge layer are the same thing.


8. Deterministic Validation: The Gate Every Handoff Passes

LLMs propose; deterministic systems validate. These four layers complement each other. None replaces another.

Layer What it checks Worldview
OWL 2 reasoning Ontology + extensions are logically consistent Open world: a missing fact is unknown
SHACL Extracted data has the required shape Closed world: a missing required property is a violation
JSON Schema API payloads between services Structural
Business rules Maturity after closing date, ISO 4217 currency, non-increasing step-downs Domain logic

A common mistake is expecting an OWL reasoner to flag "every facility must have a currency". Under the open-world assumption it won't. That's SHACL's job:

@prefix sh:  <http://www.w3.org/ns/shacl#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix ex:  <https://example.org/credit#> .

ex:LoanFacilityShape a sh:NodeShape ;
  sh:targetClass ex:LoanFacility ;
  sh:property [ sh:path ex:commitmentAmount ; sh:minCount 1 ; sh:maxCount 1 ] ;
  sh:property [ sh:path ex:currency ; sh:minCount 1 ; sh:pattern "^[A-Z]{3}$" ] ;
  sh:property [ sh:path ex:maturityDate ; sh:maxCount 1 ; sh:datatype xsd:date ] .

One more check deserves its own line: "Does this FIBO IRI exist in the pinned release?" It's a simple lookup, and it eliminates an entire class of hallucination.


9. Building Block 3: Two Learning Loops, Never Merged

This is the core idea of the architecture.

Figure 5: Two governed learning loops — Loop 1 evolves what we capture (schema), Loop 2 improves how well we capture it (agent prompts via GEPA), joined only by informational links

Figure 5 — Loop 1 (left) turns business requests and coverage gaps into approved schema versions. Loop 2 (right) turns reviewer corrections into better agent instructions through GEPA, offline evaluation and release gates. Dashed lines show the loops informing each other; they never write to each other's artifacts.

Loop 1: Schema Evolution Loop 2: Extraction Quality
Question What should we capture? How well do we capture it?
Triggers Business requests; coverage gaps; recurring uncovered content Reviewer corrections; golden-set regressions; validation failures
Changes Enterprise schema, overlays, mappings Agent instructions, few-shot policy
Never changes Agent prompts Schema, ontology, validation rules
Approver Business data owner (+ ontology steward) AI engineering + independent reviewer
Artifact credit_agreement.core@2.4.0 value-extractor@1.8.0
Rollback Previous schema + migration mapping Previous prompt release

Why keep them apart? Attribution and blast radius. If one correction could change both the schema and the prompts, you'd never know which change caused the next regression, and you couldn't roll one back without the other.


10. Loop 1 in Practice: Conversational Schema Evolution

Business users shouldn't need to know OWL, SHACL or JSON Schema. They should be able to talk to the system the way they'd talk to a good credit analyst.

10.1 Two kinds of request

  • Information requests run against the current schema and data: "What's the leverage covenant in the Acme agreement?"
  • Schema evolution requests start Loop 1: "From now on, track sustainability-linked margin adjustments."

Asking a question about an attribute must never change the schema by itself.

10.2 What the conversation looks like

Analyst: We need to track early termination properly: what triggers it, the notice required and who has to approve it.

Assistant: Your schema (credit_agreement.core@2.3.0) has one attribute, termination_condition, that combines all three. I propose:

Proposed attribute Type Mapping
termination_trigger List of structured conditions Close (FIBO credit events), pending validation
termination_notice_period Duration + business-day convention Extension
termination_approval_authority List of typed approvals Extension

I'd deprecate termination_condition rather than delete it, so existing reports keep working. That makes this v2.4.0 (minor). Reprocessing scope: sections typed termination and prepayment across 312 agreements, about 4% of total pages. I've sent this to the data owner for approval.

Data owner: Approved. Keep board approval and regulatory approval separate.

Assistant: Done. approval_authority.type is an enum (board, regulatory, lender_majority, other), each with evidence. I've recorded the decision so future proposals follow it. v2.4.0 is published, and targeted reprocessing has started.

The conversation isn't what matters. What matters is the structured, versioned, validated, approved change it produces.

10.3 Route every piece of feedback to the right place

The chat is a client of the harness, not the decision-maker. Feedback comes in four kinds:

Feedback Example Goes to
Value correction "The margin is 2.25%, not 2.50%" Golden set → Loop 2
Schema change "Also track the excess cash flow sweep" Draft attribute → data-owner approval → Loop 1
Mapping error "That party is a sponsor, not a guarantor" Ontology steward
Classification error "This is an amendment, not a new agreement" Classifier golden set → Loop 2

Also separate who can propose from who can approve. Any analyst can propose; only the domain data owner approves.

10.4 Memory: structured, not stuffed

Pasting old chats into a prompt isn't memory. Use three kinds:

Memory Contents Authority
Schema memory Approved schemas, mappings, versions Authoritative (it is the registry)
Interaction memory Decisions, preferred terminology, rejected proposals Tenant-scoped context
Learning memory Corrections, golden examples, curated traces Tenant-scoped; feeds Loop 2 only

Multi-tenant isolation is non-negotiable. One client's confidential agreements must never become another client's training signal. On AWS, Amazon Bedrock AgentCore Memory supports namespaces with IAM conditions, and AWS's own guidance warns against relying on a tenant_id metadata filter alone. For the strictest needs, give each tenant its own memory store.


11. Loop 2 in Practice: GEPA for Extraction Quality

11.1 What GEPA is

GEPA (Genetic-Pareto) is a reflective prompt optimizer (Agrawal et al., arXiv 2507.19457, ICLR 2026 Oral). Think of it as an experienced reviewer who reads the logs of failed runs, works out why they failed, and rewrites the instructions. It runs the system, collects execution traces, has a reflection model read them in plain language, and proposes better instructions. Instead of keeping one "best" prompt, it keeps a Pareto frontier of candidates that each do best on different examples.

In the paper's final version, GEPA outperforms GRPO by about 6% on average (up to 20%) with up to 35x fewer rollouts, and outperforms MIPROv2 by over 10%. Two cautions: gains vary by task, and none of the benchmarks were financial documents. Treat the paper as evidence that the mechanism works, then prove it on your own data.

11.2 Why GEPA fits this architecture

GEPA learns from textual feedback, and this platform produces a lot of it:

  • Reviewer notes: "This is the springing covenant on the revolver, not a maintenance covenant on the term loan."
  • SHACL reports: "LoanFacility lacks currency."
  • Evidence failures: "Value 4.50 has no supporting span in the cited clause."
  • IRI checks: "Proposed IRI does not resolve in release 2026Q3."

A single accuracy score throws all of this away. GEPA can use it.

11.3 Boundaries

GEPA may change GEPA may never change
Agent 1/2/3 instructions FIBO, the pinned release or OKF ontology concepts
Gap-discovery guidance (alias vs. new) The authoritative schema
Few-shot selection policy SHACL shapes, business rules, approval workflows
Output formatting guidance Tenant isolation boundaries

11.4 From correction to golden example

A reviewer sees that Agent 3 extracted one leverage ratio (4.50), but the agreement has a step-down schedule and a temporary holiday after acquisitions. The reviewer fixes the value and writes a note, and the platform turns it into a golden example:

{
  "example_id": "gold-lev-0193",
  "tenant": "tenant-a",
  "section_path": "doc-ca-0042/art7/s7.11",
  "attribute_id": "attr-fin-cov-max-leverage",
  "expected": [
    {"test_period_end": "2027-03-31", "max_ratio": 4.50},
    {"test_period_end": "2027-09-30", "max_ratio": 4.25},
    {"test_period_end": "2028-03-31", "max_ratio": 4.00}
  ],
  "reviewer_note": "Step-down table in 7.11(a); acquisition holiday in 7.11(c) raises the ratio by 0.50x for four quarters — capture as a conditional adjustment, not a separate threshold.",
  "schema_version": "credit_agreement.core@2.3.0"
}

The reviewer_note is what GEPA's reflection step learns from.

11.5 Implementation sketch (DSPy)

import dspy

class ExtractAttribute(dspy.Signature):
    """Extract one schema attribute from a routed credit-agreement section.
    Use resolved definitions. Cite clause IDs for every value. Return status
    'not_found' if absent. Never compute ratios yourself; return inputs instead."""
    section_text: str = dspy.InputField()
    resolved_definitions: str = dspy.InputField()
    attribute_contract: str = dspy.InputField(desc="registry contract, JSON")
    value: list[dict] = dspy.OutputField()
    evidence: list[dict] = dspy.OutputField()
    status: str = dspy.OutputField(desc="extracted | derived | not_found")

program = dspy.ChainOfThought(ExtractAttribute)

def extraction_metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
    feedback = []

    # 1) Correctness vs. reviewer-approved value
    value_score = compare_normalized(pred.value, gold.expected)        # 0..1
    if value_score < 1.0:
        feedback.append(f"Value mismatch. Reviewer note: {gold.reviewer_note}")

    # 2) Evidence faithfulness — deterministic check against the evidence index
    unsupported = [e for e in (pred.evidence or []) if not span_supports(e, pred.value)]
    if unsupported or not pred.evidence:
        feedback.append("Missing or unsupported evidence. Cite the exact clause and table row.")

    # 3) Fabrication guard
    fabricated = gold.expected == [] and pred.status != "not_found"
    if fabricated:
        feedback.append("Attribute is absent; return 'not_found' instead of inferring a value.")

    # 4) Deterministic validators (SHACL + business rules)
    report = validate_against_shapes(pred.value, gold.attribute_id)
    feedback += [f"Validation: {v.message}" for v in report.violations]

    penalty = 0.3 * bool(unsupported) + 0.5 * fabricated + 0.1 * len(report.violations)
    return dspy.Prediction(score=max(0.0, value_score - penalty),
                           feedback="\n".join(feedback) or "Correct and well-evidenced.")

optimizer = dspy.GEPA(
    metric=extraction_metric,
    auto="light",
    reflection_lm=dspy.LM("<strong-reflection-model>", temperature=1.0),
    candidate_selection_strategy="pareto",
    track_stats=True,
)
optimized = optimizer.compile(program, trainset=tenant_a_train, valset=tenant_a_val)

Design notes:

  • Validators are code. The LLM never grades its own evidence or ontology compliance.
  • Tenant scoping. Build datasets per tenant. An optimized prompt can carry patterns from its training data, so govern it like a model artifact.
  • Stratify by overlay. A prompt that improves revolvers but quietly breaks ABL borrowing-base extraction must fail the gate, even if the average improves.

11.6 Where to start

Start with Agent 1 and Agent 3 on the core schema. They have the most labeled feedback and the smallest blast radius. Expand to Agent 2 and alignment only after the golden set covers every overlay, prompts have survived a FIBO upgrade and a major schema version, and monitoring can attribute changes to a specific prompt version.


12. Evaluation: What to Measure From Day One

Dimension Metric How
Extraction Exact and normalized match vs. golden values, per overlay
Evidence faithfulness % of values with a verifiable span Deterministic check + sampled audit
Fabrication % of not_found cases where a value was returned Golden negatives
Schema coverage % of substantive sections covered Coverage analysis
Gap discovery Proposal acceptance rate; alias-vs-new accuracy Approval outcomes
Mapping safety Invented-IRI rate (target: 0) Resolution against pinned release
Validation health SHACL violations per 100 documents Validator reports
Schema churn Changes per version by type; % of pages reprocessed Registry diff
Ontology drift Live mappings affected by a candidate FIBO release Release diff
Calibration Confidence vs. reviewer outcome Reliability analysis
Human effort Review minutes per document; % auto-accepted Workflow telemetry

Release gate: every release, whether schema or prompt, must show no regression in evidence faithfulness, fabrication or invented-IRI rate, and an improvement on its target metric.


13. Governance and Model Risk

Many articles still cite SR 11-7. On 17 April 2026, the US Federal Reserve, OCC and FDIC issued revised model risk management guidance (SR 26-2 / OCC Bulletin 2026-13), which supersedes it. Two points matter here:

  • The revised guidance says generative and agentic AI models are not within its scope, while a bank's broader risk management and governance practices should still apply.
  • It excludes deterministic rule-based processes from the definition of a model. Your SHACL shapes and business rules are software controls, not models.

My view: because LLM agents currently sit outside the formal MRM perimeter, architects have to supply the discipline themselves. This architecture does that:

  1. Inventory and tiering of every agent, prompt version and optimizer run, by downstream materiality.
  2. Effective challenge: golden sets, metrics and release decisions get independent review.
  3. Attribute-level lineage from document → clause → value → schema version → prompt version → consumer, consistent with the BCBS 239 emphasis on data lineage.
  4. Change control per loop: data owners approve schema changes; gates plus humans approve prompt releases; FIBO upgrades need an impact analysis.
  5. HITL thresholds based on calibrated confidence and materiality.
  6. Jurisdiction checks: under the EU AI Act as amended by the Digital Omnibus, high-risk obligations for creditworthiness assessment of natural persons now apply from December 2027. Corporate credit-agreement extraction isn't credit scoring in itself, but if outputs feed decisions about individuals, review the classification with counsel.

14. Scaling Across Business Lines: One Platform, Many Domain Packs

Credit agreements are the hardest case, but large banks face the same problem in home lending and wealth management, with very different profiles:

Home lending Corporate finance Wealth management
Documents Loan files with 100+ document types 200–600 page credit agreements, amendments Account opening, KYC, trusts, investment policy statements
Shape Very high volume, mostly standard forms Low volume, highly bespoke Medium volume, mixed
Dominant loop Loop 2: quality at volume Loop 1: schema evolution Both
Best-fit standard MISMO, mapped to FIBO FIBO loan and debt modules FIBO securities and party modules; ISO 20022 where relevant
Hardest part Sorting documents within a loan file, PII, fair-lending controls Definitions, cross-references, covenant logic Ownership structures, KYC/AML evidence

Don't build three platforms. Build one shared platform with a domain pack per business line:

SHARED PLATFORM (built once)
  ingestion + canonical document model · ontology compiler + OKF bundles ·
  schema registry · deterministic validation · harness (orchestration, traces,
  replay, evaluation, release gates) · feedback router · governance + isolation

DOMAIN PACK (one per business line)
  ontology subset + standard mappings · core schema + overlays ·
  golden sets per document type · review policy · data owners · prompt versions

Two lessons:

  1. Use the domain's own standard where one exists. A FIBO-first mortgage schema would reinvent what MISMO already defines. FIBO's job is to be the bridge between domains.
  2. The bridge is the strategic payoff. Once all three business lines share one party and contract model, you can recognize one client across them: a mortgage borrower who guarantees their company's credit facility and is the grantor of a family trust. That connected client view is worth more than better extraction in any one business line.

15. Reference Implementation on AWS (One Option)

Capability AWS option Vendor-neutral equivalent
OCR, layout, tables Amazon Textract; Bedrock Data Automation Docling, Azure Document Intelligence
Agent reasoning Amazon Bedrock foundation models Any governed LLM endpoint
Governed backbone AWS Step Functions Temporal, Airflow
Agentic steps Bedrock AgentCore Runtime + Strands Agents LangGraph
Ontology graph Amazon Neptune (RDF/SPARQL) with pinned FIBO + extensions GraphDB, Stardog, Apache Jena
Agent knowledge layer OKF bundles compiled from OWL, in Git, served from S3 with a concept index Same
Validation pySHACL + OWL reasoner in containers Same
Schema registry Aurora PostgreSQL + Git for schema artifacts Postgres + Git
Extracted values Aurora / DynamoDB with lineage columns Any relational store
Memory Bedrock AgentCore Memory with tenant namespaces Letta, Zep, custom
Optimization DSPy + GEPA as batch jobs; prompt registry Same
Evidence store Amazon S3 Object storage
Audit CloudTrail + append-only audit table Append-only log

Run the governed backbone (ingest → classify → bind schema → extract → validate → review → publish) as a deterministic workflow, and keep agentic autonomy inside individual steps. The audit trail stays linear even when the reasoning isn't.


16. A Phased Roadmap

Phase Weeks Scope Exit criteria
0: Foundations 0–4 Canonical document model, definitions graph, evidence index; pinned FIBO compiled to OKF; registry with core@1.0.0; golden set of 20 agreements across 2–3 variants Every value traceable to a bounding box
1: Extract + Validate 5–10 Agents 1 and 3 on the core schema; SHACL; review UI; not_found handling Baseline accuracy per overlay; fabrication and invented-IRI rates measured
2: Loop 1 11–16 Overlays; coverage analysis; Agent 2 gap discovery; chat proposals; semver + deprecation; targeted reprocessing A minor schema change completes end-to-end, with audit, in under a day
3: Loop 2 17–24 GEPA on Agents 1 and 3; stratified gates; shadow releases; FIBO upgrade drill Held-out improvement with zero regression on faithfulness and fabrication

Key idea: Build Loop 1 before Loop 2. Optimizing prompts against a schema that changes every week means optimizing against a moving target.


17. Anti-Patterns to Avoid

  • "Self-evolving" as self-modifying. Agents writing directly to the schema or ontology.
  • A v1 schema per document. Thousands of incomparable schemas.
  • One monolithic schema instead of a core + overlays.
  • Schema explosion. Every one-off clause becomes an attribute.
  • Copying FIBO wholesale as your extraction schema.
  • Hand-converting FIBO to Markdown, so typed semantics become untyped links.
  • LLM-generated IRIs that look official but don't resolve.
  • An unpinned ontology that silently picks up a release with removed concepts.
  • Breaking changes without deprecation.
  • Confidence treated as probability without calibration.
  • Human review on every layer, which can't scale and invites rubber-stamping.
  • One loop for everything, so nothing can be attributed.
  • Averages across variants that hide regressions on rare overlays.
  • Tenant isolation by metadata filter only.
  • Ten microservices on day one.

18. What Each Architect Should Take Away

Role Your three takeaways
Enterprise Architect Make it a shared platform with domain packs · keep two loops with separate approvers · the payoff is a connected client view across business lines
Solution Architect Deterministic backbone with agentic steps inside · route sections instead of scanning pages · build the canonical document model before any agent
Data Architect Core + overlays with semver and deprecation · a lineage stamp on every value · "not found" is a valid state
AI / ML Engineer Metrics that return textual feedback · stratify evaluation by overlay · validators are code, never the LLM
Ontology Engineer Pin FIBO; compile OKF from OWL · explicit mapping states · LLMs adjudicate, humans approve
Platform Engineer Tenant isolation by namespace or silo · version everything (parser, prompt, model, schema, ontology, bundle) · targeted reprocessing as a first-class job

Conclusion

Financial document AI is moving from answering questions about documents to maintaining governed knowledge derived from them. For long, heterogeneous agreements, that takes three things RAG doesn't provide:

  • An ontology that gives data stable meaning: FIBO, pinned, mapped, never invented, and delivered to agents as provenance-carrying OKF bundles.
  • A schema registry that turns what we capture into a versioned, layered contract that grows with the portfolio without breaking consumers.
  • Two learning loops, kept apart: one that evolves what we capture through approved proposals, and one that improves how well we capture it through gated, reflective optimization.

The teams that do this well won't be the ones with the longest context windows. They'll be the ones whose schemas can change quickly because every change is proposed, aligned, validated, approved and traceable.

Where to start on Monday morning: pick one agreement family. Pin one FIBO release. Build a golden set of 20 documents across two or three variants. Ship the canonical document model, then Loop 1, then Loop 2. Measure evidence faithfulness, fabrication rate and invented-IRI rate from day one.

If you're building something similar, I'd like to compare notes. Reply here or connect on LinkedIn. Next in the DataOps Labs series: evidence-first ingestion for 450-page agreements, covering definitions graphs, cross-reference resolution and a parser bake-off.


DataOps Labs · Education and Knowledge Democratised. If this was useful, share it with an architect who's fighting a 450-page PDF this week.

6 views

About the author

  • I'm Ayyanar Jeyakrishnan ; aka AJ. With over 21 years in IT, I'm a passionate Multi-Cloud Architect specialising in crafting scalable and efficient cloud solutions. I've successfully designed and implemented multi-cloud architectures for diverse organisations, harnessing AWS, Azure, and GCP. My track record includes delivering Machine Learning and Data Platform projects with a focus on high availability, security, and scalability. I'm a proponent of DevOps and MLOps methodologies, accelerating development and deployment. I actively engage with the tech community, sharing knowledge in sessions, conferences, and mentoring programs. Constantly learning and pursuing certifications, I provide cutting-edge solutions to drive success in the evolving cloud and AI/ML landscape.

No comments yet. Be the first to comment.

Self-Evolving Financial Ontologies: How Multi-Agent AI Learns to Discover Enterprise Schemas from Financial Documents