Schema Validation for Parsed Expense Data

Raw document extraction is only the entry point for accurate CAM reconciliation: the strings and numbers that come out of a parser are unproven until a deterministic validation layer certifies that every field is well-typed, every amount ties out to the penny, and every record carries the lease context downstream allocation needs. This validation stage is part of the Automated Invoice Parsing & Data Ingestion pipeline, sitting immediately after extraction and before GL code mapping for CAM expenses — it is the gate that decides whether a parsed invoice is trustworthy enough to enter the recoverable pool or must be quarantined for human review. Skip it, and OCR misreads, transposed figures, and vendor formatting drift flow untouched into tenant statements, surfacing months later as disputes and audit findings that force costly restatements.

Prerequisites & Data Contracts #

Validation cannot invent context it was never given, so the correctness of this stage depends entirely on the contract it inherits from extraction. Before any rule runs, three upstream artifacts must already exist and be attached to the record under inspection.

The first is the raw extraction payload produced by an engine such as coordinate-aware table extraction with pdfplumber. That payload must preserve original bounding-box coordinates and per-field OCR confidence scores alongside the extracted text. Confidence is not decoration — the validation layer uses it to decide whether a numerically valid amount is nonetheless too uncertain to trust, and coordinates give an accountant a deterministic way to trace a rejected field back to its exact location on the source page.

The second is a stored, immutable source document plus its content hash. Validation records the hash on every accepted and rejected record so that an auditor sampling a CAM line item can be shown the exact PDF it came from, byte-for-byte. Without this binding, a passing schema check proves internal consistency but not provenance, and provenance is what survives an audit.

The third is the lease and GL context the invoice will be measured against: the property’s chart of accounts, the lease-defined recoverable categories from defining CAM expense categories in commercial leases, and the capitalization threshold above which a repair becomes a capital improvement. These are read, not computed, at validation time — typically from a lease abstraction database. The contract is explicit: extraction owns what the document says, the lease abstraction owns what the lease permits, and validation is the only stage that holds both at once and can reconcile them.

The output contract is equally strict. Every record leaving this stage is either an accepted, fully-typed object carrying its ruleset version, or a quarantined record carrying a structured error payload — never a silently coerced half-valid row. Nothing reaches GL code mapping for CAM expenses without passing through this binary.

Rule Design: Structural, Financial, and Lease-Logic Layers #

A production validation layer is not a single if cascade; it is three ordered layers, each of which must pass before the next runs. Ordering matters because a financial check on an un-coerced string is meaningless, and a lease-logic check on an invoice whose totals do not reconcile is worse than meaningless — it is misleading.

Layer one, structural enforcement, coerces types and rejects malformed shapes: dates must parse as ISO 8601, every monetary field must become a two-place Decimal, and required fields (invoice_id, vendor_name, invoice_date, property_id, total_amount, and at least one line item) must be present. Type coercion happens here and only here, so that every downstream layer operates on canonical values.

Layer two, financial consistency, enforces the arithmetic identity that defines a well-formed invoice. The sum of every line item’s pre-tax amount and tax must reconcile to the stated invoice total within a one-cent tolerance:

The tolerance exists because vendors round line items independently before summing; it is deliberately tight — a penny, not a dollar — because a wider band hides exactly the transposition errors this layer is meant to catch.

Layer three, lease logic, validates business rules that only the lease can adjudicate: a recoverable_pct must fall in the closed interval $[0, 100]$, an expense whose amount exceeds the property’s capitalization threshold must be flagged rather than passed as an in-year operating cost, and a category the lease excludes cannot be presented as recoverable pass-through. When a lease_clause_ref is present, the extracted percentage is cross-checked against the abstracted clause, and any divergence is surfaced rather than trusted. This is where a valid-looking invoice claiming a non-recoverable capital replacement as operating CAM is stopped, well upstream of the pro rata allocation algorithms that would otherwise distribute the error across every tenant.

Python Implementation #

The three layers map cleanly onto Pydantic v2, which gives runtime enforcement, automatic coercion, and explicit error messaging in one declaration. Field constraints handle layer one, field_validator hooks normalize monetary values, and model_validator(mode="after") expresses the cross-field financial and lease-logic checks that no single field can see on its own. All monetary arithmetic uses Python’s decimal module — never float — because binary floating point cannot represent most cent values exactly, and summing thousands of them drifts an invoice out of tolerance for reasons that have nothing to do with the data.

from __future__ import annotations

from datetime import date
from decimal import Decimal, ROUND_HALF_UP
from typing import Optional

from pydantic import BaseModel, Field, field_validator, model_validator

CENT = Decimal("0.01")


def to_cents(value: object) -> Decimal:
    """Coerce any numeric-ish input to a two-place Decimal (banker-safe).

    Accepts int, float, str, or Decimal. Floats are stringified first so
    we quantize the *decimal* value the vendor intended, not its binary
    approximation.
    """
    if isinstance(value, Decimal):
        dec = value
    else:
        dec = Decimal(str(value))
    return dec.quantize(CENT, rounding=ROUND_HALF_UP)


class CAMLineItem(BaseModel):
    """A single recoverable-or-not expense line from a vendor invoice."""

    description: str
    gl_code: str
    amount: Decimal = Field(gt=0, description="Pre-tax line-item amount")
    tax_amount: Decimal = Field(default=Decimal("0.00"), ge=0)
    recoverable_pct: Optional[Decimal] = Field(default=None, ge=0, le=100)
    lease_clause_ref: Optional[str] = None
    ocr_confidence: Decimal = Field(default=Decimal("1.00"), ge=0, le=1)

    @field_validator("amount", "tax_amount", mode="before")
    @classmethod
    def enforce_two_decimals(cls, v: object) -> Decimal:
        # Layer one: canonicalize money before any arithmetic sees it.
        return to_cents(v)


class ParsedCAMInvoice(BaseModel):
    """A parsed invoice awaiting certification for the recoverable pool."""

    invoice_id: str
    vendor_name: str
    invoice_date: date
    service_date: Optional[date] = None  # GAAP period matching, distinct from invoice_date
    property_id: str
    total_amount: Decimal
    line_items: list[CAMLineItem] = Field(min_length=1)
    ruleset_version: str = "v2"
    source_hash: Optional[str] = None

    @field_validator("total_amount", mode="before")
    @classmethod
    def coerce_total(cls, v: object) -> Decimal:
        return to_cents(v)

    @model_validator(mode="after")
    def validate_line_item_sum(self) -> "ParsedCAMInvoice":
        # Layer two: the arithmetic identity of a well-formed invoice.
        calculated = sum(
            (item.amount + item.tax_amount for item in self.line_items),
            start=Decimal("0.00"),
        )
        if abs(calculated - self.total_amount) > CENT:
            raise ValueError(
                f"Lease math mismatch on {self.invoice_id}: line items sum to "
                f"{calculated}, invoice total is {self.total_amount}"
            )
        return self

The validate_line_item_sum model validator is the single most valuable rule in the layer: it fails fast on the exact class of error — OCR misreads, dropped rows, vendor rounding mistakes — that would otherwise require a manual journal correction after month-end close. Because it runs after field coercion, both sides of the comparison are already canonical two-place decimals, so the only thing the comparison can reveal is a genuine arithmetic discrepancy. For the full layered architecture — including the quarantine writer, ruleset versioning, and streaming validation — see Building a CAM Data Validation Layer.

Layer three’s lease logic is best expressed as a separate validator so it can be toggled per property and versioned independently:

class CapitalizationError(ValueError):
    """Raised when an expense should be capitalized, not expensed in-year."""


def apply_lease_logic(
    invoice: ParsedCAMInvoice,
    capitalization_threshold: Decimal,
    min_confidence: Decimal = Decimal("0.85"),
) -> list[str]:
    """Return a list of lease-logic warnings; empty means the invoice is clean.

    Warnings are structured strings the quarantine writer serializes into the
    error payload. A non-empty result routes the invoice to human review
    rather than the recoverable pool.
    """
    warnings: list[str] = []
    for item in invoice.line_items:
        # Low OCR confidence is flagged, never silently trusted.
        if item.ocr_confidence < min_confidence:
            warnings.append(
                f"{item.gl_code}: OCR confidence {item.ocr_confidence} "
                f"below floor {min_confidence}"
            )
        # A repair above the cap-ex line is a capital improvement, not CAM.
        if item.amount > capitalization_threshold:
            warnings.append(
                f"{item.gl_code}: amount {item.amount} exceeds capitalization "
                f"threshold {capitalization_threshold} — flag for cap-ex review"
            )
    return warnings

Validation Rules & Edge Cases #

The failure modes that actually break CAM reconciliation are rarely the obvious ones. A well-designed validation layer anticipates the following, each of which has surfaced in real vendor feeds:

• Line items that sum correctly but tax that does not. A vendor may present a correct pre-tax subtotal while the tax column is scanned from a smudged cell. Including tax_amount in the reconciliation identity — rather than checking only pre-tax amounts — catches this; validating pre-tax alone would pass a materially wrong invoice.
• Rounding-induced false rejects. If line items are stored at full precision but the total is quantized, an exact-arithmetic comparison fails on invoices that are actually correct. Quantizing both sides to cents before comparing, as the implementation does, removes the false positive while preserving the one-cent tolerance for genuine drift.
• Negative and credit lines. Vendor credits and reversals arrive as negative amounts, but the amount field is constrained gt=0. Credits must be modeled as a distinct line type with their own sign rules, or a legitimate credit memo is rejected as malformed. Decide this at contract time, not by loosening the constraint.
• Multi-property statements. A single vendor invoice may bill across several assets. A property_id on the invoice header is insufficient; each line item may need its own property tag, and validation must reject a statement whose line-level properties disagree with the header unless a split is explicitly declared.
• Recoverability claimed against an excluded category. An invoice may assert recoverable_pct = 100 on an expense the lease excludes entirely. Range-checking the percentage passes it; only cross-referencing the category against the lease exclusion list catches it — a check developed in best practices for CAM expense exclusion tracking.
• Service period straddling the reconciliation year. GAAP recognizes cost in the period the service was rendered, not billed. A December service invoiced in January belongs to the prior CAM year, so service_date is validated as a first-class field distinct from invoice_date; collapsing the two silently misstates year-end pools.

Records that trip any of these rules do not halt the pipeline. They are written to a quarantine store with a structured error payload — the rule that failed, the offending field, its coordinates and confidence, and the source hash — so that an accountant can triage them without blocking the invoices that passed. Transient infrastructure failures (an API timeout, a locked connection) are distinct from data failures and are retried with exponential backoff and jitter; only permanently invalid records land in the dead-letter queue.

Integration Points #

Validation is a middle stage, defined as much by what it hands off as by what it enforces. Its accepted output flows directly into GL code mapping for CAM expenses, which can assume — because this stage guarantees it — that every amount is a clean two-place Decimal, every required field is present, and the invoice reconciles. That guarantee is what lets the mapper focus on classification rather than defensive re-checking.

Downstream of mapping, validated and coded records feed the reconciliation engine and the pro rata allocation algorithms that compute each tenant’s share, as well as the cap logic in managing expense caps and controllable limits. Every one of those computations inherits the integrity — or the defects — of what validation certified, which is why the stage is deliberately strict.

On the input side, the quarantine payload is itself an integration surface: it feeds a review dashboard and a metrics stream. A rising rejection rate for a specific vendor is an early signal of a changed invoice template, and catching that at validation is far cheaper than discovering it in a failed year-end tie-out. At portfolio scale, where thousands of invoices arrive in a single close window, synchronous validation becomes a bottleneck; decoupling it via async batch processing for high-volume invoices lets validation run concurrently across workers while still enforcing the same strict contract on every record. The ruleset version stamped on each accepted record ties the whole chain together — an audit can reproduce exactly which rules certified any given line item.

Testing & Verification #

Lease-math correctness is not something to trust to inspection; it is something to pin with fixtures. The test strategy centers on constructing invoices whose arithmetic outcome is known, then asserting that validation accepts the clean ones and rejects the broken ones with the right error — verifying both the pass path and, just as importantly, the fail path.

import pytest
from decimal import Decimal
from pydantic import ValidationError


def _line(amount: str, tax: str = "0.00", **kw) -> CAMLineItem:
    return CAMLineItem(
        description="HVAC maintenance",
        gl_code="5100-HVAC",
        amount=Decimal(amount),
        tax_amount=Decimal(tax),
        **kw,
    )


def test_reconciling_invoice_is_accepted() -> None:
    invoice = ParsedCAMInvoice(
        invoice_id="INV-001",
        vendor_name="Acme Facilities",
        invoice_date="2026-01-15",
        property_id="PROP-42",
        total_amount="1075.50",
        line_items=[_line("900.00", "75.50"), _line("100.00")],
    )
    # 900.00 + 75.50 + 100.00 == 1075.50, exactly.
    assert invoice.total_amount == Decimal("1075.50")


def test_transposed_total_is_rejected() -> None:
    with pytest.raises(ValidationError, match="Lease math mismatch"):
        ParsedCAMInvoice(
            invoice_id="INV-002",
            vendor_name="Acme Facilities",
            invoice_date="2026-01-15",
            property_id="PROP-42",
            total_amount="1750.50",  # digits transposed from 1075.50
            line_items=[_line("900.00", "75.50"), _line("100.00")],
        )


def test_float_input_does_not_drift_out_of_tolerance() -> None:
    # 0.1 + 0.2 in float is 0.30000000000000004; to_cents must absorb this.
    invoice = ParsedCAMInvoice(
        invoice_id="INV-003",
        vendor_name="Acme Facilities",
        invoice_date="2026-01-15",
        property_id="PROP-42",
        total_amount=0.30,
        line_items=[_line("0.10"), _line("0.20")],
    )
    assert invoice.total_amount == Decimal("0.30")


def test_capitalization_threshold_flags_large_repair() -> None:
    invoice = ParsedCAMInvoice(
        invoice_id="INV-004",
        vendor_name="Acme Roofing",
        invoice_date="2026-01-15",
        property_id="PROP-42",
        total_amount="12000.00",
        line_items=[_line("12000.00")],
    )
    warnings = apply_lease_logic(invoice, capitalization_threshold=Decimal("5000.00"))
    assert any("cap-ex review" in w for w in warnings)

The tolerance handling deserves its own assertion because it is the rule most likely to regress: a fixture built from float inputs (0.10 + 0.20) proves that coercion, not luck, keeps the invoice in tolerance. Property-based testing extends this well — generating random line-item sets, computing the true total in Decimal, and asserting that the reconciling invoice always validates while a total perturbed by more than a cent always fails. Fixtures should also assert on the contents of the error, not merely that one was raised, so a future refactor cannot quietly change which rule fired.

Frequently Asked Questions #

Why validate line-item sums instead of trusting the vendor’s stated total? Because the stated total and the line items are extracted from different regions of the document and fail independently. A smudged total cell or a dropped line produces an internally inconsistent invoice that looks fine field-by-field; only the cross-field reconciliation identity catches it, and catching it here prevents a manual journal correction after close.

Why must monetary amounts use Decimal rather than float? Binary floating point cannot represent most cent values exactly, so summing thousands of float amounts accumulates fractional-cent drift and pushes correct invoices out of tolerance. Quantizing every amount to a two-place Decimal keeps the reconciliation identity exact and the audit math reproducible.

What happens to an invoice that fails validation? It is written to a quarantine store with a structured error payload — the failed rule, the offending field, its coordinates and OCR confidence, and the source-document hash — and routed to a review dashboard. It never enters the recoverable pool and never blocks the invoices that passed.

How does validation tell a capital improvement apart from a recoverable repair? It compares each line amount against the property’s lease-defined capitalization threshold. A repair-coded line above that threshold is flagged for cap-ex review rather than expensed in-year, because misclassifying a capital replacement into the operating pool overstates recoverable CAM for every tenant.

Why is service date validated separately from invoice date? GAAP recognizes a cost in the period the service was rendered, not the period it was billed. A December service invoiced in January belongs to the prior CAM year, so service_date is a first-class field; collapsing it into invoice_date silently misstates year-end pools.

Where This Fits #

Schema validation is the control point that turns noisy extraction output into records an accountant can defend line by line. By coercing types before it checks arithmetic, enforcing the line-item reconciliation identity to the penny, and adjudicating lease logic against the abstracted clause before anything reaches a pool, it guarantees the contract every downstream stage relies on. It consumes the output of coordinate-aware table extraction with pdfplumber, hands certified records to GL code mapping for CAM expenses, scales through async batch processing for high-volume invoices, and ultimately bounds the integrity of every tenant share the pro rata allocation algorithms produce — which is precisely why it is engineered to be strict rather than forgiving.

Related #

• Automated Invoice Parsing & Data Ingestion — the parent pipeline this validation stage belongs to, from raw document to reconciled GL entry.
• Building a CAM Data Validation Layer — the full layered architecture: quarantine writer, ruleset versioning, and streaming validation.
• GL code mapping for CAM expenses — the downstream stage that consumes every record this gate certifies.
• Coordinate-aware table extraction with pdfplumber — the upstream extractor whose payload and confidence scores this stage validates.
• Best practices for CAM expense exclusion tracking — the lease exclusion logic behind layer three’s recoverability checks.