CAM Reconciliation

Correcting OCR Errors in Financial Invoice Data

Optical character recognition on a scanned invoice rarely fails loudly; it substitutes a confident wrong digit, and correcting those errors before a charge reaches a tenant ledger is a distinct engineering discipline. This page is the repair stage of OCR for scanned CAM invoices, the image-based branch of the broader automated invoice parsing and data ingestion pipeline. It picks up where recognition ends: it takes the word boxes recovered by extracting scanned invoice tables with Tesseract OCR, scans each recovered amount for the character-level confusions that plague monetary fields, repairs what can be repaired deterministically, checks every figure against the invoice’s own arithmetic, and routes anything still doubtful to a person. The goal is narrow and unforgiving: a recovered charge is either correct to the cent or it is quarantined — a plausible-looking wrong number is the one outcome the reconciliation system must never emit.

The correction pipeline for a recovered CAM invoice amount A recovered amount cell, carrying its text and a recognizer confidence, is scanned for known digit confusions, normalized to an exact Decimal, and checked against the invoice's own footing arithmetic. A confidence-weighted risk score then routes the cell: a low-risk repair is auto-corrected and flows on to schema validation, while a high-risk cell is flagged for human review before any figure reaches the ledger. Recovered cell text + conf Detect O·0 l·1 ,·. Normalize money → Decimal Foot lines = total Score conf-weighted Route Auto-corrected to schema check Flagged human review low risk high risk
The repair stage: scan a recovered amount cell for known confusions, normalize it to a Decimal, check it against the invoice's own arithmetic, and route by risk to an automatic fix or a human.

Context & When to Use This Approach

Recognition and correction are two different jobs, and conflating them is why so many scanned-invoice pipelines leak wrong numbers. The recovery stage answers “what does this pixel region say?”; the correction stage answers “can I trust what it said, and if not, can I mechanically repair it or must a person look?” You need a dedicated correction stage the moment recovered figures start feeding money math rather than a search index. The concrete triggers in a common-area-maintenance feed are recognizable:

  • A vendor sends the same scanned template every month — a snow-removal contractor, a fire-alarm inspection firm, a water utility — and the recognizer makes the same mistakes on it repeatedly, so a systematic repair pays for itself.
  • Recovered charges flow downstream into GL code mapping for CAM expenses and, eventually, the pro-rata allocation algorithm that splits cost across tenants, so a single misread digit becomes a mis-billed reconciliation.
  • The invoice carries its own checksums — line amounts that must foot to a printed subtotal — and you want to use that arithmetic to catch recognition errors that individually looked confident.
  • You need every automatic repair to be explainable to an auditor: which characters were changed, why the change was safe, and which figures a human confirmed rather than the machine guessing.

If you are eyeballing a handful of scans by hand, none of this is worth building. The stage below is for volume, where a person cannot re-read every figure and the pipeline must decide, cell by cell, what it may fix silently and what it must escalate. It assumes recovery has already run: each amount cell arrives as recognized text plus a per-cell confidence score, and this stage never re-opens the image except to crop a region for the reviewer. Repairs here are confined to money and other strictly-typed numeric fields; a garbled word in a free-text description is left for the recognizer’s own confidence to flag, because there is no arithmetic to check it against.

Step-by-Step Implementation

The pattern is a short funnel. First widen the net — flag every character-level tell that an amount may be misread. Then attempt a deterministic repair into an exact value. Then subject that value to the invoice’s own arithmetic, which is a far stronger signal than any single recognizer score. Finally, fold recognizer confidence, the character flags, and the arithmetic result into one risk number that decides auto-correct versus review. Every monetary value is a decimal.Decimal from the instant a token is meant to be money — never float — so no binary rounding is ever mistaken for an OCR error, and a thousand summed line items still foot to the cent.

Step 1 — Detect suspicious tokens before touching them

Correction begins with suspicion, not repair. Scan each recovered amount for the confusions a print-oriented recognizer is known to make — a letter where a digit belongs (O/0, l/1/I, S/5, B/8), and the separator ambiguity between a thousands comma and a decimal point. Recording why a token is doubtful, as structured flags rather than a bare pass/fail, lets the later routing step weigh the evidence instead of discarding it.

from __future__ import annotations

from dataclasses import dataclass

# Print-recognizer confusions that flip a numeric field alphabetic or vice versa.
DIGIT_LOOKS_LIKE: dict[str, str] = {
    "O": "0", "o": "0", "Q": "0", "D": "0",
    "l": "1", "I": "1", "|": "1",
    "S": "5", "s": "5",
    "B": "8", "Z": "2", "g": "9",
}


@dataclass(frozen=True)
class Suspicion:
    """One structured reason a recovered amount cell may be mis-recognized."""

    kind: str      # "alpha_in_amount", "ambiguous_separator", or "substitution"
    detail: str    # human-readable note for the review queue


def scan_amount_token(token: str) -> list[Suspicion]:
    """Flag the character-level tells that a money cell was misread.

    Runs before any repair, so the corrector knows *why* a token is doubtful,
    not merely that it failed to parse. Returns an empty list for a clean token.
    """
    flags: list[Suspicion] = []
    letters = [ch for ch in token if ch.isalpha()]
    if letters:
        flags.append(Suspicion("alpha_in_amount", f"letters {''.join(letters)!r} in a money field"))
    if "," in token and "." in token:
        # Both separators present: grouping vs. decimal must be disambiguated.
        flags.append(Suspicion("ambiguous_separator", "comma and period both present"))
    for ch in token:
        if ch in DIGIT_LOOKS_LIKE:
            flags.append(Suspicion("substitution", f"{ch!r} is often misread for {DIGIT_LOOKS_LIKE[ch]!r}"))
    return flags

Step 2 — Normalize the token and re-parse it to Decimal

With the tells recorded, attempt a deterministic repair. Map only the high-confidence letter-to-digit confusions — the ones that are almost never legitimately alphabetic inside an amount — then disambiguate the separators by position: whichever of comma or period appears last is the decimal point, and the other is grouping. Anything that still cannot be coerced into a money shape returns None, which is a deliberate refusal to guess rather than a silent zero.

from __future__ import annotations

import re
from decimal import Decimal, InvalidOperation, ROUND_HALF_UP

CENTS = Decimal("0.01")
_KEEP = re.compile(r"[^0-9.,\-]")


def repair_amount(token: str) -> Decimal | None:
    """Coerce a recovered amount into an exact Decimal, applying digit repairs.

    High-confidence letter-to-digit confusions are mapped to their digit, the
    thousands and decimal separators are disambiguated by position, and the
    result is quantized to cents. Returns None when the token cannot be made to
    look like money, so the caller routes it to review instead of inventing a value.
    """
    text = token.replace("$", "").replace(" ", "").strip()
    # Repair confusions only inside what is meant to be a numeric field.
    repaired = "".join(DIGIT_LOOKS_LIKE.get(ch, ch) for ch in text)
    repaired = _KEEP.sub("", repaired)
    if not any(ch.isdigit() for ch in repaired):
        return None

    last_dot, last_comma = repaired.rfind("."), repaired.rfind(",")
    if last_dot == -1 and last_comma == -1:
        candidate = repaired
    else:
        decimal_sep = "." if last_dot > last_comma else ","
        group_sep = "," if decimal_sep == "." else "."
        candidate = repaired.replace(group_sep, "").replace(decimal_sep, ".")

    try:
        return Decimal(candidate).quantize(CENTS, rounding=ROUND_HALF_UP)
    except InvalidOperation:
        return None

Step 3 — Validate against the invoice’s own arithmetic

A recovered figure that parses cleanly can still be wrong, and the strongest evidence of that is arithmetic the invoice already contains. Two identities matter. Per line, the quantity times the unit rate must reconcile to the extended amount, line_total=quantity×unit_rateline\_total = quantity \times unit\_rate; across the page, the recovered line amounts must foot to the printed subtotal:

∑i=1namounti=subtotal\sum_{i=1}^{n} amount_i = subtotal

Footing is a page-level checksum that catches a misread digit even when every individual cell’s confidence looked acceptable, and per-line extension localizes the blame to a single row. Consider a monthly bill from Cardinal Mechanical for Beacon Hill Plaza: three service lines print at $1,204.50, $860.00, and $415.25 against a subtotal of $2,479.75. If the recognizer reads the first as $1.204,50 and the normalizer recovers $1,204.50, the page still foots; if it reads it as $7,204.50, the sum overshoots by exactly $6,000 and the row is pinpointed.

from __future__ import annotations

from dataclasses import dataclass
from decimal import Decimal

TOLERANCE = Decimal("0.01")


@dataclass(frozen=True)
class LineItem:
    """One recovered invoice line: recognized text already parsed to Decimal."""

    description: str
    quantity: Decimal
    unit_rate: Decimal
    amount: Decimal          # recovered extended amount
    amount_confidence: int   # recognizer score 0..100 for the amount cell


def line_extends(line: LineItem) -> bool:
    """True when quantity * unit_rate reconciles to the extended amount.

    Per-row footing localizes the error: one failing line points at the exact
    cell to re-examine instead of condemning the whole invoice.
    """
    expected = (line.quantity * line.unit_rate).quantize(TOLERANCE)
    return abs(expected - line.amount) <= TOLERANCE


def invoice_foots(lines: list[LineItem], printed_subtotal: Decimal) -> bool:
    """True when the recovered line amounts sum to the printed subtotal.

    The invoice carries its own checksum. A scan that does not foot has a
    misread digit somewhere, even if every cell's confidence looked fine.
    """
    recovered_total = sum((li.amount for li in lines), Decimal("0"))
    return abs(recovered_total - printed_subtotal) <= TOLERANCE

Step 4 — Score confidence and route the correction

The final step turns three streams of evidence into one decision. Recognizer confidence, the count of character-level suspicions, and the arithmetic result combine into a risk score in the interval [0,1][0, 1]:

risk=wc(1−conf100)+ws⋅min⁡(s, 3)3+wa⋅frisk = w_c\left(1 - \frac{conf}{100}\right) + w_s \cdot \frac{\min(s,\ 3)}{3} + w_a \cdot f

Here confconf is the recognizer’s own confidence, ss is the number of suspicions, and ff is $1$ when the arithmetic fails and $0$ when it holds. The arithmetic weight waw_a is set highest deliberately: a cell that fails to foot is untrustworthy no matter how confident the recognizer was, so arithmetic evidence must be able to override optimism. A repair is auto-accepted only when its risk sits below the amount threshold and a Decimal was actually recovered; everything else carries its reasons to a reviewer who sees the cropped source beside the value.

from __future__ import annotations

from dataclasses import dataclass
from decimal import Decimal
from enum import Enum

# Weights tuned so a single arithmetic failure outweighs a merely low score.
W_CONFIDENCE = Decimal("0.4")
W_SUSPICION = Decimal("0.15")
W_ARITHMETIC = Decimal("0.6")
AMOUNT_RISK_THRESHOLD = Decimal("0.25")


class Route(str, Enum):
    AUTO_CORRECTED = "auto_corrected"
    REVIEW = "review"


@dataclass(frozen=True)
class CorrectionOutcome:
    """The verdict for one amount cell: what it became and how it was decided."""

    route: Route
    corrected_amount: Decimal | None
    risk: Decimal
    reasons: list[str]


def route_correction(line: LineItem, subtotal_ok: bool) -> CorrectionOutcome:
    """Score an amount cell, then auto-correct it or send it to a human.

    Recognizer confidence, character-level suspicions, and the arithmetic
    checks combine into a risk score. A repaired value is trusted only when
    the score is below the amount threshold and a Decimal repair succeeded;
    everything else routes to review with its accumulated reasons.
    """
    raw_token = str(line.amount)
    suspicions = scan_amount_token(raw_token)
    corrected = repair_amount(raw_token)

    conf_gap = Decimal(100 - line.amount_confidence) / Decimal(100)
    suspicion_load = min(Decimal(len(suspicions)), Decimal(3)) / Decimal(3)
    arithmetic_ok = line_extends(line) and subtotal_ok
    arithmetic_fail = Decimal(0) if arithmetic_ok else Decimal(1)

    risk = (W_CONFIDENCE * conf_gap
            + W_SUSPICION * suspicion_load
            + W_ARITHMETIC * arithmetic_fail)

    reasons = [s.detail for s in suspicions]
    if not arithmetic_ok:
        reasons.append("line does not extend or invoice does not foot")

    safe = risk < AMOUNT_RISK_THRESHOLD and corrected is not None
    return CorrectionOutcome(
        route=Route.AUTO_CORRECTED if safe else Route.REVIEW,
        corrected_amount=corrected,
        risk=risk,
        reasons=reasons,
    )

Gotchas & Known Limitations

The mistakes that actually corrupt a reconciliation are subtle, because a bad correction looks exactly like a good one until year-end. Treat this as a pre-flight checklist before trusting the stage against a new vendor:

Verification

The pure functions here — suspicion scanning, amount repair, footing, and routing — are deterministic, so they get exact-equality tests rather than the tolerance-bounded golden files that recognition itself requires. Assert that repair_amount("$1,204.50") and repair_amount("1.204,50") both return Decimal("1204.50"), that a substituted token like repair_amount("12O4.50") recovers Decimal("1204.50") because the O maps to 0, and that a hopeless token such as repair_amount("--") returns None instead of a fabricated figure. For the arithmetic layer, build a small LineItem set whose amounts sum to a known subtotal and assert invoice_foots is True, then perturb one amount by more than a cent and assert it flips to False. The routing tests are the ones that matter most: a high-confidence cell whose line foots must return Route.AUTO_CORRECTED with a risk below the threshold, while the same cell with its line broken must return Route.REVIEW even though nothing about the recognizer’s confidence changed — proving that arithmetic evidence overrides optimism exactly as the weights intend. Round the suite out with property-based tests that generate random line sets, compute the true subtotal in Decimal, and assert that a reconciling invoice never routes a clean cell to review while a subtotal perturbed past the tolerance always does. A cell that clears every gate here re-enters the automated invoice parsing and data ingestion pipeline as though it had been recognized cleanly, while everything flagged is held for a person — the same discipline the recovery step in extracting scanned invoice tables with Tesseract OCR applies one stage earlier.