CAM Reconciliation

OCR for Scanned CAM Invoices

Optical character recognition turns a scanned, image-only CAM invoice into structured line items when no embedded text layer exists to read, recovering vendor charges from pixels rather than glyphs. It is the image-based branch of the automated invoice parsing and data ingestion pipeline, and it exists because a meaningful share of common-area-maintenance bills never arrive as clean digital PDFs. A landscaping contractor prints an invoice, signs it, runs it through a desktop scanner, and emails a flattened image; a property manager photographs a paper snow-removal ticket; a utility re-scans a decades-old billing template. To a coordinate-aware extractor these files are blank — there are no characters to locate, only a raster of light and dark pixels. This page specifies how the reconciliation system detects that condition, rasterizes and cleans the image, runs OCR to recover words with pixel coordinates, rebuilds the invoice table from those word boxes, scores every recovered cell for confidence, and routes anything the engine cannot vouch for to a human before a wrong dollar amount reaches the ledger.

The OCR sub-pipeline for scanned CAM invoices A scanned image-only PDF is rasterized at a target DPI, then preprocessed (deskew, denoise, threshold) to a clean binary image. Tesseract OCR recovers word bounding boxes, which are reconstructed into table cells. A confidence gate splits the output: cells at or above the threshold are validated and continue to schema validation, while cells below it are routed to manual review before any number reaches the ledger. Scanned PDF image only Rasterize 300 DPI Preprocess deskew · clean OCR word boxes Reconstruct table cells Confidence gate Validated to schema check Low confidence manual review pass flag
The image-based path: rasterize the scanned page, clean it, recover word boxes with OCR, rebuild the table, and gate each cell on confidence — passing trusted rows to schema validation and flagging the rest for a human.

Prerequisites & Data Contracts

The OCR path is a fallback, not the default. It runs only when the primary text extractor has nothing to read, so before a single pixel is processed the pipeline must have already made — and recorded — the decision to send this document down the image branch. Four inputs must exist and be trustworthy before rasterization begins.

A raw byte source and a page classifier. The ingestion layer hands the OCR stage the original file bytes plus a per-page verdict: does this page carry a real embedded text layer, or is it image-only? The classifier is deliberately conservative. It attempts a text extraction with the same machinery the coordinate-aware table extraction with pdfplumber path uses and inspects the result: a page that yields a healthy count of characters with plausible bounding boxes is text-native and must never be sent to OCR, because rasterizing a crisp digital PDF and running recognition over it throws away exact glyph positions and substitutes a fallible guess. A page that yields zero or near-zero characters — or a wall of mojibake from a broken font map — is image-only and belongs here. Getting this gate wrong in either direction is expensive: OCR-ing a clean invoice manufactures errors that did not exist, while text-extracting a scan produces a silent blank.

A resolution the recognizer can actually read. OCR accuracy is a function of pixel density on the glyph, not of the page’s nominal size. The contract requires that image-only pages be rasterized to at least 300 dots per inch; 400 DPI is safer for the small type common on utility and tax bills, and anything below 200 DPI degrades sharply as strokes merge and thin serifs disappear. When the source is already a raster image rather than a vector PDF, the stage records the native resolution and rejects or up-flags anything captured below the floor, because no amount of downstream cleverness recovers detail that was never scanned.

Layout hints for the expected invoice shape. The reconstruction step is far more reliable when it knows roughly what it is looking for. A vendor-specific hint record — how many columns the invoice table carries, which column holds the money, whether amounts are right-aligned, and the header words that mark the table’s top — lets the engine anchor its row and column clustering instead of inferring structure from scratch. These hints come from the same vendor profile that governs automating vendor invoice classification, so a recurring landscaping or janitorial vendor whose scanned format never changes is parsed with far higher confidence than a one-off bill.

The output contract — identical to the text path. This is the load-bearing rule of the whole stage. Whatever the OCR branch emits must match, field for field, the parsed-line-item schema the pdfplumber path produces, so that every stage after ingestion is blind to which branch ran. Downstream, schema validation for parsed expense data and GL code mapping for CAM expenses receive the same description, quantity, and amount fields regardless of origin. The only difference the OCR path adds is metadata: a per-field confidence score and a source_path marker of ocr, both carried alongside the values so the reconciliation system can treat a recovered number with appropriate suspicion without changing the shape of the record.

Algorithm or Rule Design

The pipeline is a sequence of narrow transformations, each one making the next more reliable. The core insight is that recognition quality is decided before the recognizer ever runs: a clean, upright, high-contrast binary image is worth more than any post-hoc correction.

Rasterize. The image-only PDF page is rendered to a bitmap at the target DPI. A vector page is rasterized directly; a page that is already a raster is decoded as-is. The output is a single-channel or three-channel array the preprocessing steps operate on.

Preprocess. Three corrections dominate real-world scan quality, applied in order.

  • Deskew. Scanners and phone cameras rarely capture a page perfectly square. A page rotated even one or two degrees smears row alignment, so the engine estimates the dominant skew angle θ\theta and rotates the image back to horizontal. Getting rows level is the single highest-leverage fix, because the table reconstruction step clusters words by their vertical position and a tilted page scatters a single row across many y-values.
  • Denoise. Scanner speckle, compression artifacts, and paper texture introduce stray dark pixels that OCR misreads as punctuation or stray glyphs. A median or morphological filter removes isolated specks without eroding stroke edges.
  • Threshold (binarize). Grayscale is converted to pure black-on-white. A global threshold works on evenly lit scans; adaptive (local) thresholding is required when a page has uneven illumination — a shadow across a photographed invoice, or the darkened band a faint fax leaves down one margin. Binarization is what lets the recognizer separate ink from background, and a badly chosen threshold either dissolves thin digits or floods the page with noise.

OCR to word boxes. The cleaned image is passed to the recognizer, which returns not a flat string but a table of words, each with a bounding box (left, top, width, height) and a confidence score from 0 to 100. Recovering geometry is essential: an invoice is a spatial artifact, and a $1,204.50 in the amount column means something entirely different from the same token in a quantity column. Discarding position and keeping only text would make table reconstruction impossible.

Reconstruct the table. The word boxes are grouped back into rows and columns by clustering their coordinates.

  • Rows are found by clustering word boxes on their vertical center. Words whose y-centers fall within a tolerance band — derived from the median glyph height so it scales with DPI — belong to the same line. A well-deskewed page makes this clustering nearly trivial; a skewed one makes it guesswork, which is why deskew runs first.
  • Columns are found by clustering the same boxes on their horizontal position, seeded by the layout hint when one exists. The amount column is identified either from the hint or by detecting the column whose tokens parse as currency.
  • Each (row, column) intersection becomes a cell: the concatenation of the word tokens that fall inside it, in reading order.

Aggregate confidence. Every recovered word carries a recognizer confidence. A cell’s confidence is the aggregate of its words — and the safe aggregate is the minimum, not the mean, because a single misread digit in an amount is a material error even when the surrounding characters are perfect. For a cell of kk words with per-word confidences cic_i:

ccell=min⁡1≤i≤kcic_{\text{cell}} = \min_{1 \le i \le k} c_i

The cell is then tested against a threshold τ\tau. Anything below it is flagged:

route={validatedccell≥τreviewccell<τ\text{route} = \begin{cases} \text{validated} & c_{\text{cell}} \ge \tau \\ \text{review} & c_{\text{cell}} < \tau \end{cases}

Amount cells get a stricter τ\tau than description cells: a garbled word in a line-item description is often still legible in context, but a wrong digit in a recoverable charge propagates straight into a tenant’s reconciliation.

Python Implementation

The implementation below is typed and deterministic in its non-OCR logic, and — as with every stage in the reconciliation system — money is Decimal from the first moment a currency token is recognized, never float. The recognizer emits text; the moment that text is meant to be a dollar amount it is normalized and parsed into a Decimal, so no binary rounding ever touches a figure that will land on a statement. The code references pdf2image for rasterization, Pillow and numpy for preprocessing, and pytesseract for recognition; those libraries handle pixels, while the money path is pure standard-library decimal (docs.python.org/3/library/decimal.html).

from __future__ import annotations

from dataclasses import dataclass, field
from decimal import Decimal, InvalidOperation, ROUND_HALF_UP

import numpy as np
from PIL import Image, ImageFilter

CENTS = Decimal("0.01")


@dataclass(frozen=True)
class WordBox:
    """One recognized word with its pixel geometry and recognizer confidence."""
    text: str
    left: int
    top: int
    width: int
    height: int
    confidence: int  # recognizer score, 0..100

    @property
    def y_center(self) -> float:
        return self.top + self.height / 2

    @property
    def x_left(self) -> float:
        return float(self.left)


@dataclass
class Cell:
    """A reconstructed table cell: the words at one row/column intersection."""
    text: str
    confidence: int          # aggregated (minimum) word confidence
    column: str              # logical column name, e.g. "description" or "amount"


def preprocess(page: Image.Image) -> Image.Image:
    """Deskew, denoise, and binarize a rasterized page before recognition.

    Order matters: rows must be level before clustering, and the image must
    be clean before it is thresholded to pure black-on-white.
    """
    gray = page.convert("L")
    angle = estimate_skew_angle(np.asarray(gray))
    deskewed = gray.rotate(-angle, expand=True, fillcolor=255)
    denoised = deskewed.filter(ImageFilter.MedianFilter(size=3))
    return binarize(np.asarray(denoised))


def estimate_skew_angle(pixels: np.ndarray) -> float:
    """Estimate the dominant text skew in degrees via a projection profile.

    Rotating by a few candidate angles and scoring row-ink variance finds the
    angle at which text lines are most sharply horizontal.
    """
    best_angle, best_score = 0.0, -1.0
    for candidate in np.arange(-5.0, 5.1, 0.5):
        rotated = _rotate_pixels(pixels, candidate)
        row_sums = rotated.sum(axis=1)
        score = float(np.var(row_sums))  # sharpest rows -> highest variance
        if score > best_score:
            best_angle, best_score = float(candidate), score
    return best_angle

The rasterization and recognition call sits at the boundary between pixels and structured data. pytesseract.image_to_data returns exactly the per-word geometry the reconstruction step needs, and this is where each raw row becomes a typed WordBox.

from __future__ import annotations

import pytesseract
from pytesseract import Output


def recognize_words(image: Image.Image, min_confidence: int = 0) -> list[WordBox]:
    """Run OCR and return typed word boxes with pixel geometry.

    ``image_to_data`` yields one record per recognized token, including its
    bounding box and a 0..100 confidence. Empty tokens (whitespace) are
    dropped; -1 confidence rows (structural, not words) are skipped.
    """
    data = pytesseract.image_to_data(image, output_type=Output.DICT)
    boxes: list[WordBox] = []
    for i, raw_text in enumerate(data["text"]):
        text = raw_text.strip()
        conf = int(data["conf"][i])
        if not text or conf < 0:
            continue
        if conf < min_confidence:
            continue
        boxes.append(
            WordBox(
                text=text,
                left=int(data["left"][i]),
                top=int(data["top"][i]),
                width=int(data["width"][i]),
                height=int(data["height"][i]),
                confidence=conf,
            )
        )
    return boxes

Reconstruction groups the boxes into rows by vertical proximity, then parses the amount column into Decimal. The money parser is where OCR’s favorite failure — confusing the thousands separator with the decimal point — is neutralized before it can corrupt a figure.

from __future__ import annotations

import re
from decimal import Decimal, InvalidOperation, ROUND_HALF_UP

CURRENCY_CHARS = re.compile(r"[^0-9.,\-]")


def cluster_rows(boxes: list[WordBox], line_tolerance: float) -> list[list[WordBox]]:
    """Group word boxes into rows by clustering on their vertical centers.

    ``line_tolerance`` should scale with glyph height (roughly half the median
    word height) so the banding is DPI-independent.
    """
    rows: list[list[WordBox]] = []
    for box in sorted(boxes, key=lambda b: b.y_center):
        if rows and abs(box.y_center - rows[-1][0].y_center) <= line_tolerance:
            rows[-1].append(box)
        else:
            rows.append([box])
    # Read each row left-to-right.
    return [sorted(row, key=lambda b: b.x_left) for row in rows]


def parse_ocr_amount(token: str) -> Decimal | None:
    """Parse an OCR'd currency token into an exact Decimal, or None if unusable.

    Handles both 1,234.56 (US) and 1.234,56 (EU) grouping by treating the
    *last* separator as the decimal point and stripping the rest. Money is
    Decimal end to end so no binary rounding touches a recoverable charge.
    """
    cleaned = CURRENCY_CHARS.sub("", token).strip()
    if not cleaned or cleaned in {"-", ".", ","}:
        return None

    last_dot, last_comma = cleaned.rfind("."), cleaned.rfind(",")
    decimal_sep = "." if last_dot > last_comma else ","
    group_sep = "," if decimal_sep == "." else "."

    normalized = cleaned.replace(group_sep, "").replace(decimal_sep, ".")
    try:
        return Decimal(normalized).quantize(CENTS, rounding=ROUND_HALF_UP)
    except InvalidOperation:
        return None


def cell_confidence(words: list[WordBox]) -> int:
    """Aggregate word confidences into a cell score using the minimum.

    A single misread digit in an amount is a material error, so the weakest
    word — not the average — governs whether the cell is trusted.
    """
    return min((w.confidence for w in words), default=0)

Every stage after this point sees the same record shape as the text path: a description, a quantity, and a Decimal amount, plus a confidence integer and an ocr source marker. The reconciliation system therefore reasons about a scanned invoice and a native PDF identically, differing only in how much it trusts each recovered field.

Validation Rules & Edge Cases

The recognizer is a probabilistic component in a domain that demands penny accuracy, so the validation layer is where most of the real engineering lives. Each of the following is a distinct failure the OCR path must catch rather than pass through as a plausible-looking wrong number.

  • Rotated and skewed scans. A page fed sideways through a scanner, or photographed at an angle, breaks row clustering before recognition even matters. Deskew must run first, and the estimated angle should itself be validated: a correction larger than a few degrees, or one the projection profile is unsure about, is a signal to flag the whole page rather than trust a heroically un-rotated result.
  • Faint stamps, watermarks, and signatures. A “PAID” stamp, a wet signature, or a light watermark overlapping the amount column produces stray dark strokes the recognizer folds into adjacent digits. Aggressive-enough denoising and a well-chosen threshold suppress most of it; what survives shows up as depressed word confidence, which is exactly what the gate is for.
  • Comma and period confusion in amounts. This is the OCR error with the largest blast radius. A recognizer that reads 1,204.50 as 1.204,50, or drops a separator to yield 120450, changes a charge by orders of magnitude. The parse_ocr_amount rule — treat the last separator as the decimal point — resolves the common US/EU ambiguity, and a separate range check flags any single line item whose parsed amount is implausible against the vendor’s history.
  • Merged and split columns. When two columns sit close together, recognition can merge a quantity and a rate into one token, or split a single description across phantom columns. Seeding column clustering with the vendor layout hint, and validating that the number of reconstructed columns matches the expected count, catches both. A row whose column count disagrees with the hint is quarantined, not guessed at.
  • Digit substitutions. The recognizer’s classic confusions — 0/O, 1/l/I, 5/S, 8/B — turn a numeric field alphabetic or vice versa. An amount token that still contains letters after cleaning is rejected outright; the systematic repair of these substitutions is the subject of correcting OCR errors in financial invoice data.
  • Line-item totals that must foot. The strongest validation is arithmetic the invoice already contains: quantity times unit rate should equal the extended amount, and the line items should sum to the printed subtotal. A scanned invoice whose recovered numbers do not foot has a recognition error somewhere, and footing pinpoints the offending row even when every individual cell’s confidence looked acceptable.
  • Multi-page scans. A scanned CAM invoice frequently runs several pages, with the table continuing across page breaks and the total landing only on the last sheet. Rows must be stitched across pages in scan order and the running subtotal carried forward, exactly as described in handling multi-page commercial invoices; a mid-table page break must never be read as the end of the invoice.

Integration Points

The OCR stage is a producer that hands its output to the same consumers as the text path, so its integration surface is defined entirely by the shared parsed-line-item contract.

  • Feeds schema validation. Recovered rows flow first to schema validation for parsed expense data, which enforces types, required fields, and value ranges. Because OCR output carries a confidence score the text path does not need, the validation layer can apply a stricter policy to low-confidence fields — treating a recovered amount below the confidence threshold as a hard validation failure that diverts the record to quarantining invalid CAM expense records.
  • Feeds GL code mapping. Validated rows then reach GL code mapping for CAM expenses, which assigns each line item its general-ledger account. The mapping logic neither knows nor cares that the description was recovered by OCR — it operates on the normalized text — which is precisely why the identical-schema contract matters.
  • Diverges from the text branch by design. The image path is the deliberate counterpart to coordinate-aware table extraction with pdfplumber: where that stage reads exact character coordinates from an embedded text layer, this one reconstructs approximate coordinates from recognized pixels. They converge on the same output schema so the pipeline branches once, at the page classifier, and never again.
  • Escalates to human review. Anything the confidence gate flags leaves the automated pipeline entirely and enters a review queue, where an operator sees the cropped source image beside the recovered value and confirms or corrects it. A corrected value re-enters the pipeline at schema validation as though it had been recognized cleanly, and the correction is captured as training signal for the vendor profile.
  • Deep implementation of the table step. The mechanics of turning Tesseract word boxes into a clean table — tolerance tuning, column seeding, and header detection — are treated at length in extracting scanned invoice tables with Tesseract OCR.

Testing & Verification

OCR resists the exact-equality testing the rest of the reconciliation system enjoys, because recognition is probabilistic and a golden run on one machine’s recognizer version may differ slightly on another. The strategy is therefore two-layered: deterministic unit tests for the pure logic, and tolerance-bounded golden-file tests for the recognition path.

The pure functions — amount parsing, row clustering, confidence aggregation — have no randomness and are tested for exact behavior. These are the functions where a bug corrupts money, so they get the strictest assertions.

from decimal import Decimal


def test_parse_amount_us_grouping() -> None:
    assert parse_ocr_amount("$1,204.50") == Decimal("1204.50")


def test_parse_amount_eu_grouping() -> None:
    # Last separator is the comma, so it is the decimal point.
    assert parse_ocr_amount("1.204,50") == Decimal("1204.50")


def test_parse_amount_rejects_alpha_garble() -> None:
    # A stray letter from an O/0 substitution must not silently parse.
    assert parse_ocr_amount("12O4.50") == Decimal("1204.50") or \
        parse_ocr_amount("12O4.50") is None


def test_cell_confidence_uses_minimum() -> None:
    words = [
        WordBox("Landscaping", 0, 0, 10, 10, 96),
        WordBox("1,204.50", 20, 0, 10, 10, 71),
    ]
    # The weakest word governs the cell, not the average.
    assert cell_confidence(words) == 71

The recognition path is tested against a small library of fixtures: real scanned CAM invoices — a janitorial bill, a snow-removal ticket, a utility statement — each stored as a source image beside a hand-verified expected result. Each fixture is run through the full pipeline and compared to its golden file, but with tolerances calibrated to OCR’s nature: description fields are compared with a character-error-rate tolerance rather than exact equality, while amount fields, being the numbers that must be exactly right, are asserted for exact Decimal equality — a recovered charge is either correct to the cent or it is a test failure. A fixture whose amounts no longer match is a regression; a fixture whose descriptions drift beyond the error-rate tolerance is a recognizer or preprocessing regression worth investigating. Keeping deliberately hard fixtures — a skewed page, a stamped page, a multi-column bill — in the suite ensures the preprocessing and reconstruction steps are exercised, not just the easy cases.

Frequently Asked Questions

When should the pipeline use OCR instead of text extraction? Only when there is no embedded text layer to read. A native digital PDF carries exact character positions, and the coordinate-aware table extraction with pdfplumber path reads them precisely; OCR is a lossy reconstruction that should never touch a document a text extractor can handle. The page classifier makes this call automatically by attempting text extraction first and routing to OCR only when the page yields no usable characters — a scanned image, a photograph of paper, or an image-only PDF export. Running OCR on a clean invoice manufactures recognition errors where none existed, so the rule is strict: image-only in, OCR; text-native, never.

What resolution does a scanned invoice need for reliable recognition? At least 300 dots per inch, and 400 DPI for invoices with small type such as utility and tax bills. Recognition accuracy depends on how many pixels land on each character stroke, not on the page’s physical size, so a document scanned below 200 DPI degrades sharply as thin strokes merge and serifs vanish. When the source is a photograph rather than a formal scan, the effective resolution on the text matters more than the camera’s megapixel count — a distant or blurry capture of small print will underperform a modest but sharp scan. Pages below the resolution floor are up-flagged for rescanning rather than pushed through and trusted.

How does the system handle handwritten annotations on a scanned invoice? It does not try to recognize them as data. Handwriting — a manager’s initials, a scribbled approval, a penned correction — is outside the reliable range of print-oriented OCR, and attempting to parse it introduces noise. The engine treats handwriting the same way it treats stamps and signatures: as visual clutter to suppress during denoising, and, where a handwritten mark overlaps a printed value, as a confidence-depressing event that routes the affected cell to human review. A person confirms what the annotation means; the recognizer never guesses at a handwritten number that will hit the ledger.

How is the confidence threshold chosen, and can one threshold serve every field? No single threshold fits every field, because the cost of an error differs by column. Amount cells get the strictest threshold, since a wrong digit in a recoverable charge flows straight into a tenant’s reconciliation, while description cells tolerate a lower bar because context usually makes a slightly garbled description legible. Thresholds are tuned empirically against the fixture library: set the amount threshold where it catches the recognition errors observed in testing without flooding the review queue with correct-but-cautious cells. Because the cell score is the minimum of its word confidences, the threshold governs the weakest character in the cell, which is the right thing to protect for money.

Where This Fits

OCR for scanned invoices is the reconciliation system’s guarantee that a vendor’s choice to send an image instead of a digital file does not become a gap in the ledger. By classifying image-only pages, rasterizing them at a resolution the recognizer can read, cleaning them into upright high-contrast binaries, recovering words with pixel coordinates, rebuilding the invoice table, and gating every cell on confidence, the stage turns a flattened scan into the same structured, Decimal-precise line items the coordinate-aware table extraction with pdfplumber path produces from native PDFs. It hands its trusted output to schema validation for parsed expense data and then to GL code mapping for CAM expenses, routes everything it cannot vouch for to a human, and in doing so keeps the entire automated invoice parsing and data ingestion pipeline blind to the difference between a pixel and a glyph.