Extracting Scanned Invoice Tables with Tesseract OCR

Scanned CAM invoices arrive as flat images with no table structure, so pulling their line items means rebuilding rows and columns from the raw word boxes Tesseract reports, one pixel-anchored token at a time. This page is the extraction procedure for image-only bills inside OCR for scanned CAM invoices, the image branch of the broader Automated Invoice Parsing and Data Ingestion pipeline. A coordinate-aware PDF reader sees nothing in these files — there is no embedded text layer, only a raster of light and dark pixels — so the whole burden of recovering “Parking lot sweeping … $2,000.00” falls on optical character recognition plus a reconstruction step that turns a scatter of recognized words back into a grid a reconciliation engine can trust. Get the reconstruction wrong and an amount lands in the description column, or two vendor charges collapse into one line, and the defect rides silently downstream until a tenant disputes the year-end statement.

Context & When to Use This Approach #

Reach for this procedure the moment a CAM bill has no readable text layer — when a coordinate-aware PDF reader returns empty pages even though the invoice plainly shows a vendor table. The concrete triggers are recognizable in any commercial-real-estate feed:

• A janitorial or landscaping contractor prints, signs, and scans an invoice, then emails a flattened image-only PDF with no embedded characters.
• A property manager photographs a paper snow-removal ticket on a phone and drops the JPEG into the intake folder.
• A utility or municipality re-scans a decades-old billing template whose text was never digital to begin with.
• A vendor “prints to PDF” from a fax cover sheet, producing a document whose visible numbers are pixels, not glyphs.

When a document does carry a real text layer, do not run OCR at all — extract the coordinates directly, which is faster and lossless, and belongs to the pdfplumber route rather than here. This page is specifically for the raster case, where the amount $2,000.00 exists only as dark pixels and must be recognized, positioned, and slotted back into a column before it can be reconciled. The reconstruction matters as much as the recognition: Tesseract will happily read every token on the page, but it returns them as an unordered bag of word boxes with pixel coordinates and no notion of which belong to the same line or the same column. A scanned CAM ledger is a table precisely because of that two-dimensional structure, so the work here is to invert the flattening — to take the coordinates Tesseract preserves and rebuild the rows and columns the scan destroyed.

Everything below keeps monetary values in decimal.Decimal from the first parse onward, because OCR output is exactly where a stray float does the most damage: a recognizer that reports 1,860.00 as the string 1860.00 must be summed against dozens of sibling lines to tie out to a pool total, and binary rounding drift on top of recognition noise makes a failed reconciliation impossible to attribute. Recognized amounts and their confidence scores travel together so that the downstream confidence gate — and the human review it feeds — can act on the weakest cell in each row.

Step-by-Step Implementation #

The pipeline is five stages: rasterize the PDF to images, clean each image so Tesseract reads it reliably, run image_to_data to recover word boxes with coordinates, group those boxes back into rows and columns, then parse the amount cell to Decimal and emit a typed row. Each stage is small and independently testable, which matters because scanned input is noisy and you will tune thresholds per vendor template.

Step 1 — Rasterize the scanned PDF to page images #

A scanned PDF is a container for page images; the first job is to get those images out at a resolution Tesseract can read. Use pdf2image to render each page to a Pillow image at 300 DPI, the practical floor for the 8-to-10-point type on a dense CAM ledger. Lower resolutions blur adjacent digits — a 3 and an 8 fuse — and inflate the recognition-error rate before any structure work begins.

from __future__ import annotations

from pathlib import Path

from pdf2image import convert_from_path
from PIL import Image


def rasterize_invoice(pdf_path: Path, dpi: int = 300) -> list[Image.Image]:
    """Render each page of a scanned invoice to a Pillow image at OCR-grade DPI.

    300 DPI is the practical floor for the small type on a dense CAM ledger;
    below it, adjacent digits blur together and recognition accuracy collapses.
    """
    return convert_from_path(str(pdf_path), dpi=dpi)

Step 2 — Preprocess each page for reliable recognition #

Raw scans are skewed, speckled, and unevenly lit. Convert to grayscale, threshold to a clean bi-level image with Otsu’s method, and correct the page skew before OCR — deskewing is the single highest-leverage step for table work, because Tesseract’s row grouping and the coordinate grouping later both assume text lines are horizontal. A page tilted two degrees smears every row’s vertical band into its neighbors.

from __future__ import annotations

import numpy as np
from PIL import Image


def _otsu_threshold(arr: np.ndarray) -> int:
    """Compute a global threshold that best separates ink from paper."""
    hist, _ = np.histogram(arr, bins=256, range=(0, 256))
    total = arr.size
    sum_all = float(np.dot(np.arange(256), hist))
    sum_bg, weight_bg, best_var, best_t = 0.0, 0.0, -1.0, 127
    for t in range(256):
        weight_bg += hist[t]
        if weight_bg == 0:
            continue
        weight_fg = total - weight_bg
        if weight_fg == 0:
            break
        sum_bg += t * hist[t]
        mean_bg = sum_bg / weight_bg
        mean_fg = (sum_all - sum_bg) / weight_fg
        between = weight_bg * weight_fg * (mean_bg - mean_fg) ** 2
        if between > best_var:
            best_var, best_t = between, t
    return best_t


def _estimate_skew(binary: np.ndarray, limit: float = 6.0, step: float = 0.5) -> float:
    """Find the rotation (degrees) that best aligns text rows horizontally.

    Rotating a well-aligned page concentrates ink into sharp horizontal bands,
    which maximizes the variance of the row-wise ink projection.
    """
    base = Image.fromarray(binary)
    best_angle, best_score = 0.0, -1.0
    angle = -limit
    while angle <= limit:
        rotated = np.asarray(base.rotate(angle, resample=Image.NEAREST, fillcolor=255))
        projection = (255 - rotated).sum(axis=1).astype(float)
        score = float(np.var(projection))
        if score > best_score:
            best_score, best_angle = score, angle
        angle += step
    return best_angle


def preprocess(page: Image.Image) -> Image.Image:
    """Grayscale, Otsu-threshold, and deskew a page before OCR."""
    arr = np.asarray(page.convert("L"))
    binary = np.where(arr > _otsu_threshold(arr), 255, 0).astype(np.uint8)
    angle = _estimate_skew(binary)
    deskewed = Image.fromarray(binary).rotate(
        -angle, resample=Image.BICUBIC, fillcolor=255, expand=False
    )
    return deskewed

Step 3 — Read word boxes with Tesseract image_to_data #

Run Tesseract in data mode via pytesseract.image_to_data, which returns not just text but a parallel array of bounding boxes and per-token confidence. Wrap each recognized token in a typed WordBox and discard blanks and low-confidence noise up front, so the grouping stage works only with tokens the recognizer actually stands behind. Keeping conf on every box is what lets the reconciliation system later route a shaky cell to review instead of trusting it blindly.

from __future__ import annotations

from dataclasses import dataclass

import pytesseract
from PIL import Image


@dataclass(frozen=True)
class WordBox:
    """One recognized token with its pixel bounding box and OCR confidence."""

    text: str
    left: int
    top: int
    width: int
    height: int
    conf: float

    @property
    def x_center(self) -> float:
        return self.left + self.width / 2

    @property
    def y_center(self) -> float:
        return self.top + self.height / 2


def read_word_boxes(page: Image.Image, min_conf: float = 40.0) -> list[WordBox]:
    """Recover word boxes from a preprocessed page, dropping blanks and noise."""
    data = pytesseract.image_to_data(page, output_type=pytesseract.Output.DICT)
    boxes: list[WordBox] = []
    for i, raw in enumerate(data["text"]):
        token = raw.strip()
        conf = float(data["conf"][i])
        if not token or conf < min_conf:
            continue  # empty cell or a token the recognizer is unsure of
        boxes.append(
            WordBox(
                text=token,
                left=int(data["left"][i]),
                top=int(data["top"][i]),
                width=int(data["width"][i]),
                height=int(data["height"][i]),
                conf=conf,
            )
        )
    return boxes

Step 4 — Group word boxes into rows and columns #

This is the reconstruction. First band the boxes into rows by their vertical centers: two tokens belong to the same line when their y_center values sit within a tolerance derived from the text height. Then infer column seams from the horizontal gaps between token centers — a gap much larger than the typical inter-word gap marks the boundary between two columns. Writing the sorted x-centers as $c\_1 \le c\_2 \le \dots \le c\_n$, with adjacent gaps $g\_i = c\_{i+1} - c\_i$ and median gap $\tilde{g}$, a seam sits between two tokens when

where $r$ is a gap ratio around 1.8. This adapts to each vendor’s layout instead of hard-coding pixel columns, so the same code handles a two-column landscaping bill and a five-column utility ledger.

from __future__ import annotations

from typing import Sequence


def group_rows(boxes: Sequence[WordBox], row_tol: float = 10.0) -> list[list[WordBox]]:
    """Band word boxes whose vertical centers fall within row_tol into table rows."""
    rows: list[list[WordBox]] = []
    for box in sorted(boxes, key=lambda b: b.y_center):
        for row in rows:
            if abs(row[0].y_center - box.y_center) <= row_tol:
                row.append(box)
                break
        else:
            rows.append([box])
    for row in rows:
        row.sort(key=lambda b: b.left)  # restore left-to-right reading order
    return rows


def infer_column_edges(rows: Sequence[Sequence[WordBox]], gap_ratio: float = 1.8) -> list[float]:
    """Infer column boundaries from the x-centers of every recovered token.

    A gap between adjacent x-centers that is large relative to the median gap
    marks a seam between two columns, so the layout is learned, not hard-coded.
    """
    centers = sorted(b.x_center for row in rows for b in row)
    if len(centers) < 2:
        return []
    gaps = [b - a for a, b in zip(centers, centers[1:])]
    median_gap = sorted(gaps)[len(gaps) // 2] or 1.0
    edges: list[float] = []
    for a, b, gap in zip(centers, centers[1:], gaps):
        if gap >= gap_ratio * median_gap:
            edges.append((a + b) / 2)
    return edges


def assign_column(box: WordBox, edges: Sequence[float]) -> int:
    """Return the 0-based column index for a box given inferred column edges."""
    index = 0
    for edge in edges:
        if box.x_center > edge:
            index += 1
    return index

Step 5 — Parse the amount cell to Decimal and emit typed rows #

With rows and columns rebuilt, bucket each row’s tokens into cells, treat the rightmost cell that parses as money as the recoverable amount, and join the remaining left-hand cells into the description. Every amount is coerced through one function built on Python’s decimal module — stripping currency symbols and thousands separators, honoring accounting-style parenthesized negatives, and quantizing to cents — so no float ever touches the money path. The row carries the minimum token confidence, giving the confidence gate a single scalar to threshold on.

from __future__ import annotations

from dataclasses import dataclass
from decimal import Decimal, InvalidOperation
from typing import Optional, Sequence

CENT = Decimal("0.01")


@dataclass
class InvoiceRow:
    """One reconstructed table row: a description and its recoverable amount."""

    description: str
    amount: Decimal
    min_conf: float  # weakest OCR confidence across the row's tokens


def parse_amount(token: str) -> Optional[Decimal]:
    """Parse an OCR money token to a two-place Decimal, or None if it is not money."""
    cleaned = token.replace("$", "").replace(",", "").replace(" ", "").strip()
    if cleaned.startswith("(") and cleaned.endswith(")"):
        cleaned = "-" + cleaned[1:-1]  # accounting-style negative, e.g. (125.00)
    if not cleaned:
        return None
    try:
        return Decimal(cleaned).quantize(CENT)
    except InvalidOperation:
        return None


def emit_rows(rows: Sequence[Sequence[WordBox]], edges: Sequence[float]) -> list[InvoiceRow]:
    """Turn grouped word boxes into typed InvoiceRow records.

    The rightmost cell that parses as money is the amount; the remaining
    left-hand cells join into the expense description.
    """
    emitted: list[InvoiceRow] = []
    column_count = len(edges) + 1
    for row in rows:
        buckets: list[list[WordBox]] = [[] for _ in range(column_count)]
        for box in row:
            buckets[assign_column(box, edges)].append(box)
        cell_text = [" ".join(b.text for b in bucket) for bucket in buckets]
        amount: Optional[Decimal] = None
        amount_col = -1
        for col in range(column_count - 1, -1, -1):
            candidate = parse_amount(cell_text[col])
            if candidate is not None:
                amount, amount_col = candidate, col
                break
        if amount is None:
            continue  # header, subtotal label, or a non-line row
        description = " ".join(
            cell_text[col] for col in range(column_count) if col != amount_col
        ).strip()
        emitted.append(
            InvoiceRow(description=description, amount=amount, min_conf=min(b.conf for b in row))
        )
    return emitted

Gotchas & Known Limitations #

Table reconstruction from OCR fails in predictable, template-specific ways. Treat this as a pre-flight checklist before trusting the parser against a new vendor’s scan:

• Under-resolution recognition errors. Below 300 DPI, 0/8, 1/7, and 5/6 fuse in the amount column. Rasterize at 300 DPI or higher and confirm the recognition-error rate on a known page before scaling to a batch.
• Uncorrected skew smears the rows. A page tilted even one or two degrees pushes a token’s y_center into the neighboring line’s band, so group_rows merges two invoice lines into one. Deskew first, and widen row_tol only as a last resort.
• A tight min_conf deletes real data. Filtering confidence too aggressively drops faint-but-correct tokens and leaves a row with no amount cell. Keep the threshold modest here and let the downstream confidence gate, not the reader, make the accept-or-review call.
• Multi-line descriptions split into phantom rows. A description that wraps to a second visual line becomes a second grouped row with no amount. Merge an amount-less row into the one above it before emitting, rather than discarding it.
• Column seams collapse on sparse pages. infer_column_edges needs enough tokens for a stable median gap; a two-line invoice can misplace the seam. Fall back to a fixed column model when a page yields too few tokens to learn from.
• Recognized digits still need semantic checks. Even a clean, high-confidence read can be a valid-looking wrong number — a transposed pair, a dropped decimal. That correction belongs to correcting OCR errors in financial invoice data, which cross-checks amounts against column and pool totals.
• float anywhere in the amount path. A single float(cell) reintroduces binary drift that summing a hundred recovered rows will amplify past a one-cent tie-out. Keep every amount in Decimal from parse_amount onward.

Verification #

Prove the reconstruction on fixtures whose grouped shape and amounts you know, asserting on the two properties OCR table work most often violates: that word boxes band into the right number of rows, and that the amount column parses to the exact expected Decimal. Because the grouping and parsing functions are pure and take plain WordBox values, you can test them without invoking Tesseract at all — feed synthetic boxes that mimic a two-column scan.

from decimal import Decimal


def _sample_boxes() -> list[WordBox]:
    """Synthetic word boxes standing in for a two-column scanned CAM invoice."""
    return [
        WordBox("Landscaping", 40, 96, 104, 24, 96.0),
        WordBox("4,120.00", 222, 100, 86, 24, 94.0),
        WordBox("Snow", 40, 150, 58, 24, 91.0),
        WordBox("removal", 112, 152, 94, 24, 93.0),
        WordBox("2,000.00", 222, 150, 86, 24, 95.0),
        WordBox("HVAC", 40, 206, 64, 24, 92.0),
        WordBox("1,860.00", 222, 208, 86, 24, 90.0),
    ]


def test_word_boxes_group_into_expected_rows() -> None:
    rows = group_rows(_sample_boxes(), row_tol=10.0)
    edges = infer_column_edges(rows)
    out = emit_rows(rows, edges)
    assert len(out) == 3
    assert out[1].description == "Snow removal"
    assert out[1].amount == Decimal("2000.00")


def test_amounts_sum_to_pool_total() -> None:
    rows = group_rows(_sample_boxes())
    edges = infer_column_edges(rows)
    total = sum((r.amount for r in emit_rows(rows, edges)), Decimal("0.00"))
    # The recovered lines must tie out to the invoice's stated pool total.
    assert total == Decimal("7980.00")

The tie-out assertion is the one that matters most: a reconstructed table whose line amounts sum to the invoice’s stated total, row for row, is one you can hand upward with confidence. A mismatch localizes the defect to extraction — a merged row, a misplaced seam, a mis-read digit — rather than letting it masquerade as an allocation error three stages later. Once the recovered rows tie out, pass each InvoiceRow into schema validation for parsed expense data for typed contract enforcement before it reaches the ledger, and treat a rising reconstruction-failure rate for one vendor as an early signal that their scan template changed.

Extracting the table is only the recovery half of image-based ingestion inside OCR for scanned CAM invoices; once you hold typed rows with confidence scores, continue to correcting OCR errors in financial invoice data to catch the valid-looking wrong numbers that recognition alone will never flag.

Related #

• OCR for scanned CAM invoices — the parent topic this page implements, covering rasterization, confidence gating, and review routing end to end.
• Correcting OCR errors in financial invoice data — the next stage, which validates recovered amounts against column and pool totals and auto-fixes common misreads.
• Schema validation for parsed expense data — the typed contract every reconstructed InvoiceRow must satisfy before it reaches the ledger.