CAM Reconciliation

Automating Vendor Invoice Classification

A CAM vendor writes whatever they want on a line item — HVAC PM Q3, Cmn Area Sweep, Roof mem. repl, lot re-stripe — and your reconciliation needs every one of those strings resolved to a single general-ledger category before a dollar can accumulate into a recoverable pool. Vendor invoice classification is the specific job of turning that free-text description into a canonical account code with an attached confidence score, and it is the fallback path that GL code mapping for CAM expenses hands off to whenever an exact dictionary lookup misses. This page builds that classifier end to end: a normalization step, a deterministic rule layer, a fuzzy token_set_ratio fallback, a vendor-history prior, and a hard confidence gate that routes anything uncertain to human review instead of guessing. It is one focused component of the wider Automated Invoice Parsing & Data Ingestion pipeline.

Vendor-invoice classification cascade: exact match, fuzzy-plus-history blend, and a confidence gate A normalized vendor description first tries an exact synonym match (auto-assigned at confidence 1.00 on a hit). A miss falls through to fuzzy token-set matching, whose score is blended with a vendor posting-history prior as 0.75 times fuzzy plus 0.25 times prior. A gate auto-assigns items scoring 0.85 or higher and routes everything below to the PENDING_REVIEW human queue. Raw vendor description Normalize text expand CRE abbreviations Fuzzy match token-set ratio → score Blend confidence Vendor posting history prior 0.75·fuzzy + 0.25·prior Exact synonym match? confidence ≥ 0.85? AUTO-ASSIGN AUTO-ASSIGN PENDING_REVIEW method=exact · 1.00 method=fuzzy | vendor → human review queue hit miss yes no

Context & When to Use This Approach

Reach for an automated classifier the moment a portfolio outgrows a hand-maintained lookup table — typically when a single month-end brings in hundreds of distinct vendor descriptions across dozens of properties and the same service arrives spelled six different ways. A pure exact-match dictionary is correct but brittle: it maps common area maintenance perfectly and then drops cmn area maint and CAM - sweep/blow straight into a review queue that no accountant has time to clear. A classifier that normalizes text and falls back to fuzzy similarity absorbs that vendor-side inconsistency without inventing categories.

The scope here is deliberately narrow. This component does not parse PDFs — that is handled upstream by coordinate-aware table extraction with pdfplumber — and it does not decide recoverability, which is a lease fact read from your lease abstraction database. It consumes an already-validated line item and emits a category plus a confidence number. Use it specifically when descriptions are short, noisy, and drawn from a stable finite set of expense types; do not reach for a heavyweight NLP model when a 40-entry synonym table and a fuzzy ratio will classify 95% of your traffic deterministically and explainably.

Step-by-Step Implementation

The classifier is built in five steps. Every monetary value flows through the decimal module — binary float cannot represent cent amounts exactly, and drift across thousands of line items produces pools that fail to tie out.

Step 1 — Normalize the raw description

Vendor strings carry casing, punctuation, and abbreviation noise that defeats naive equality. Collapse each description to a canonical form first, so CAM — Sweep/Blow and cam sweep blow become the same token stream.

from __future__ import annotations

import re


_WS = re.compile(r"\s+")
_NON_ALNUM = re.compile(r"[^a-z0-9 ]+")

# Domain abbreviations that recur across CRE vendor invoices.
_ABBREVIATIONS: dict[str, str] = {
    "cmn": "common",
    "maint": "maintenance",
    "mgmt": "management",
    "repl": "replacement",
    "pm": "preventative maintenance",
    "hvac": "hvac",
    "lndscp": "landscaping",
    "jan": "janitorial",
}


def normalise(description: str) -> str:
    """Lower-case, strip punctuation, and expand CRE abbreviations."""
    text = _NON_ALNUM.sub(" ", description.lower())
    tokens = [_ABBREVIATIONS.get(tok, tok) for tok in _WS.sub(" ", text).split()]
    return " ".join(tokens)

Expanding abbreviations before matching is what lets HVAC PM and Preventative HVAC Maintenance collapse onto the same key instead of scoring as a weak fuzzy pair.

Step 2 — Assign a canonical category with a deterministic ruleset

The first classification pass is an exact lookup against a curated synonym table keyed on normalized text. This path is reproducible, carries full confidence, and should cover the large majority of recurring traffic. The categories themselves are not invented here — they mirror the ones set in defining CAM expense categories in commercial leases.

from decimal import Decimal
from enum import Enum

from pydantic import BaseModel, Field, field_validator


class GLCategory(str, Enum):
    UTILITIES = "6100"
    JANITORIAL = "6200"
    LANDSCAPING = "6300"
    SECURITY = "6400"
    REPAIRS = "6500"
    MANAGEMENT_FEE = "6900"
    CAPITAL_IMPROVEMENT = "1700"


PENDING_REVIEW = "PENDING_REVIEW"

# Normalized description -> canonical category. Owned by accounting, versioned.
SYNONYMS: dict[str, GLCategory] = {
    normalise(k): v
    for k, v in {
        "common area maintenance sweep": GLCategory.JANITORIAL,
        "parking lot cleaning": GLCategory.JANITORIAL,
        "hvac pm": GLCategory.REPAIRS,
        "landscaping maintenance": GLCategory.LANDSCAPING,
        "security patrol": GLCategory.SECURITY,
        "electricity": GLCategory.UTILITIES,
        "roof membrane replacement": GLCategory.CAPITAL_IMPROVEMENT,
        "management fee": GLCategory.MANAGEMENT_FEE,
    }.items()
}


class LineItem(BaseModel):
    """A validated invoice line item entering classification."""

    description: str = Field(min_length=1)
    amount: Decimal
    vendor_id: str
    property_id: str

    @field_validator("amount")
    @classmethod
    def _cents(cls, v: Decimal) -> Decimal:
        return v.quantize(Decimal("0.01"))


class Classification(BaseModel):
    """The classifier's output contract, one per line item."""

    category: str          # a GLCategory value or PENDING_REVIEW
    confidence: Decimal
    method: str            # "exact" | "fuzzy" | "vendor" | "review"

Step 3 — Fall back to fuzzy similarity for unseen phrasings

When no synonym matches exactly, compare the normalized description against every synonym key with a token-set ratio, which ignores word order and duplicated tokens — exactly the noise CRE vendors introduce. The rapidfuzz scorer returns 0–100; rescale it into a Decimal in [0, 1] so it composes with the other signals.

from rapidfuzz import fuzz


def best_fuzzy(norm_desc: str) -> tuple[GLCategory | None, Decimal]:
    """Return the closest synonym category and its 0..1 similarity."""
    best_cat: GLCategory | None = None
    best_score = Decimal("0")
    for key, category in SYNONYMS.items():
        raw = fuzz.token_set_ratio(norm_desc, key)  # 0..100
        score = (Decimal(str(raw)) / Decimal("100")).quantize(Decimal("0.01"))
        if score > best_score:
            best_cat, best_score = category, score
    return best_cat, best_score

Step 4 — Blend the fuzzy score with a vendor-history prior

A landscaping vendor that has posted to 6300 on its last forty invoices is strong evidence even when the description is garbled. Fold that history in as a discounted prior so a weak fuzzy hit backed by consistent vendor behaviour can still clear the bar, while a novel vendor with a garbled string cannot.

from typing import Mapping

W_FUZZY = Decimal("0.75")
W_HISTORY = Decimal("0.25")
THRESHOLD = Decimal("0.85")


def history_share(
    vendor_id: str,
    category: GLCategory,
    vendor_history: Mapping[str, Mapping[GLCategory, int]],
) -> Decimal:
    """Fraction of this vendor's prior invoices posted to `category`."""
    counts = vendor_history.get(vendor_id, {})
    total = sum(counts.values())
    if not total:
        return Decimal("0")
    return (Decimal(counts.get(category, 0)) / Decimal(total)).quantize(Decimal("0.01"))

Step 5 — Classify, gating everything uncertain to review

The public entry point runs exact match first, then the blended fuzzy-plus-history path, and refuses to assign a real category below the threshold. A sub-threshold item becomes PENDING_REVIEW — never a guess.

def classify(
    item: LineItem,
    vendor_history: Mapping[str, Mapping[GLCategory, int]],
) -> Classification:
    norm = normalise(item.description)

    # 1. Exact synonym match -> full confidence, fully explainable.
    exact = SYNONYMS.get(norm)
    if exact is not None:
        return Classification(category=exact.value, confidence=Decimal("1.00"), method="exact")

    # 2. Blend fuzzy similarity with the vendor's posting history.
    candidate, fuzzy_score = best_fuzzy(norm)
    if candidate is None:
        return Classification(category=PENDING_REVIEW, confidence=Decimal("0.00"), method="review")

    prior = history_share(item.vendor_id, candidate, vendor_history)
    confidence = (W_FUZZY * fuzzy_score + W_HISTORY * prior).quantize(Decimal("0.01"))

    if confidence >= THRESHOLD:
        method = "vendor" if prior > fuzzy_score else "fuzzy"
        return Classification(category=candidate.value, confidence=confidence, method=method)

    # 3. Below threshold -> route to review, never fabricate a category.
    return Classification(category=PENDING_REVIEW, confidence=confidence, method="review")

For the residual tail of genuinely novel descriptions that no synonym resembles, an optional NLP classifier can be slotted in behind Step 5, but its prediction must pass through the same confidence gate and review queue — the design never lets a model post a category an auditor cannot trace.

Gotchas & Known Limitations

  • Token-set ratio erases quantity and order. 3 month landscaping and landscaping score near-identically, which is fine for category but dangerous if you ever lean on the description for amount validation. Keep classification and amount checks separate.
  • A blended line item must not be classified. A single line reading Jan–Mar landscaping + irrigation repair spans two categories; averaging it corrupts both pools. Detect conjunctions (+, &, and) after normalization and force those items to review for an accountant to split.
  • Capital-versus-operating is an amount decision, not a text one. Replace roof membrane classifies cleanly, but whether it is expensed 6500 or capitalized 1700 depends on the property’s lease-defined capitalization threshold. Flag any repair-coded line above that threshold for capitalization review rather than trusting the string alone.
  • Vendor history can entrench a past mistake. If a vendor was historically miscoded, the prior will keep reinforcing the wrong category. Rebuild the history table only from accountant-approved postings, never from the classifier’s own auto-assignments.
  • Confidence clustering at 0.84–0.86 signals a dictionary gap, not a model problem. The fix is to promote the recurring review-queue descriptions into the synonym table so they move to the deterministic path permanently — the same tuning discipline described in threshold tuning for allocation accuracy.
  • rapidfuzz scores are integers 0–100. Forgetting to rescale before blending with the [0, 1] history prior silently pins every confidence above the threshold. The rescale in Step 3 is load-bearing.

Verification

Classification correctness is pinned with table-driven fixtures that map known descriptions to expected categories, plus an invariant that no sub-threshold item ever receives a real code. Because confidence is Decimal quantized to two places, equality assertions are exact and there is no floating-point tolerance to reason about.

from decimal import Decimal


def test_exact_match_is_full_confidence() -> None:
    item = LineItem(
        description="CAM — Sweep/Blow",  # normalises to a known synonym
        amount=Decimal("1200.00"),
        vendor_id="v-1",
        property_id="prop-1",
    )
    result = classify(item, vendor_history={})
    assert result.method == "exact"
    assert result.confidence == Decimal("1.00")


def test_history_prior_lifts_a_weak_fuzzy_hit() -> None:
    history = {"v-9": {GLCategory.LANDSCAPING: 40}}
    item = LineItem(
        description="lndscp maintenence",  # misspelled, still a landscaping job
        amount=Decimal("875.00"),
        vendor_id="v-9",
        property_id="prop-1",
    )
    result = classify(item, vendor_history=history)
    assert result.category == GLCategory.LANDSCAPING.value


def test_novel_vendor_and_string_routes_to_review() -> None:
    item = LineItem(
        description="zzz unrecognisable vendor note",
        amount=Decimal("500.00"),
        vendor_id="brand-new-vendor",
        property_id="prop-1",
    )
    result = classify(item, vendor_history={})
    assert result.category == PENDING_REVIEW

Beyond unit fixtures, cross-check the classifier against a held-out month of accountant-approved postings and track the exact-match rate over time: a healthy classifier drives more traffic onto the deterministic path each quarter as recurring descriptions are promoted into the synonym table. A rising review-queue volume is the early warning that a new vendor or a renamed service has outrun the dictionary.

This classifier feeds its category and confidence straight back into GL code mapping for CAM expenses, which layers the lease-driven recoverable-versus-excluded split on top; if your upstream records are not yet type-safe, wire in the validation gate from building a CAM data validation layer before this stage ever runs.