Retrying Failed Invoice Batches with Idempotency

Re-running a failed CAM invoice batch is only safe when the second attempt cannot double-post a recovery that the first attempt already committed to the general ledger, and that guarantee comes from a content-addressed idempotency key rather than from hoping the retry never overlaps the original. This is the fault-tolerance layer of async batch processing for high-volume invoices: the async worker pool gives you throughput, but a distributed run over an unreliable ERP will always redeliver some units of work, and without a deduplication gate that redelivery becomes a duplicated tenant charge. The pattern below turns an unavoidable at-least-once delivery model into an exactly-once effect on the ledger, so a batch can be retried aggressively, replayed after a crash, or resubmitted by a nervous accountant with no risk of applying the same recovery twice.

Context & When to Use This Approach #

The moment invoice processing moves from a single synchronous script to a fan-out over workers, the delivery guarantee silently changes. A worker that posts a batch to the ERP and then dies before recording success will, on restart, see the batch as unfinished and post it again. A network partition can make a successful write look failed. A message broker that promises “at least once” will, by design, occasionally deliver the same batch twice. None of these are exotic — they are the normal weather of distributed processing — and every one of them ends the same way without a guard: a tenant billed twice for the same $12,400.00 chiller repair.

Reach for the idempotent-retry pattern on this page when any of the following are true:

• Batches are applied to a shared side-effecting system — a general ledger, an ERP posting API, a tenant-billing table — where a duplicate write is a real financial event, not just wasted work.
• Your runner already fans out with asyncio and you need crashes and redeliveries to be recoverable without a human first proving that the original attempt did not partially complete.
• Accountants must be able to safely resubmit a batch that failed mid-close without opening a spreadsheet to check what already posted.
• Failed work must not silently vanish: a batch that can never succeed has to land somewhere an operator can inspect it, which is where this pattern hands off to quarantining invalid CAM expense records.

The distinction that makes the whole design work is between delivery and effect. You cannot make an unreliable transport deliver each batch exactly once — that is a distributed-systems impossibility in the general case. What you can do is make the effect exactly once: for any number of deliveries $d \ge 1$, the committed side effects equal $\min(d, 1) = 1$. The idempotency key is the identity that lets a redelivery recognize itself and decline to act. Records that survive this stage flow on to GL code mapping for CAM expenses, so a duplicate that slips past here is a defect every downstream posting inherits.

Step-by-Step Implementation #

The mechanism has four moving parts: a key derived from the content of the work (not from a timestamp or a random UUID), a store that remembers which keys have already taken effect, a retry loop that backs off on transient errors, and a quarantine exit for work that never succeeds. Every monetary value stays in Python’s decimal module — never float — because the key itself hashes the amounts, and binary drift would make two economically identical batches hash differently. Keep that constraint in mind as you read: the type annotations below are not decoration but a contract, since a single float leaking into a line item would quietly change the hash and defeat every guarantee the rest of the design depends on.

Step 1 — Derive the idempotency key from batch content #

The key must be a pure function of the recoverable work, so that the same batch always produces the same key no matter when or where it is retried. Hash a canonical serialization of the batch with hashlib: sort the line items, quantize every amount to cents as a Decimal string, and digest the result. Two batches that recover the same amounts against the same GL codes for the same period collapse to one key; a single changed cent produces a different key.

from __future__ import annotations

import hashlib
import json
from dataclasses import dataclass
from decimal import Decimal, ROUND_HALF_UP


@dataclass(frozen=True)
class InvoiceBatch:
    """A retryable unit of work: one property's recoveries for one period."""

    property_id: str
    period: str                                   # e.g. "2026-06"
    line_items: tuple[tuple[str, Decimal], ...]   # (gl_code, recoverable_amount)


def _canonical(batch: InvoiceBatch) -> str:
    """Serialize a batch deterministically so equal work hashes equally.

    Amounts are quantized to cents as Decimal strings — never float — so
    1200, 1200.00 and 1200.004 all canonicalize to the same "1200.00".
    """
    items = [
        [gl, str(amount.quantize(Decimal("0.01"), rounding=ROUND_HALF_UP))]
        for gl, amount in sorted(batch.line_items)
    ]
    payload = {"property_id": batch.property_id, "period": batch.period, "items": items}
    return json.dumps(payload, sort_keys=True, separators=(",", ":"))


def idempotency_key(batch: InvoiceBatch) -> str:
    """Content-addressed identity: the same recoverable work maps to the same key."""
    digest = hashlib.sha256(_canonical(batch).encode("utf-8")).hexdigest()
    return f"cam-batch:{batch.property_id}:{batch.period}:{digest[:16]}"

Content addressing is what makes an accountant’s manual resubmit safe: the resubmitted batch carries no memory of the first attempt, yet it hashes to the identical key, so the dedup gate recognizes it. A random or time-based key would defeat the entire mechanism — every retry would look brand new.

Step 2 — Back the key with a deduplication store #

The key is only useful if something durable remembers which keys have already taken effect. Model that as a small async store whose commit is idempotent — re-committing a key that already exists is a no-op — and whose reads and writes are serialized so two concurrent workers cannot both believe they are first. In production this is a row in Postgres with a UNIQUE constraint on the key, or a conditional write in DynamoDB; the in-memory version below has the identical contract.

from __future__ import annotations

import asyncio
from dataclasses import dataclass
from datetime import datetime, timezone
from decimal import Decimal
from typing import Optional


@dataclass(frozen=True)
class PostResult:
    """The recorded effect of applying a batch, keyed by its idempotency key."""

    key: str
    gl_confirmation: str
    recovered_amount: Decimal
    committed_at: datetime


class DedupStore:
    """An async, atomically gated ledger of already-applied batches."""

    def __init__(self) -> None:
        self._results: dict[str, PostResult] = {}
        self._lock = asyncio.Lock()

    async def get(self, key: str) -> Optional[PostResult]:
        async with self._lock:
            return self._results.get(key)

    async def commit(self, result: PostResult) -> None:
        """Record the effect. Re-committing the same key is a deliberate no-op."""
        async with self._lock:
            self._results.setdefault(result.key, result)

Storing the PostResult rather than a bare boolean matters: when a redelivery is skipped, the caller still needs to return the original GL confirmation and recovered amount, so the whole batch behaves as though it ran once and returned the same answer.

Step 3 — Retry transient failures under exponential backoff #

An ERP posting endpoint fails in two very different ways. A transient failure — a timeout, a 503, a dropped connection — should be retried, because the same request will likely succeed shortly. A permanent failure — a malformed batch, a closed accounting period — will never succeed and must be quarantined instead. The retry loop below distinguishes them by budget: it retries up to max_attempts, spacing attempts with full-jitter backoff so a fleet of workers does not synchronize into a thundering retry storm.

The delay before attempt $n$ is drawn uniformly below an exponentially growing ceiling:

Crucially, the idempotency gate runs before any posting: a batch whose key is already recorded returns its prior result and never touches the ledger a second time.

from __future__ import annotations

import asyncio
import random
from datetime import datetime, timezone
from decimal import Decimal, ROUND_HALF_UP
from typing import Awaitable, Callable

# The poster receives the batch AND its key so the ERP can dedup server-side.
GLPoster = Callable[[InvoiceBatch, str], Awaitable[str]]


class PoisonBatch(Exception):
    """Raised when a batch exhausts its retry budget and must be quarantined."""

    def __init__(self, key: str, attempts: int, last_error: str) -> None:
        super().__init__(f"{key} failed after {attempts} attempts: {last_error}")
        self.key = key
        self.attempts = attempts
        self.last_error = last_error


async def apply_once_with_retry(
    batch: InvoiceBatch,
    poster: GLPoster,
    store: DedupStore,
    *,
    max_attempts: int = 5,
    base_delay: float = 0.5,
) -> PostResult:
    """Apply a batch to the GL with an exactly-once effect, retrying transient errors.

    The idempotency gate runs first: a key already in the store returns its
    prior result untouched. Otherwise the poster is retried under full-jitter
    backoff; a batch that never succeeds is raised as a PoisonBatch to quarantine.
    """
    key = idempotency_key(batch)

    prior = await store.get(key)
    if prior is not None:
        return prior  # at-least-once delivery collapses to a single effect

    last_error = "unknown"
    for attempt in range(1, max_attempts + 1):
        try:
            confirmation = await poster(batch, key)
        except Exception as exc:  # transient ERP/network failure
            last_error = f"{type(exc).__name__}: {exc}"
            if attempt == max_attempts:
                break
            ceiling = base_delay * (2 ** (attempt - 1))
            await asyncio.sleep(random.uniform(0, ceiling))
            continue

        recovered = sum((amt for _, amt in batch.line_items), Decimal("0.00"))
        result = PostResult(
            key=key,
            gl_confirmation=confirmation,
            recovered_amount=recovered.quantize(Decimal("0.01"), rounding=ROUND_HALF_UP),
            committed_at=datetime.now(timezone.utc),
        )
        await store.commit(result)
        return result

    raise PoisonBatch(key, max_attempts, last_error)

Step 4 — Quarantine poison messages past the retry budget #

A batch that keeps failing must not loop forever, and it must not disappear. When the budget is spent, apply_once_with_retry raises PoisonBatch — a poison message, in queue terminology — carrying the key, the attempt count, and the last error. The batch runner catches it and routes the payload to a durable quarantine, exactly as the parent async batch processing for high-volume invoices runner routes an invalid document, so one hopeless batch never stalls the thousands beside it.

async def run_batches(
    batches: list[InvoiceBatch],
    poster: GLPoster,
    store: DedupStore,
    quarantine: list[PoisonBatch],
) -> list[PostResult]:
    """Apply every batch concurrently; collect poison batches for review."""

    async def _guarded(batch: InvoiceBatch) -> PostResult | None:
        try:
            return await apply_once_with_retry(batch, poster, store)
        except PoisonBatch as poison:
            quarantine.append(poison)  # durable dead-letter store in production
            return None

    outcomes = await asyncio.gather(*(_guarded(b) for b in batches))
    return [r for r in outcomes if r is not None]

Step 5 — Turn at-least-once delivery into an exactly-once effect #

The four parts now compose into the guarantee. Because the poster is handed the idempotency key, the ERP can reject a duplicate request server-side even in the narrow window between the store read and the store commit — the same technique payment APIs use for idempotency keys. Locally, the dedup store short-circuits every redelivery after the first success. The result is that no matter how many times a batch is delivered, crashed-and-restarted, or manually resubmitted, the recovery posts once and only once. Delivery stays at-least-once; the effect becomes exactly-once.

This is worth stating in operational terms, because it changes how an accountant works a stuck close. Retrying stops being a dangerous last resort that first demands proof the original attempt did nothing, and becomes the ordinary, safe response to any ambiguous failure. When the ERP times out and no one can tell whether the write landed, the correct action is simply to resubmit: if the batch already posted, the key catches it and returns the same confirmation; if it did not, the retry completes it. Either way the ledger ends in the one correct state, which is exactly the property that lets the fan-out run unattended overnight.

Gotchas & Known Limitations #

The failure modes here are subtle because the happy path looks identical whether or not the guard actually works — you only discover a broken key at year-end when a tenant is double-charged. Treat this as a pre-flight checklist:

• Non-deterministic keys. Any timestamp, UUID, or float-derived value in the key means every retry looks new and the guard silently does nothing. Hash only the stable content, with amounts canonicalized as quantized Decimal strings.
• Read-then-post race with no server-side key. The local get/commit window can let two concurrent workers both post before either commits. Pass the key to the poster so the ERP dedups server-side; the local store is an optimization, not the sole guarantee.
• Committing before the write is durable. If you record the key before the GL confirms, a crash between the two leaves the store claiming success that never happened, and the recovery is silently dropped. Commit the key only after the poster returns a confirmation.
• Retrying permanent failures. A closed period or a schema violation will fail on every attempt; burning five retries on it just delays the quarantine. Classify permanent errors and quarantine them immediately instead of spending the budget.
• No jitter in the backoff. Fixed exponential delays let a fleet of workers retry in lockstep and hammer a recovering ERP. Full jitter — a uniform draw below the ceiling — spreads the load.
• Unbounded quarantine. A poison batch that lands in a dead-letter store nobody watches is a lost recovery. Alert on quarantine depth and give operators a replay path once the root cause is fixed.
• Key collisions across periods. Two different periods that happen to recover identical amounts must not share a key. Scope the key with property_id and period, as above, so identity is content plus context.

Verification #

Prove the two properties that matter — equal work hashes equally, and a duplicate delivery causes zero additional GL writes — with assertions on both, rather than trusting that the happy path implies the guard works. The middle test is the load-bearing one: it counts calls into the poster and asserts a second delivery adds none.

import asyncio
from decimal import Decimal


def _batch(hvac: str = "1200.00") -> InvoiceBatch:
    return InvoiceBatch(
        property_id="PROP-7",
        period="2026-06",
        line_items=(("5100-HVAC", Decimal(hvac)), ("5200-LOT", Decimal("300.00"))),
    )


def test_equal_work_hashes_equally() -> None:
    # 1200 and 1200.00 canonicalize identically -> one key, so a resubmit is safe.
    assert idempotency_key(_batch("1200")) == idempotency_key(_batch("1200.00"))


def test_retry_then_succeed_applies_exactly_once() -> None:
    calls = {"n": 0}

    async def flaky(batch: InvoiceBatch, key: str) -> str:
        calls["n"] += 1
        if calls["n"] < 3:
            raise ConnectionError("ERP timeout")  # transient
        return "GL-OK"

    async def run() -> None:
        store = DedupStore()
        b = _batch()
        r1 = await apply_once_with_retry(b, flaky, store, base_delay=0.0)
        assert r1.recovered_amount == Decimal("1500.00")
        posts_after_first = calls["n"]
        # A duplicate delivery of the same batch touches the GL zero more times.
        r2 = await apply_once_with_retry(b, flaky, store, base_delay=0.0)
        assert r2.gl_confirmation == r1.gl_confirmation
        assert calls["n"] == posts_after_first          # exactly-once effect

    asyncio.run(run())


def test_poison_batch_is_quarantined_and_uncommitted() -> None:
    async def always_fails(batch: InvoiceBatch, key: str) -> str:
        raise TimeoutError("GL unreachable")

    async def run() -> None:
        store = DedupStore()
        try:
            await apply_once_with_retry(
                _batch(), always_fails, store, max_attempts=3, base_delay=0.0
            )
        except PoisonBatch as poison:
            assert poison.attempts == 3
            assert await store.get(poison.key) is None   # nothing was committed
        else:
            raise AssertionError("expected a PoisonBatch")

    asyncio.run(run())

Setting base_delay=0.0 keeps the tests fast while still exercising the retry path. Extend the suite with a property-based test that delivers the same batch a random number of times through a poster which counts real writes, then asserts the write count is always exactly one — the concrete statement of $effects = \min(d, 1) = 1$ for any delivery count $d \ge 1$. A concurrency test that fires many apply_once_with_retry calls for one key through asyncio.gather and asserts a single confirmation is worth adding once your dedup store is the real database.

Once the guard is green, retried batches feed cleanly into GL code mapping for CAM expenses with no risk of a duplicated recovery, and the whole retry loop belongs to async batch processing for high-volume invoices; when a batch is genuinely un-postable rather than merely unlucky, hand it to quarantining invalid CAM expense records instead of retrying it forever.

Related #

• Async batch processing for high-volume invoices — the parent stage whose worker pool this idempotent-retry loop makes crash-safe and resubmittable.
• Quarantining invalid CAM expense records — where a poison batch that exhausts its retry budget is routed for human review and replay.
• GL code mapping for CAM expenses — the downstream stage that consumes each recovery this loop applies exactly once.