A successful payment is not a completed workflow
A customer pays, the provider sends an event, and your worker grants access. The difficult case begins when the worker commits the entitlement but crashes before acknowledging the message. A retry now looks indistinguishable from unfinished work unless the database records both the business effect and the processing decision.
This article proposes an architecture for that boundary. The event inbox records what arrived; a business ledger records what changed; reconciliation detects what never arrived. Each answers a different operational question. Combining them into a single processed boolean hides the evidence needed to repair an incident.
Stripe documents that webhook deliveries can arrive out of order and that event timestamps are insufficient for ordering or deduplication. Verify signatures against the raw request body, then persist the accepted event before acknowledging it. These are provider-specific requirements, not assumptions to infer from an HTTP success response. Stripe webhook documentation
Give transport identity and business identity separate constraints
Use a unique inbox key containing the provider, account, environment, and event ID. The account and environment prevent test traffic or connected-account identifiers from sharing the wrong namespace. Restrict raw payload access and retention because webhook bodies may contain customer data.
For fulfillment, introduce a different key such as the order ID plus fulfillment type. A second event describing the same paid order must not produce a second shipment, credit, or license. The following is an illustrative schema fragment; provider credentials, authorization, and retention controls belong elsewhere.
CREATE TABLE webhook_inbox (
provider text NOT NULL,
provider_account text NOT NULL,
environment text NOT NULL,
event_id text NOT NULL,
received_at timestamptz NOT NULL DEFAULT now(),
state text NOT NULL DEFAULT 'pending',
attempts integer NOT NULL DEFAULT 0,
PRIMARY KEY (provider, provider_account, environment, event_id)
);
CREATE TABLE fulfillment_effects (
order_id uuid NOT NULL,
effect_type text NOT NULL,
recorded_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (order_id, effect_type)
);
The uniqueness constraint is the concurrency boundary. A preliminary SELECT followed by an unconstrained INSERT does not protect against two workers making the same decision simultaneously.
Commit local effects together
Inside one transaction, lock the relevant order, validate its transition, insert the fulfillment effect, update the entitlement, and mark the inbox event complete. If any step fails, roll back all of them. If the effect already exists, record a duplicate decision without repeating fulfillment.
External side effects cannot join this transaction. An email, shipment request, or remote credit needs an outbox record committed alongside the local effect. Send it later using a stable operation identifier and the destination’s documented retry contract. Never hold an order lock while waiting on an external API.
Stripe's idempotency keys apply to outgoing API requests, with documented retention and parameter-matching behavior. They are not a replacement for a permanent local business ledger. Reconcile uncertain outcomes before reissuing an old operation outside the provider's retention window. Stripe idempotent requests
Treat state transitions as evidence-dependent decisions
| Incoming situation | Safe decision | Evidence to retain |
|---|---|---|
| Same event delivered twice | Reuse the recorded result | Inbox identity and original decision |
| Different event for an already fulfilled order | Skip repeated fulfillment | Business effect key |
| Cancellation arrives before creation | Retrieve current provider state or defer | Missing prerequisite and retry deadline |
| Refund follows successful payment | Apply a separate refund transition | Refund identifier and affected entitlement |
| Provider request times out | Reconcile before issuing a new operation | Stable request key and uncertainty state |
Do not impose a simplistic ordering such as “paid always wins.” Refunds, disputes, and reversals are legitimate later business events. Encode permitted transitions and their prerequisites rather than comparing arbitrary status strings or timestamps.
Reconciliation is part of the write path
Periodically compare provider records with local orders using a bounded time window and a durable checkpoint. Include an overlap window so delayed records are revisited. Upsert findings by stable identifiers; a reconciliation pass must be safe to repeat.
Distinguish recoverable lag from contradictions. An unpaid local order with a confirmed remote payment can enter a repair workflow. An order fulfilled twice requires investigation and compensation. Store the repair reason, actor, previous state, and new state, not just the corrected final value.
Assign an owner and maximum acceptable age to unresolved discrepancies. A dashboard full of old “pending” rows is a silent failure even when the endpoint returns only successful responses.
Test the crash boundaries before launch
Inject failure after inbox persistence, after the entitlement update but before transaction commit, and after commit but before acknowledgment. Deliver the same event concurrently. Deliver refund and payment events in reverse order. Disable the worker while keeping the webhook endpoint healthy, then measure recovery after restarting it.
Assert business invariants: at most one fulfillment effect per order and type; no entitlement without a valid paid-state decision; no acknowledged event lost before durable acceptance. Track inbox age, retry count, unresolved reconciliation age, and repaired contradictions. Request throughput alone cannot tell you whether customers received the correct access.
The release decision should depend on these invariants and recovery exercises. A webhook integration is complete when it can explain and repair an interrupted workflow, not merely when a test event returns HTTP 200.