Two months ago, I ordered pizza from a well-known pizza chain. Just after I pressed Pay, the app showed an error message, so I simply placed the order again, paid, and went on with my day. A while later I received SMS messages with two charges from that same restaurant.
Payment processing involves some intricate details - some of which even surprised me the first time I designed a payment integration. In this article I will unpack a few of those details and show how careless design leads to real damage such as the same order being delivered twice, access granted against inactive subscriptions, and goods shipped against payments that never completed. Along the way, we will discover how to design a system that protects itself from exactly these problems.
The fragile nature of webhooks
Nearly all modern payment providers (like Stripe) rely on sending events asynchronously through a webhook to their consumers - our apps (events such as Payment Pending, Payment Completed, and so on). The consumer relies on those events to provide business value (such as shipping an order or granting access to a digital service). But many people miss that those payment providers neither guarantee the ordering of those events nor that each one is delivered only once. For instance, a Payment Pending event can reach you after a Payment Completed event [1] and a single Payment Completed can reach you twice. Leaving that unhandled can cause real trouble.
Without a proper design, receiving a Payment Completed event twice means a product is shipped twice, to a customer who only paid once.
Out-of-order events are easier to miss. Say a subscription was created and later ended, If Subscription Ended arrives before Subscription Created and you process them in that order, you grant access to someone whose subscription is already gone.
Avoiding duplicate and out-of-order events
So how do we avoid all this?
The Inbox Pattern
Most payment providers deliver webhooks using at-least-once-style retries, they keep sending until the consumer answers with a 2xx. This is the reason why we get duplicates.
To help handle this duplication, webhook events contain a deduplication key, a unique key that stays the same across retries. That key is what the Inbox Pattern builds on.
The idea is simple. Instead of processing the event the moment it arrives, first store it in a table with a unique key constraint on the event id. If a duplicate shows up, the constraint rejects it so it never gets stored.
Later a background worker processes the events from the database records, which are guaranteed not to be duplicates.
Handling out-of-order events
Here we need to be careful with any process that assumes events arrive in order. The right fix depends on the case, but one approach that works broadly - and recommended by Stripe - is when an event is ambiguous or out of the expected order, call the API to fetch the current state and build your internal state from that, rather than trusting the event's payload as the most recent source of truth.
The actual design - a generalizable model
What if the system crashes midway?
Suppose our application crashes or times out after it records the payment but before it starts processing the order, We've taken the money and never shipped the order. (Do you see it? That's exactly what could have happened with my pizza.)
For this we will need to use the Transactional Outbox Pattern.
In this article we saw how to protect a service from events arriving out of order, and how to store events so a payment is never processed twice. Next time we'll use Idempotency Keys and the Outbox Pattern so we never take a customer's money without delivering their pizza. 🙂