Paid in Stripe, Unsubscribed in the DB: Preventing Double Charges

TL;DR

The payment succeeded in Stripe, but only the webhook's DB update failed, so the screen still showed the old plan. Every time the user retried, a separate subscription was created. After refunding and recovering the DB, I added PostgreSQL locking, persistence of incomplete Checkouts, and state-transition tests for the webhook.

Who this is for

Anyone implementing subscription billing with Stripe Checkout and webhooks. Especially if you think "I added Stripe's idempotency keys, so double charges can't happen."

Before	After
Each retry created a new Customer and a new Subscription	Modify the existing Subscription in place, on the same object
A partial webhook failure rolled back all DB updates	Re-runnable upserts and state guards
Relied on a few-second idempotency key	DB locking and a unique constraint on incomplete Checkouts
Assumed webhooks arrive roughly in order	Tested duplicates, delays, and out-of-order delivery

It started with "It didn't apply, so I paid again"

A single inquiry arrived at Kotonia, which I run solo.

The user had intended to move from an old plan to a higher one. But the screen kept showing the old plan, and "Manage subscription" showed no new contract either. Assuming it hadn't gone through, they paid again.

Meanwhile, three invoices had arrived in their inbox. At that point it was clearly not a "display lag" but likely multiple billing objects being created.

This retry is not abnormal behavior. If you pay and the screen doesn't change, you think "maybe it didn't submit" and press again. It was an action the design side should have absorbed safely.

Two different truths formed in Stripe and the app DB

On investigation, Stripe held multiple Customers and Subscriptions whose metadata pointed to the same app user. The payments themselves had succeeded.

But the app DB still held the Customer ID and Subscription ID of the first, old-plan contract. Stripe's Customer Portal opens tied to that Customer ID, so contracts created under other Customers were invisible to the user.

Stripe:  old plan + changed plan + retries
App DB:  old plan only
Portal:  only the contract of the Customer the DB points to

Both the payment system and the app were correct within the data each one held. But the system as a whole was wrong.

The webhook arrived, but the transaction failed

The contract-completion webhook processing was, roughly, in this order:

1. Update the contract state in users
2. Create a workspace for the subscriber
3. Add the user as an administrator
4. Commit once everything succeeds

On the first contract, a "My Workspace" had already been created. On a plan change it INSERTed with the same name, colliding with a unique constraint. The whole transaction rolled back, wiping even step 1's user update.

On top of that, the webhook Endpoint's API version was newer than the type definitions of the Stripe client in use, and for some Subscription events deserialization failed — a condition that overlapped.

This wasn't a single bug. "A Checkout that can create a new contract," "insufficient re-runnability of the webhook transaction," and "an API version gap" lined up in series and became an incident.

The first thing I did was refunds and recovery, not fixing the cause

Before the permanent fix, I made a state where the user wouldn't lose out.

• Matched Customers, Subscriptions, Invoices, and Charges on Stripe by metadata
• Kept exactly one Subscription for the intended higher plan
• Canceled the old plan and the duplicates, and refunded the corresponding payments
• Aligned the app DB's Customer ID, Subscription ID, and plan to the kept contract
• Aligned the workspace's monthly usage cap to the new plan

Since some Charges had already been refunded during the investigation, I always checked Stripe's latest state before acting, so as not to refund the same payment twice. During incident response, it matters not only to "do the right thing quickly" but also to "not duplicate a right action already taken."

For a plan change on an existing contract, don't create a new Subscription

The biggest fix was splitting the contract flow in two.

Has an active Subscription
  → Change the Price of the same Subscription in the Billing Portal

Has no active Subscription
  → Create a Checkout Session

Rather than opening Checkout for existing subscribers, I update the existing Subscription via the Billing Portal's subscription_update_confirm flow. This removed the structure where every plan change adds a Customer and a Subscription.

When a new Checkout is genuinely needed, I reuse the Customer ID if it remains in the DB. Only on the first time do I pass the account's email address, and I also save the user ID and plan in client_reference_id and metadata so I can cross-reference during support.

Stop retries with PostgreSQL locking and incomplete Checkouts

Stripe's Idempotency Key (a feature that treats API operations with the same key as one) is necessary, but it alone can't guarantee uniqueness across the whole app.

If the key changes between requests, it becomes a different operation. In a multi-instance setup, an in-process Mutex isn't shared either. So I take a per-user PostgreSQL advisory transaction lock when creating a Checkout.

SELECT pg_advisory_xact_lock(:billing_namespace, :user_id);

Further, I save the Stripe Checkout Session ID, plan, URL, expiry, and status in the DB. For rows with status = 'open', I set a partial unique index that allows only one per user.

CREATE UNIQUE INDEX one_open_checkout_per_user
ON subscription_checkout_sessions (user_id)
WHERE status = 'open';

With this, mashing the same plan's button returns the existing Checkout URL. Even with another tab, a retry minutes later, or concurrent execution from multiple servers, you can't create two incomplete Checkouts. If there's an incomplete Checkout for a different plan, new creation is rejected, and the Session expires in 31 minutes.

Webhooks don't arrive "once, in order"

Stripe's official webhook docs also state that the same event may arrive multiple times and that delivery in creation order is not guaranteed. You must not treat duplicates and delays as anomalies.

As countermeasures, I implemented these rules:

• Store the Stripe Event ID uniquely to prevent reprocessing the same event
• Lock the user row with FOR UPDATE on checkout.session.completed
• If there's another active/trialing Subscription, don't let a late Checkout replace it
• A resend of the same Subscription can be safely re-applied
• Apply subscription.deleted only when it matches the Subscription ID the DB currently points to
• Make workspace and member additions ON CONFLICT ... DO UPDATE
• Keep a fallback that reads only the minimal fields after signature verification, for when a new Stripe API version can't be read by the types

Plan changes arrive as subscription.updated, so I update not only the user's plan but also the existing workspace's monthly cap in the same transaction.

Instead of monkey testing, break the state transitions

A billing flow is a poor match for monkey testing that randomly clicks browser buttons. It involves real payments, asynchronous webhooks, and an external Customer Portal, and waiting for a result to complete is slow and flaky.

This time, instead of UI operations, I generated a sequence of events against the billing state.

CheckoutCompleted (normal contract)
CheckoutCompleted (resend of the same event)
CheckoutCompleted (a late, duplicate contract)
SubscriptionDeleted (a past contract)

On top of fixed cases, I ran 50,000 steps of duplicate and delayed events with a deterministic random seed. The invariant: in any order, the intended Subscription ID and active state must not be changed by some other, older event.

Then I actually connected to the Stripe Sandbox and confirmed that creating a Checkout twice with the same Idempotency Key returns the same Session ID. The test Session was expired on the spot. I also applied the migration to PostgreSQL and confirmed that a second open row is rejected by the unique constraint.

The "external vs. DB split" that's hard to catch even in tests

Even with all this, I can't claim to reduce billing inconsistencies to zero. That's because a successful API operation on Stripe and a commit to my own DB can't be one ACID transaction.

For example, if the network drops right after Stripe creates a Checkout, the app never receives the success. When the DB save fails, I made untrackable Checkout Sessions auto-expire, but the space of failure patterns is wide.

The next lever is reconciliation (periodic matching). Every day, compare active Subscriptions on Stripe with the contract state in the app DB, and alert on:

• A single user with multiple active Subscriptions
• Active on Stripe but inactive in the DB
• The Customer–Subscription ownership the DB points to doesn't match
• Stripe's Price doesn't match the DB's plan and usage cap

Tests are for "not creating violations that shouldn't happen." Reconciliation is for "quickly finding violations that happened anyway." Billing needs both, I think.

Seven lessons from the incident

1. Don't treat a user's retry as an anomaly. If it doesn't apply, pressing again is natural.
2. Separate changing an existing contract from creating a new one. Even starting from the same button, the meaning differs.
3. Don't use an Idempotency Key as a substitute for exclusive locking. You need app-side unique constraints and state management.
4. Assume webhooks are duplicated, delayed, and out of order. Decide applicability by current state and target ID, not just event ID.
5. Make every operation inside the webhook transaction re-runnable. Suspect the places where you need an upsert instead of an INSERT.
6. Align the external API version with the SDK's type definitions. A successful signature verification and a successful payload type conversion are different problems.
7. Keep both tests and periodic reconciliation. Preventing and detecting inconsistencies are different layers of defense.

How this lives on in Kotonia

This fix is in the subscription contracts and plan changes that start from Kotonia's pricing page. Not just button-mashing and separate tabs, but a retry minutes later also merges into the same incomplete Checkout.

At the same time, I now treat billing work as more than "writing payment code" — refunds, DB recovery, prevention, and detection as one operation. For anything involving money, I came to feel it only becomes a feature once you've built it all the way to "even on failure, you can safely roll back," rather than just "it works."

---

This article is based on an actual incident, but customer information such as invoice numbers, email addresses, Stripe Customer IDs, and Subscription IDs has been omitted or anonymized.