How do I handle duplicate and retried webhooks safely on my Droplet?

Question

I have a Node.js app on a Droplet behind Nginx that receives webhooks from a payment provider. Last week the same event arrived twice, a few minutes apart, and one customer got credited two times. The provider docs say they retry when they do not get a 2xx response, but my endpoint returned 200 both times.

What is the right way to handle this? Should I process the event right away or do it later? How do I make sure the same event is never applied twice? And how do I check that the request really came from the provider?

Bobby · Answer

Hi there, First, duplicates are normal. Most providers use at-least-once delivery, so your code has to handle the same event twice. Even a perfect endpoint will see duplicates sometimes. To see why you got a retry after returning 200, look at the status Nginx logged, not the one your app logged. They can differ: 499 means the provider closed the connection before you answered. Their timeout is often short. 504 means Nginx gave up waiting for your app. If your app was slow, it can log 200 while the provider only saw a timeout. Add the response time to a log so you can see this: log_format webhook '$remote_addr [$time_local] "$request" $status $request_time'; access_log /var/log/nginx/webhook.log webhook; Also check the provider dashboard for the same URL registered twice, for example a test and a live endpoint. That gives you two deliveries for every event. Whatever the cause, the fix is the same. Do these three things: Verify the signature on the raw body, not on parsed JSON. One reordered key breaks the check. The header name and format depend on your provider, so check their docs. Save the event ID with a unique constraint, in the same database transaction as the work. Use the ID the provider sends, because it stays the same across retries. Keep the handler short. If the work is a quick database write, do it in that same transaction. If it is slow, save the event and hand it to a job queue, then return 2xx. const crypto = require('crypto'); function verify(rawBody, signature, secret) { const expected = crypto.createHmac('sha256', secret).update(rawBody).digest('hex'); const a = Buffer.from(expected); const b = Buffer.from(signature || ''); return a.length === b.length && crypto.timingSafeEqual(a, b); } app.post('/webhooks/payments', express.raw({ type: 'application/json' }), async (req, res) => { if (!verify(req.body, req.get('x-signature'), process.env.WEBHOOK_SECRET)) { return res.sendStatus(401); } const event = JSON.parse(req.body); const client = await pool.connect(); try { await client.query('BEGIN'); const result = await client.query( 'INSERT INTO processed_events (event_id) VALUES ($1) ON CONFLICT DO NOTHING', [event.id] ); if (result.rowCount === 1) { await creditCustomer(client, event); // same transaction } await client.query('COMMIT'); res.sendStatus(200); } catch (err) { await client.query('ROLLBACK'); res.sendStatus(500); } finally { client.release(); } }); The table is simple: CREATE TABLE processed_events ( event_id text PRIMARY KEY, received_at timestamptz NOT NULL DEFAULT now() ); Because the insert and the credit are in one transaction, a failure rolls back both. The provider then retries, and the retry works. A duplicate finds the ID already there, does nothing and still returns 200. To see how senders react to slow or failing endpoints, try this webhook delivery simulator . You choose the endpoint response, including timeouts and random failures, and watch the retries and backoff. You can also redeliver the same message, and tamper with a signed body to see the check fail.

alexdo · Answer

Hi there, Bobby’s answer covers the dedup and signature-verification side about as thoroughly as it gets, one thing worth adding on the nginx side since that 504 scenario is worth being able to actually fix, not just diagnose. nginx’s proxy_read_timeout, proxy_send_timeout, and proxy_connect_timeout all default to 60 seconds. If your webhook handler does any real work before returning (a slow database write, an external API call, anything that isn’t instant), it’s easy to blow past that window under load, even if the handler reliably finishes in under a second during normal testing. nginx gives up, logs the 504, and closes the connection to the provider, who never sees your 200 and retries, exactly the gap Bobby’s logging tip surfaces. The fix is to raise proxy_read_timeout specifically for the webhook location, not globally: location /webhooks/payments { proxy_pass http://127.0.0.1:3000; proxy_read_timeout 15s; } Scoping it to the webhook location rather than bumping it site-wide matters, since a high global timeout just means other genuinely broken or hung requests also sit around longer before nginx gives up on them. 15 seconds is a reasonable starting point for something that should already be fast per Bobby’s advice to keep the handler short and queue slow work, tune it against what your own webhook route actually needs. Regards

Become a contributor for community

DigitalOcean Documentation

Resources for startups and AI-native businesses

The developer cloud

Start building today

Report this

How do I handle duplicate and retried webhooks safely on my Droplet?

Related questions

Why does my Node.js app use way more memory in production than in local testing?

My Droplet got flooded with requests overnight. What should I do now, and how do I prepare for next time?

What Ubuntu packages and services should I check before deploying to a fresh Droplet?

Mysql Data disappears intermittently

What is the best way to send automated email reports from a Python script on a Droplet?

How does SMTP actually work, and why do my emails keep ending up in spam?

Self-hosting observability for your AI agents on a $6 Droplet

Best way to securely access a homelab remotely without relying only on Tailscale?

VLESS VPN connects but no traffic passes through (DigitalOcean VPS + x-ui + Xray)

Nodemailer stopped working after migrating from shared hosting to a DigitalOcean Droplet

New account with GitHub Student Pack credits cannot create Droplets or use Launchpad

What’s the status of CopyFail patches in shared hosts?

Report this

How do I handle duplicate and retried webhooks safely on my Droplet?

Related questions

Why does my Node.js app use way more memory in production than in local testing?

My Droplet got flooded with requests overnight. What should I do now, and how do I prepare for next time?

What Ubuntu packages and services should I check before deploying to a fresh Droplet?

Mysql Data disappears intermittently

What is the best way to send automated email reports from a Python script on a Droplet?

How does SMTP actually work, and why do my emails keep ending up in spam?

Self-hosting observability for your AI agents on a $6 Droplet

Best way to securely access a homelab remotely without relying only on Tailscale?

VLESS VPN connects but no traffic passes through (DigitalOcean VPS + x-ui + Xray)

Nodemailer stopped working after migrating from shared hosting to a DigitalOcean Droplet

New account with GitHub Student Pack credits cannot create Droplets or use Launchpad

What’s the status of CopyFail patches in shared hosts?