Retries and replay

Delivery is a queue, not a side effect of ingestion. An event is written as a row that says "this endpoint owes this event", and a scheduled job sends it — which is the only reason retries can exist at all. An in-request POST has nowhere to record that it should be tried again.

The schedule

Seven attempts across about seven hours, backing off each time.

| Attempt | After | | ------- | ----------- | | 1 | immediately | | 2 | 30 seconds | | 3 | 2 minutes | | 4 | 10 minutes | | 5 | 45 minutes | | 6 | 2 hours | | 7 | 4 hours |

Anything that answers 2xx is done. Anything else is retried, including a timeout — which is why a handler should acknowledge first and work afterwards.

The dead letter

After the seventh attempt the delivery stops and the event stays. Nothing is discarded, so an endpoint that was down for a morning is a morning of events waiting rather than a morning of events lost.

They are listed under Developers → Events, with the response code and body of every attempt, and each one can be replayed by hand once the endpoint is fixed.

curl https://pidgeon.ai/api/v1/events/evt_9c41b/replay -X POST -H "Authorization: Bearer $PIDGEON_API_KEY"

Delivery is at-least-once

A 2xx that never reaches us — a connection dropped after your handler committed — is a delivery we will try again. Handlers have to tolerate seeing the same event twice.

Every event carries a stable id. Recording the ones you have processed and returning early on a repeat is the whole of the fix.

{ "id": "evt_9c41b…", "type": "message.received" }

Meta-events are logged, never delivered

webhook.failed and webhook.dead_lettered appear in the event log so you can see what happened. They are not themselves delivered to your endpoint, because an endpoint that is failing is the last place to send news that it is failing.