Webhook retries pushed payments-worker p95 past its rolling baseline
One deploy, two UTC windows, and the trace evidence that isolated the retry loop.
The rollback held through a complete five-minute verification window.
Incident p95
912 ms
+611 ms
- baseline p95
- 301 ms · 24h rolling
- service
- payments-worker · prod
- onset
- 14:02:11Z · UTC
- 24h rolling baseline
- 2026-08-05 14:01:47 UTC to 2026-08-06 14:01:46 UTC
- p95 · ms · 184,240 of 184,240 samples
- incident window
- 2026-08-06 14:02:11 UTC to 2026-08-06 14:18:00 UTC
- p95 · ms · 2,410 of 2,437 samples
Current health refreshed 14:31:09Z.
Trace evidence
Baseline against incident
The retry span moved by 959 ms while database and flush time held.
http.server
463 ms → 1,284 ms
webhook.retry
190 ms → 1,149 ms
pg.query
118 ms → 118 ms
signal.flush
2 ms → 2 ms
| Span | Baseline | Incident | Change |
|---|---|---|---|
| http.server | 463 ms | 1,284 ms | +821 ms |
| webhook.retry | 190 ms | 1,149 ms | +959 ms |
| pg.query | 118 ms | 118 ms | +0 ms |
| signal.flush | 2 ms | 2 ms | +0 ms |
Release evidence
The deploy that changed
The selected release raised retry attempts, backoff, and the total budget in one commit.
| Field | Before | 8f21c4e | Rollback |
|---|---|---|---|
| retry.maxAttempts | 1 | 3 | 1 |
| retry.backoff | 0 ms | 380 ms | 0 ms |
| retry.budget | 250 ms | 1,200 ms | 250 ms |
| retry.scope | failed webhooks | failed webhooks | failed webhooks |
UTC sequence
Evidence in order
Times are normalized to UTC. Collector skew was +3.2 seconds before correction.
- Deploy01
Deploy landed
8f21c4e by @nadia changed the webhook retry budget.
- Impact02
Regression began
payments-worker p95 left its 24h rolling envelope 24 seconds later.
- Evidence03
Deploy attributed
The release-window distance ranked 8f21c4e above two older deploys.
- Evidence04
Retry loop identified
Aligned traces isolated 1,149 ms inside webhook.retry.
- Recovery05
Rollback applied
4b9e2d1 restored the previous retry budget.
- Recovery06
Incident resolved
Five complete minutes held p95 at 309 ms and errors at 0.05%.
Failure groups
The retry loop dominates
Complete traces are ranked by matched signature, with counts and shares kept visible.
Aggregation
- 1
Repeated webhook retries
1,412 · 58.6%
Three attempts with 380 ms backoff dominate the incident trace.
- 2
Retry budget exhausted
612 · 25.4%
The 1,200 ms budget ends before the third attempt returns.
- 3
Downstream timeout
386 · 16.0%
The receiver times out after the worker has already retried.
Resolution receipt
Recovery, with receipts
Rollback, verification, and current health are separate facts with separate timestamps.
- rollback
- 4b9e2d1
- applied
- 14:21:10Z UTC
- resolved
- 14:28:11Z UTC
- verification
- 14:23:11Z–14:28:11Z UTC
- verified p95
- 309 ms
- verified samples
- 17,842
Current health: healthy
Health refreshed 14:31:09Z UTC.