Preventing Race Conditions in a Polling-Fallback Webhook System

[Litis-Trion250]

Select Topic Area

Question

Body

Integrating with a third-party payment provider that notifies us of transaction status changes via Webhooks. However, due to the nature of their service, we are instructed to always use a polling fallback mechanism to fetch the final, authoritative transaction data after receiving a webhook event.

The Problem: We are experiencing a critical race condition and data inconsistency issue.
Scenario:

1. Time T0: A transaction completes.

2. Time T1 (Webhook): We receive the transaction. completed webhook event.

3. Time T2 (Polling): Our webhook handler immediately calls the third-party API to fetch the full transaction details.

4. Time T3 (Data Delay): The third-party provider's internal database replication is slightly delayed. The API call at T2 returns the OLD (pre-completion) status because the database hasn't updated yet.

5. Time T4 (Correct Data): A few milliseconds later, the database updates, but our system has already processed the transaction with the incorrect status from T2.

[Rocky-Dewan]

The Core Solution: Exponential Backoff and Retry

Since you are mandated to poll for the authoritative data, you must assume the data might not be ready immediately. Instead of a single poll, your handler needs to poll repeatedly until the expected state is confirmed or a timeout is reached.Process Flow:

1. Receive Webhook: Event transaction.completed is received.

2. Start Retry Loop: Begin a loop with a maximum number of attempts (e.g., 5-10) and a total timeout (e.g., 30 seconds).

3. Poll API: Call the third-party API to fetch the transaction details.

4. Check State:
   • Success: If the returned data confirms the expected state (e.g., status: 'completed'), break the loop and process the data.
   • Failure: If the returned data is the old state (e.g., status: 'pending'), log a warning and proceed to the next step.

5. Backoff: Wait for an increasing amount of time before the next poll (Exponential Backoff). This prevents overwhelming the provider's API.

6. Timeout: If the loop finishes without the correct state, log a critical error and trigger a manual review or a long-running background job to check later.

The Retry Logic:

import time
import requests

MAX_TRIES = 8
INIT_WAIT = 0.5
TID = 'tx_12345' 
EXP_STAT = 'completed'
API_URL = f'https://api.thirdparty.com/tx/{TID}'

for t in range(MAX_TRIES ):
    try:
        r = requests.get(API_URL, headers={'Auth': 'Bearer TOKEN'})
        r.raise_for_status()
        d = r.json()

        if d.get('status') == EXP_STAT:
            print(f"Success on try {t + 1}.")
            process_tx_data(d)
            break
        else:
            print(f"Try {t + 1}: Status {d.get('status')} != {EXP_STAT}")

    except requests.exceptions.RequestException as e:
        print(f"API Fail: {e}")

    if t < MAX_TRIES - 1:
        wait_time = INIT_WAIT * (2 ** t)
        print(f"Wait {wait_time}s...")
        time.sleep(wait_time)
else:
    print(f"CRITICAL: {TID} failed after {MAX_TRIES} tries.")
    alert_manual_review(TID)

[KRSNA-BLR]

Nice explanation! 👏
I’d also suggest adding a small random jitter to the exponential backoff.
That helps when many webhook events arrive at the same time and all start polling together — it avoids API burst traffic.

For example:

import random
wait_time = INIT_WAIT * (2 ** t) + random.uniform(0, 0.3)